Storage & Part Files

Unlike simple single-file JSON databases, file-json-db splits each collection into multiple part files. This page explains why and how that works.

Why part files?

A single large JSON array has several problems:

• The whole file must be read into memory to change one document
• A single corrupted byte can make the entire collection unreadable
• Atomic rewrite of a multi-megabyte file is slow and risky

By splitting data into smaller chunks (parts), the library can:

• Limit the amount of data loaded for a given operation
• Isolate corruption to a single part
• Keep individual writes fast

How parts are named

Part files are named part-XXXX.json where XXXX is a zero-padded number:

part-0001.json
part-0002.json
part-0003.json
…

The padding is done by the internal helper padPart(num) so that lexical sorting matches numeric order.

When is a new part created?

A new part is started when the current part reaches either:

• maxPartSize (default 256 KB), or
• maxRecords (if you set it)

const logs = db.collection({
  name: 'logs',
  maxPartSize: 128 * 1024,  // 128 KB
  maxRecords: 500           // or 500 documents
});

index.json – the collection brain

Every collection directory contains an index.json file that holds:

• nextId – next auto-increment value
• parts – map of part key → { count, size }
• idIndex – primary key → which part contains the document
• indexes – secondary indexes for the fields you configured
• corruptedParts – parts that previously failed to parse

This file is also written atomically (temp file + rename).

Atomic writes in detail

Every time a part or the index is saved, the library does:

1. Write the full content to a temporary file in the same directory: .part-0001.json.<pid>.<timestamp>.tmp
2. Call fs.rename (sync or async) to move the temp file over the real name
3. If rename fails (rare on some filesystems), retry up to 5 times with a short delay
4. On failure, clean up the temp file and throw

On POSIX systems a rename within the same filesystem is atomic. If the process crashes after writing the temp file but before the rename, the original file remains intact. The next start simply ignores the orphaned temp file.

Reading a part

When a part is needed, the library:

1. Checks whether the part is marked as corrupted → skip it
2. Reads the file as UTF-8 text
3. Parses the JSON array
4. On parse error (and errorLevel: 'debug'): copy the file once to crash/, record it in corruptedParts, emit an error, return an empty array

Typical directory after some use

data/
├── users/
│   ├── index.json
│   ├── part-0001.json
│   ├── part-0002.json
│   └── crash/
│       └── part-0003.json   ← only if a part became corrupted
└── orders/
    ├── index.json
    └── part-0001.json