Architecture

This page explains how file-json-db works under the hood. Understanding the architecture helps you make better decisions about part sizes, indexes, concurrency, and recovery.

High-level overview

┌─────────────────────────────────────────────────────────┐
│                     Your Node.js App                    │
│                                                         │
│   const db = new JSONDB('./data');                      │
│   const users = db.collection({ name: 'users', ... });  │
└──────────────────────────┬──────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────┐
│                      JSONDB instance                    │
│  - owns the root directory                              │
│  - manages Collection instances                         │
│  - exposes backup / listCollections / dropCollection    │
└──────────────────────────┬──────────────────────────────┘
                           │
           ┌───────────────┼───────────────┐
           ▼               ▼               ▼
     Collection A    Collection B    Collection C
     (users/)        (orders/)       (products/)
           │
           ├── index.json          (metadata + indexes)
           ├── part-0001.json      (documents)
           ├── part-0002.json
           └── crash/              (isolated bad parts)

1. Database = Directory

When you write new JSONDB('./data'), the library ensures that the directory ./data exists. That directory is the entire database. Everything the library knows about lives inside it.

There is no central server process, no network protocol, and no separate binary. The database is simply files on disk that your process reads and writes.

2. Collection = Subdirectory

Each collection is a subdirectory of the database root. A collection named users lives at ./data/users/.

Inside that directory you will find:

• index.json – the collection’s metadata and indexes
• part-0001.json, part-0002.json, … – the actual document storage
• (optionally) a crash/ folder containing copies of parts that failed to parse

3. Part files – why they exist

A simple JSON-file database usually puts every document of a collection into a single array inside one file. That works until the file becomes large:

• Reading the whole file into memory becomes expensive
• A single corrupted byte can make the entire collection unreadable
• Atomic rewrite of a multi-megabyte file is slower and riskier

file-json-db therefore splits a collection across multiple part files. New documents are appended to the current part. When the part reaches either the configured maxPartSize (default 256 KB) or maxRecords, a new part is created.

users/
├── index.json
├── part-0001.json   ← first ~256 KB or N documents
├── part-0002.json
└── part-0003.json

4. The index (index.json)

The in-memory (and on-disk) index is the brain of a collection. It contains:

Because the primary and secondary indexes live in memory after the collection is loaded, equality lookups on an indexed field are O(1) (or O(k) where k is the number of documents that share the same value).

5. Atomic writes

Every time the library needs to persist a part or the index, it follows the classic safe-write pattern:

1. Write the new content to a temporary file in the same directory (.part-0001.json.<pid>.<timestamp>.tmp)
2. Call fs.rename (or the async equivalent) to move the temp file over the real file
3. If the rename fails (rare race on some filesystems), retry a few times

On POSIX systems a rename within the same filesystem is atomic. If the process crashes between writing the temp file and the rename, the original file remains intact. The next start simply ignores the orphaned temp file.

6. Concurrency model

Each collection maintains an internal lock and a queue. Asynchronous methods (insertAsync, updateOneAsync, …) acquire the lock, perform the work, then release it so the next queued operation can run.

7. Snapshot transactions

When you call collection.startTransaction() the library:

• Deep-clones the current in-memory index
• Creates an overlay for part data (parts are loaded lazily)
• Routes all subsequent inserts/updates/deletes to the overlay

Disk is not touched until you call commit(). rollback() simply discards the overlay; the original files stay exactly as they were.

Only one transaction may be active on a collection at a time.

8. Self-healing / corrupted parts

If a part file cannot be parsed (invalid JSON, truncated write, etc.):

• In errorLevel: 'debug' (default) the library copies the bad file once into a crash/ subdirectory and records the part key in index.corruptedParts.
• Subsequent loads skip that part entirely.
• The rest of the collection continues to work normally.

You can later inspect the crash copies, fix them manually, delete the corruptedParts entry, and call rebuildIndexes().

9. Memory model

At any moment the following live in memory for an open collection:

• The full index.json structure
• Any part files that have been loaded for the current query or write

Parts that are not needed for the current operation are not kept in memory. This keeps the memory footprint proportional to the size of the indexes plus the parts you actually touch, not to the entire collection.

Summary