cross-org / kv

A fast, lightweight, powerful and cross-platform key-value database for Node.js, Deno, and Bun.

@cross/kv

A fast, lightweight, powerful and cross-platform key-value database for Node.js, Deno, and Bun.

Library usage:

import { KV } from "@cross/kv";

// Create an instance
const db = new KV();
await db.open("data/mydatabase.db");

// Listen for new interests of any user
db.watch(["users", {}, "interests"], (data) => {
  console.log(data);
});

// Store some values/documents indexed by users.<id>.<category>
await db.set(["users", 1, "contact"], {
  name: "Bob",
});
await db.set(["users", 1, "interests"], {
  description: "Fishing",
});

// Display all contact information connected to users with id <= 10
console.log(await db.listAll(["users", { to: 10 }, "contact"]));

db.close();

Command line client ckv:

# Install
# deno install -frgA --name ckv jsr:@cross/kv/cli

# Run without installing
deno run -A jsr:@cross/kv/cli

> open my.db
Success [10.30 ms]

> set:json my.key {"hello":"kv"}
Success [31.70 ms]

> get my.key

Key:             ["my","key"]
Operation:       SET (1)
Timestamp:       2024-05-26T19:49:49.471Z
Hash:            abdf6eb7a3fe04af920f31a599ce0cc069d29041

{ hello: "kv" }

Success [7.74 ms]

Features

• Cross-Platform & Multi-Process: Built with pure TypeScript for seamless compatibility across Node.js, Deno, and Bun, with built-in support for concurrent access by multiple processes.
• Powerful: Supports hierarchical keys, flexible mid-key range queries, and real-time data change notifications through .watch().
• Simple and Fast: Lightweight and performant storage with an in-memory index for efficient data retrieval.
• Durable: Ensures data integrity and reliability by storing each database as a single, append-only transaction ledger.
• Type-Safe: Leverages TypeScript generics for enhanced type safety when setting and retrieving values.
• Atomic Transactions: Guarantees data consistency by grouping multiple operations into indivisible units, which also improves performance.
• Flexible: Store any serializable JavaScript object (except functions and WeakMaps), and customize synchronization behavior to optimize for your specific use case.

Installation

Library:

# Using npm
npx jsr add @cross/kv

# Using Deno
deno add @cross/kv

# Using bun
bunx jsr add @cross/kv

Command line client ckv:

# Using Deno
deno install -frA --name ckv jsr:@cross/kv/cli

API Documentation

Methods

• KV(options) - Main class. Options are optional.
  • async open(filepath, createIfMissing = true) - OOpens the KV store at the specified file path, creating it if it doesn't exist (default behavior).
  • async set<T>(key, value) - Stores a value associated with the given key.
  • async delete(key) - Removes the key-value pair identified by the key.
  • async get<T>(key) - Retrieves the value associated with the specified key. Returns null if the key does not exist.
  • async *iterate<T>(query, limit, reverse) - Asynchronously iterates over the latest values matching the query
  • async listAll<T>(query, limit, reverse) - Retrieves all latest values matching the query as an array.
  • async *scan<T>(query, limit, reverse) - Asynchronously iterates over the transaction history (all set and delete operations) for keys matching the query. Optionally recurses into subkeys and fetches the associated data.
  • listKeys(query) - Returns an array of all keys matching the given query.
  • async sync() - Manually synchronizes the in-memory index with the on-disk data store.
  • watch<T>(query, callback, recursive): void - Registers a callback to be invoked whenever a matching transaction (set or delete) is added.
  • unwatch<T>(query, callback): void - Unregisters a previously registered watch handler.
  • beginTransaction() - Starts an atomic transaction, ensuring data consistency for multiple operations.
  • async endTransaction() - Commits all changes made within the transaction, or rolls back if errors occur.
  • async vacuum() - Optimizes storage by removing redundant transaction history, retaining only the latest value for each key.
  • on(eventName, eventData) - Subscribes to events like sync, watchdogError, or closing to get notified of specific occurrences.
  • isOpen() - Returns true if the database is open and ready for operations.
  • async close() - Closes the KV store, ensuring resources are released.

Keys

• Arrays of strings or numbers
• First element in a key must be a string.
• Strings must only contain alphanumeric characters, hyphens, underscores or "@".

Examples keys

["users", 123]
["products", "category", { from: 10, to: 20 }]

Values

Values (or documents) are the data you store in the database. They can be any JavaScript primitive or a complex object containing CBOR-serializable types, including:

• Numbers: (e.g., 12345)
• Strings: (e.g., "Hello, world!")
• Booleans: (e.g., true)
• Arrays: (e.g., [1, 2, 3])
• Objects: (e.g., { "name": "Alice", "age": 30 })
• Maps: (e.g., new Map([["key1", "value1"], ["key2", "value2"]]))
• Sets: (e.g., new Set([1, 2, 3]))
• Dates: (e.g., new Date())
• null

Queries

Queries are similar to keys but with additional support for ranges, specified as objects like { from: 5, to: 20 } or { from: "a", to: "l" }. An empty range ({}) matches any document.

Example queries

// All users
["users"]       
// Specific user with ID 123          
["users", 123]            
// All products in any category
["products", "category"]  
// Products in category with an id up to 20
["products", "category", { to: 20 }] 
 // Sub document "specification" of products in category 10 to 20
["products", "category", { from: 10, to: 20 }, "specifications"]
// Sub-document "author" of any book
["products", "book", {}, "author"]

Options

You can customize the behavior of the KV store using the following options when creating a new KV instance:

const db = new KV({
  autoSync: true, // Enable/disable automatic synchronization (default: true)
  syncIntervalMs: 1000, // Synchronization interval in milliseconds (default: 1000)
  ledgerCacheSize: 100, // Ledger cache size in megabytes (default: 100)
  disableIndex: false, // Disable in-memory index for faster loading but limited functionality (default: false)
});

Explanations:

• autoSync (boolean):
  • true (default): The in-memory index is automatically synchronized with the on-disk ledger in the background. This is recommended for multi-process scenarios.
  • false: Automatic synchronization is disabled. You'll need to call db.sync() manually to keep the index up-to-date with other processes. This might be suitable for single-process scenarios, or where you fine grained control over performance.
• syncIntervalMs (number): Specifies the interval (in milliseconds) between automatic synchronization operations if autoSync is enabled. A shorter interval provides more up-to-date data but may introduce more overhead.
• ledgerCacheSize (number): Sets the maximum amount of ledger data (in megabytes) to cache in memory. A larger cache can improve read performance but consumes more memory. (Default 100).
• disableIndex (boolean):
  • false (default): The in-memory index is enabled, allowing for efficient data retrieval and complex queries.
  • true: The in-memory index is disabled, resulting in faster loading times but preventing the use of get, iterate, scan, and list. This is suitable only when you need to append data to the ledger and don't require efficient querying.

Concurrency

cross/kv has a built-in mechanism for synchronizing the in-memory index with the transaction ledger, allowing multiple processes to work with the same database simultaneously.

Due to the append-only design of the ledger, each process can update its internal state by reading all new transactions appended since the last processed transaction.

Single-Process Synchronization

In single-process scenarios, explicit synchronization is often unnecessary. You can disable automatic synchronization by setting the autoSync option to false, eliminating automated .sync() calls. This can potentially improve performance when only one process accesses the database.

Multi-Process Synchronisation

In multi-process scenarios, synchronization is essential for maintaining data consistency. cross/kv offers automatic index synchronization upon each data insertion and at a configurable interval (default: 1000ms). Customizing this interval providing fine-grained control over the trade-off between consistency and performance. For strict consistency guarantees, you can manually call .sync() before reading data.

await kv.sync(); // Ensure the most up-to-date data
const result = await kv.get(["my", "key"]); // Now read with confidence

Monitoring Synchronization Events

You can subscribe to the sync event to receive notifications about synchronization results and potential errors:

const kvStore = new KV();
await kvStore.open("db/mydatabase.db");

kvStore.on("sync", (eventData) => {
  switch (eventData.result) {
    case "ready": // No new updates
    case "success": // Synchronization successful, new transactions added
    case "ledgerInvalidated": // Ledger recreated, database reopened and index resynchronized
    case "error": // An error occurred during synchronization
    default:
      // Handle unexpected eventData.result values if needed
  }
});

Contributing

Contributions are welcome! Feel free to open issues or submit pull requests.

The task deno task check runs all tests, and is a good pre-commit check. deno task check-coverage do require genhtml available through the lcov package in most distributions.

License

MIT License