filedb: A Disk-Based Key-Value Store Inspired by Bitcask in Zig

Introduction

filedb is a key-value store implemented in Zig, inspired by the design principles of Bitcask by Riak. It's engineered to provide a high-performance, disk-based solution for storing and retrieving data. The system manages record metadata in a log-structured hashtable while storing actual data in append-only disk files, ensuring efficient operations and data integrity through mechanisms like file rotation, compaction, and periodic syncing.

Installation

As filedb is a Zig library, integration typically involves adding it as a dependency to your Zig project. For those looking to interact with it as a service, filedb also offers a Redis-compatible client. Detailed build and installation instructions for the library or the Redis client can be found directly within the project's GitHub repository.

Examples

filedb provides a clear API for common key-value operations:

const filedb = @import("filedb");
const std = @import("std");

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    const allocator = gpa.allocator();
    defer _ = gpa.deinit();

    var db = try filedb.init(allocator, null);
    defer db.deinit();

    // Insert a key-value pair
    try db.put("hello", "world");

    // Retrieve a value
    const value = try db.get("hello");
    std.debug.print("Retrieved: {s}\n", .{value}); // Output: Retrieved: world

    // Delete a key-value pair
    try db.delete("hello");

    // Attempt to retrieve after deletion
    const deleted_value = db.get("hello");
    if (deleted_value) |v| {
        std.debug.print("Still found: {s}\n", .{v});
    } else {
        std.debug.print("Key not found after deletion.\n", .{}); // Output: Key not found after deletion.
    }

    // List all keys (example, requires allocation)
    var keys = try db.list(allocator);
    defer allocator.free(keys);
    std.debug.print("Keys in DB: {any}\n", .{keys});
}

For users interacting with the Redis-compatible client, the experience is familiar:

127.0.0.1:6379> PING
PONG
127.0.0.1:6379> get abcd
(nil)
127.0.0.1:6379> set abcd def
OK
127.0.0.1:6379> get abcd
"def"

Why Use filedb?

filedb offers several compelling advantages for developers seeking an efficient key-value store:

• O(1) Record Fetching: Thanks to its metadata structure, retrieving records is an O(1) operation, ensuring fast access times.
• Constant-Sized Metadata: The in-memory metadata store maintains a constant size regardless of the value's size, optimizing memory usage.
• High Throughput: By utilizing an append-only file for insertions, filedb achieves high write throughput.
• Redis Compatibility: The availability of a Redis-compatible client allows for easy integration into existing systems and familiar interaction patterns.

Links

• GitHub Repository: rajivharlalka/filedb
• In-depth Internals: FileDb Internals
• Bitcask Paper: Bitcask Paper by Riak
• Go Implementation of Bitcask: barreldb
• Zig Resources:
  • Basic MetaProgramming in Zig
  • Zig Book
  • Zig Standard Library
  • Zig Help