CallFS: High-Performance, Distributed REST API Filesystem in Go

Introduction

CallFS is an ultra-lightweight, high-performance REST API filesystem engineered to deliver precise Linux filesystem semantics across diverse storage backends. Written in Go, it seamlessly integrates with local filesystems, Amazon S3, and other S3-compatible services, as well as distributed peer-to-peer networks. Designed for speed, reliability, and horizontal scalability, CallFS is ideal for applications requiring a flexible and powerful storage solution.

Key features include multi-backend storage, a distributed architecture for horizontal scaling, cross-server operations with conflict detection, and a high-performance REST API. It also offers secure, ephemeral download links, Redis-based distributed locking for data consistency, a PostgreSQL metadata store, comprehensive security with TLS and API key authentication, and first-class observability through structured logging and Prometheus metrics. Its zero-copy I/O ensures efficient handling of large files with a low memory footprint.

Installation

Getting CallFS up and running involves a few straightforward steps, primarily setting up its dependencies and then building and running the server.

Prerequisites

• Go: Version 1.21 or later.
• PostgreSQL: A running instance for metadata storage.
• Redis: A running instance for the distributed lock manager.
• Docker & Docker Compose: Recommended for easily managing PostgreSQL and Redis.
• A valid config.yaml file (refer to config.yaml.example in the repository).

Running the Server

1. Start dependent services (PostgreSQL and Redis):

   docker-compose up -d postgres redis
   

2. Build the CallFS binary:

   go build -o callfs ./cmd/main.go
   

3. Run the server, specifying your configuration file:

   ./callfs server --config /path/to/your/config.yaml
   

Examples

CallFS exposes a clean REST API for all filesystem operations. Most endpoints require Bearer <api-key> authentication.

• Download a file or list a directory's contents: GET /v1/files/{path}
• Create a new file or directory: POST /v1/files/{path}
• Upload or update a file's content: PUT /v1/files/{path}
• Generate a secure, single-use download link for a file: POST /v1/links/generate
• Download a file using a single-use token (no authentication required): GET /v1/download/{token}

Why Use CallFS?

CallFS stands out as a robust solution for modern storage challenges due to several compelling reasons:

• Scalability and Distribution: Its distributed architecture allows for horizontal scaling, making it suitable for high-demand environments and microservices.
• Backend Flexibility: Support for local filesystems, S3, and other object storage solutions provides unparalleled flexibility in choosing your storage infrastructure.
• High Performance: Built with Go, CallFS is designed for low latency and high throughput, crucial for performance-sensitive applications.
• Linux Semantics over REST: It brings the familiarity and precision of Linux filesystem operations to a RESTful API, simplifying integration for developers.
• Security and Observability: Comprehensive security features, including TLS and API key authentication, combined with extensive Prometheus metrics and structured logging, ensure operational insight and data protection.

Links

• GitHub Repository: ebogdum/callfs
• Full Documentation: CallFS Documentation