Sidequest: A Scalable Background Job Processor for Node.js

Introduction

Sidequest is a modern, scalable background job processor specifically designed for Node.js applications. Built with TypeScript, it offers a robust solution for handling long-running tasks, scheduled operations, and asynchronous processes without blocking your main application thread. Sidequest ensures reliable job processing with support for multiple database backends, a comprehensive web dashboard for monitoring, and flexible job lifecycle management, making it ideal for production environments.

Installation

To get started with Sidequest, first install the main package:

npm install sidequest
# or
yarn add sidequest

Please ensure you are using Node.js >= 22.6.0.

Sidequest's main package does not include backend drivers to keep the application minimal. You need to install only the driver you plan to use.

PostgreSQL (recommended)

npm install @sidequest/postgres-backend
# or
yarn add @sidequest/postgres-backend

MySQL

npm install @sidequest/mysql-backend
# or
yarn add @sidequest/mysql-backend

SQLite (default, not recommended for production)

npm install @sidequest/sqlite-backend
# or
yarn add @sidequest/sqlite-backend

MongoDB

npm install @sidequest/mongo-backend
# or
yarn add @sidequest/mongo-backend

Examples

Here's how to integrate Sidequest into your application.

1. Create a Job class

Define your background task logic by extending the Job class:

// jobs/EmailJob.js
import { Job } from "sidequest";

export class EmailJob extends Job {
  async run(to, subject, body) {
    console.log(`Sending email to ${to}: ${subject}`);
    // Your email sending logic here
    return { sent: true, timestamp: new Date() };
  }
}

2. Configure and Start Sidequest

Initialize and start the Sidequest worker. You can configure your preferred backend here:

// app.js
import { Sidequest } from "sidequest";

// Start Sidequest
await Sidequest.start({
  // You can leave the config empty to use the default SQLite backend.
  // Make sure to install the SQLite backend driver if you want to use it.
  backend: {
    driver: "@sidequest/postgres-backend",
    config: "postgres://postgres:postgres@localhost:5432",
  },
});

console.log("Sidequest started! Dashboard: http://localhost:8678");

3. Enqueue Jobs

Once Sidequest is running, you can enqueue jobs from anywhere in your application:

// Somewhere in your application
import { Sidequest } from "sidequest";
import { EmailJob } from "./jobs/EmailJob.js";

// Simple job
await Sidequest.build(EmailJob).enqueue("user@example.com", "Welcome!", "Thanks for signing up!");

Why Use Sidequest?

Sidequest stands out with a comprehensive set of features designed for modern Node.js applications:

• High Performance: Utilizes worker threads for non-blocking job processing, ensuring your application remains responsive.
• Multiple Backends: Supports PostgreSQL, MySQL, SQLite, and MongoDB out of the box, offering flexibility in your data storage choices.
• ESM and CJS Support: Fully compatible with both modern ES Modules and CommonJS.
• TypeScript Support: Natively supports TypeScript jobs, especially with Node.js >= v23.6.0.
• Web Dashboard: Provides a beautiful, responsive dashboard for real-time monitoring of jobs and queues.
• Queue Management: Offers multiple queues with configurable workers and priorities to manage your workload effectively.
• Job Lifecycle Management: Features configurable retry mechanisms with exponential backoff, snooze, and fail options for robust error handling.
• Scheduled Jobs: Allows scheduling jobs to run at specific times or intervals.
• Job Uniqueness: Prevents duplicate jobs with flexible uniqueness constraints.
• CLI Tools: Includes command-line interface tools for database migrations and management.
• Monorepo Architecture: Designed with modular packages for flexible deployment and easier maintenance.

Links

• Documentation: https://docs.sidequestjs.com
• GitHub Repository: https://github.com/sidequestjs/sidequest
• Issue Tracker: https://github.com/sidequestjs/sidequest/issues
• Discussions: https://github.com/sidequestjs/sidequest/discussions