Huey: A Lightweight Python Task Queue for Redis, SQLite, and More

Introduction

Huey is a lightweight and simple task queue written in Python, designed to handle asynchronous tasks efficiently. It offers a clean and straightforward API, making it easy to integrate into your Python applications. Huey supports various storage backends, including Redis, SQLite, file-system, or even in-memory, providing flexibility for different project requirements.

Key features of Huey include multi-process, multi-thread, or greenlet task execution models, allowing you to choose the best concurrency strategy for your workload. It enables scheduling tasks to execute at a given time or after a delay, and supports recurring tasks similar to a crontab. Huey also provides mechanisms for automatically retrying failed tasks, task prioritization, result storage, expiration, and locking, along with task pipelines and chains for complex workflows.

Installation

To get started with Huey, you can install it using pip:

pip install huey

Depending on your chosen storage backend, you might need to install additional dependencies, for example, redis for Redis storage.

Examples

Here are some basic examples demonstrating how to define tasks, call them, and run the Huey consumer.

Defining and Calling a Task:

First, initialize Huey with your preferred storage backend, such as Redis:

from huey import RedisHuey, crontab

huey = RedisHuey('my-app', host='redis.myapp.com')

@huey.task()
def add_numbers(a, b):
    """A simple task to add two numbers."""
    return a + b

@huey.task(retries=2, retry_delay=60)
def flaky_task(url):
    """This task might fail and will be retried up to 2 times with a 60s delay."""
    # Assume this_might_fail is a function that can raise an exception
    return this_might_fail(url)

@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
    """A task that runs every night at 3 AM."""
    sync_all_data() # Assume sync_all_data performs a backup

Calling a task-decorated function enqueues the function call for execution by the consumer. A special result handle is returned immediately, which can be used to fetch the result once the task is finished:

# Assuming 'add_numbers' is defined in a module like 'demo.py'
from demo import add_numbers

res = add_numbers(1, 2)
print(f"Task enqueued, result handle: {res}")

# To get the result (this will block until the task is complete)
result = res()
print(f"Task result: {result}") # Output: Task result: 3

Scheduling Tasks for Future Execution:

Tasks can also be scheduled to run in the future, either after a delay or at a specific time:

# Schedule 'add_numbers' to run in approximately 10 seconds
res_scheduled = add_numbers.schedule((2, 3), delay=10)
print(f"Task scheduled, result handle: {res_scheduled}")

# Block until the scheduled task finishes (after ~10 seconds)
scheduled_result = res_scheduled(blocking=True)
print(f"Scheduled task result: {scheduled_result}") # Output: Scheduled task result: 5

Running the Huey Consumer:

The Huey consumer is responsible for picking up and executing the enqueued tasks. You can run it with different worker models:

To run the consumer with four worker processes:

huey_consumer.py my_app.huey -k process -w 4

To run the consumer with a single worker thread (default):

huey_consumer.py my_app.huey

For IO-bound workloads, you might prefer greenlets due to their lightweight nature:

huey_consumer.py my_app.huey -k greenlet -w 32

Why Use Huey?

Huey stands out as an excellent choice for managing asynchronous tasks in Python due to several compelling reasons:

Lightweight and Simple API: It is designed to be a lightweight alternative to more complex task queues, offering a clean and intuitive API that makes task definition and management straightforward.
Flexible Storage Options: With built-in support for Redis, SQLite, file-system, and in-memory storage, Huey provides versatility, allowing you to choose the backend that best fits your project's infrastructure and scale.
Robust Feature Set: Beyond basic task execution, Huey includes advanced features like scheduled and recurring tasks, automatic retries for transient failures, task prioritization, result storage, expiration, and locking, covering a wide range of asynchronous processing needs.
Scalable Concurrency Models: It supports various execution models, including multi-process, multi-thread, and greenlet workers, enabling you to optimize performance based on the nature of your tasks (CPU-bound vs. IO-bound).
Pythonic Design: Written entirely in Python, Huey integrates seamlessly into existing Python projects, leveraging decorators for task definition and providing a familiar development experience.