huey: Queue and Schedule Python Tasks

Overview

Huey lets Python applications enqueue functions for background execution and retrieve their results later. It addresses the need to move slow or scheduled work out of the main application flow without requiring a large task-queue framework.

It can use Redis, PostgreSQL, SQLite, filesystem, or in-memory storage, and supports Django integration. Workers can run as processes, threads, or greenlets, so the execution model can be chosen to suit the workload.

Key Features

• Enqueue Python functions using task decorators and receive a result handle.
• Schedule tasks for a later time or set them to run periodically.
• Retry failed tasks, with configurable retry counts and delays.
• Choose Redis, PostgreSQL, SQLite, filesystem, or in-memory storage.
• Run workers using processes, threads, or greenlets.
• Support task prioritization, result storage, expiration, locking, rate limits, and timeouts.
• Build task pipelines, chains, groups, and chords.
• Integrate with Django, including optional admin visibility and management.

Use Cases

• Python web application developers can move slow operations, such as backups or other time-consuming work, out of user-facing requests.
• Teams can run recurring maintenance jobs, such as scheduled synchronization, using periodic tasks.
• Applications that need to retry unreliable operations can configure automatic retries and delays.
• Django developers can use Huey for background jobs and optionally manage task information through Django admin.
• Projects with different infrastructure preferences can select a supported storage backend rather than relying solely on Redis.

Project Facts

• Language: Python
• License: MIT
• Stars: 6k
• Forks: 403
• Topics: dank, django, python, queue, redis, task-queue
• Archived: No

Getting Started

Run a consumer for your application’s Huey instance:

huey_consumer my_app.huey

See the README and documentation for setup, configuration, and examples.

Considerations

Huey requires a running consumer to execute queued work. Although the project describes itself as having zero dependencies, Redis-like brokers require redis-py and PostgreSQL requires psycopg. In-memory storage is available, but the appropriate backend depends on the application’s storage needs.