{"name":"Huey: A Lightweight Python Task Queue for Redis, SQLite, and More","description":"Huey is a lightweight and simple Python task queue designed for various storage backends like Redis, SQLite, or in-memory. It provides a clean API for scheduling tasks, retrying failures, managing recurring jobs, and executing them efficiently across multiple processes, threads, or greenlets. This library offers a robust solution for asynchronous task management in Python applications.","github":"https://github.com/coleifer/huey","url":"https://osrepos.com/repo/coleifer-huey","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/coleifer-huey","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/coleifer-huey.md","json":"https://osrepos.com/repo/coleifer-huey.json","topics":["Python","Task Queue","Redis","Asynchronous","Background Jobs","Developer Tools","Queueing"],"keywords":["Python","Task Queue","Redis","Asynchronous","Background Jobs","Developer Tools","Queueing"],"stars":null,"summary":"Huey is a lightweight and simple Python task queue designed for various storage backends like Redis, SQLite, or in-memory. It provides a clean API for scheduling tasks, retrying failures, managing recurring jobs, and executing them efficiently across multiple processes, threads, or greenlets. This library offers a robust solution for asynchronous task management in Python applications.","content":"## Introduction\n\nHuey is a lightweight and simple task queue written in Python, designed to handle asynchronous tasks efficiently. It offers a clean and straightforward API, making it easy to integrate into your Python applications. Huey supports various storage backends, including Redis, SQLite, file-system, or even in-memory, providing flexibility for different project requirements.\n\nKey features of Huey include multi-process, multi-thread, or greenlet task execution models, allowing you to choose the best concurrency strategy for your workload. It enables scheduling tasks to execute at a given time or after a delay, and supports recurring tasks similar to a crontab. Huey also provides mechanisms for automatically retrying failed tasks, task prioritization, result storage, expiration, and locking, along with task pipelines and chains for complex workflows.\n\n## Installation\n\nTo get started with Huey, you can install it using pip:\n\nbash\npip install huey\n\n\nDepending on your chosen storage backend, you might need to install additional dependencies, for example, `redis` for Redis storage.\n\n## Examples\n\nHere are some basic examples demonstrating how to define tasks, call them, and run the Huey consumer.\n\n**Defining and Calling a Task:**\n\nFirst, initialize Huey with your preferred storage backend, such as Redis:\n\npython\nfrom huey import RedisHuey, crontab\n\nhuey = RedisHuey('my-app', host='redis.myapp.com')\n\n@huey.task()\ndef add_numbers(a, b):\n    \"\"\"A simple task to add two numbers.\"\"\"\n    return a + b\n\n@huey.task(retries=2, retry_delay=60)\ndef flaky_task(url):\n    \"\"\"This task might fail and will be retried up to 2 times with a 60s delay.\"\"\"\n    # Assume this_might_fail is a function that can raise an exception\n    return this_might_fail(url)\n\n@huey.periodic_task(crontab(minute='0', hour='3'))\ndef nightly_backup():\n    \"\"\"A task that runs every night at 3 AM.\"\"\"\n    sync_all_data() # Assume sync_all_data performs a backup\n\n\nCalling a `task`-decorated function enqueues the function call for execution by the consumer. A special result handle is returned immediately, which can be used to fetch the result once the task is finished:\n\npython\n# Assuming 'add_numbers' is defined in a module like 'demo.py'\nfrom demo import add_numbers\n\nres = add_numbers(1, 2)\nprint(f\"Task enqueued, result handle: {res}\")\n\n# To get the result (this will block until the task is complete)\nresult = res()\nprint(f\"Task result: {result}\") # Output: Task result: 3\n\n\n**Scheduling Tasks for Future Execution:**\n\nTasks can also be scheduled to run in the future, either after a delay or at a specific time:\n\npython\n# Schedule 'add_numbers' to run in approximately 10 seconds\nres_scheduled = add_numbers.schedule((2, 3), delay=10)\nprint(f\"Task scheduled, result handle: {res_scheduled}\")\n\n# Block until the scheduled task finishes (after ~10 seconds)\nscheduled_result = res_scheduled(blocking=True)\nprint(f\"Scheduled task result: {scheduled_result}\") # Output: Scheduled task result: 5\n\n\n**Running the Huey Consumer:**\n\nThe Huey consumer is responsible for picking up and executing the enqueued tasks. You can run it with different worker models:\n\nTo run the consumer with four worker processes:\n\nbash\nhuey_consumer.py my_app.huey -k process -w 4\n\n\nTo run the consumer with a single worker thread (default):\n\nbash\nhuey_consumer.py my_app.huey\n\n\nFor IO-bound workloads, you might prefer greenlets due to their lightweight nature:\n\nbash\nhuey_consumer.py my_app.huey -k greenlet -w 32\n\n\n## Why Use Huey?\n\nHuey stands out as an excellent choice for managing asynchronous tasks in Python due to several compelling reasons:\n\n*   **Lightweight and Simple API:** It is designed to be a lightweight alternative to more complex task queues, offering a clean and intuitive API that makes task definition and management straightforward.\n*   **Flexible Storage Options:** With built-in support for Redis, SQLite, file-system, and in-memory storage, Huey provides versatility, allowing you to choose the backend that best fits your project's infrastructure and scale.\n*   **Robust Feature Set:** Beyond basic task execution, Huey includes advanced features like scheduled and recurring tasks, automatic retries for transient failures, task prioritization, result storage, expiration, and locking, covering a wide range of asynchronous processing needs.\n*   **Scalable Concurrency Models:** It supports various execution models, including multi-process, multi-thread, and greenlet workers, enabling you to optimize performance based on the nature of your tasks (CPU-bound vs. IO-bound).\n*   **Pythonic Design:** Written entirely in Python, Huey integrates seamlessly into existing Python projects, leveraging decorators for task definition and providing a familiar development experience.\n\n## Links\n\n*   **Huey Documentation:** [https://huey.readthedocs.io/](https://huey.readthedocs.io/){:target=\"_blank\"}\n*   **Huey GitHub Repository:** [https://github.com/coleifer/huey](https://github.com/coleifer/huey){:target=\"_blank\"}","metrics":{"detailViews":5,"githubClicks":3},"dates":{"published":null,"modified":"2026-02-17T12:01:18.000Z"}}