{"name":"MRQ: A Distributed Python Task Queue with Redis and Gevent","description":"MRQ is a distributed worker task queue written in Python, leveraging Redis and gevent for efficient task management. It aims to combine the simplicity of RQ with the performance capabilities of Celery, offering a robust solution for handling heterogeneous jobs. Developed by Pricing Assistant, it provides a comprehensive dashboard, per-job logs, and flexible job management features.","github":"https://github.com/pricingassistant/mrq","url":"https://osrepos.com/repo/pricingassistant-mrq","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/pricingassistant-mrq","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/pricingassistant-mrq.md","json":"https://osrepos.com/repo/pricingassistant-mrq.json","topics":["Python","Task Queue","Redis","Gevent","Distributed Systems","Worker","Asynchronous","MongoDB"],"keywords":["Python","Task Queue","Redis","Gevent","Distributed Systems","Worker","Asynchronous","MongoDB"],"stars":null,"summary":"MRQ is a distributed worker task queue written in Python, leveraging Redis and gevent for efficient task management. It aims to combine the simplicity of RQ with the performance capabilities of Celery, offering a robust solution for handling heterogeneous jobs. Developed by Pricing Assistant, it provides a comprehensive dashboard, per-job logs, and flexible job management features.","content":"## Introduction\n\nMRQ, or Mr. Queue, is a powerful and opinionated distributed worker task queue built in Python. It utilizes Redis for queue management and gevent for concurrent task execution, aiming to strike a balance between the simplicity of RQ and the high performance of Celery. Originally developed at Pricing Assistant, MRQ is designed to handle a wide range of jobs, from I/O-bound to CPU-bound tasks, efficiently and reliably.\n\n## Installation\n\nGetting started with MRQ is straightforward. You'll need Redis and MongoDB installed and running as dependencies.\n\n1.  **Install Dependencies**: Ensure you have Redis and MongoDB running.\n2.  **Install MRQ**: Use pip to install the library.\n\n    bash\n    pip install mrq\n    \n\n3.  **Start Servers**: Launch your MongoDB and Redis instances.\n\n    bash\n    mongod &\n    redis-server &\n    \n\n## Examples\n\nLet's walk through a quick example to demonstrate MRQ's capabilities.\n\n### Write your first task\n\nCreate a directory and a `tasks.py` file with a simple task:\n\npython\nfrom mrq.task import Task\nimport urllib.request # Changed from urllib2 for Python 3 compatibility\n\n\nclass Fetch(Task):\n\n    def run(self, params):\n\n        with urllib.request.urlopen(params[\"url\"]) as f:\n          t = f.read()\n          return len(t)\n\n\n### Run it synchronously\n\nYou can execute tasks directly using `mrq-run`:\n\nbash\nmrq-run tasks.Fetch url http://www.google.com\n\n\nThis will execute the task immediately and print its result.\n\n### Run it asynchronously\n\nTo queue tasks for asynchronous processing, use `mrq-run` with a queue name:\n\nbash\nmrq-run --queue fetches tasks.Fetch url http://www.google.com && \\\n  mrq-run --queue fetches tasks.Fetch url http://www.yahoo.com && \\\n  mrq-run --queue fetches tasks.Fetch url http://www.wordpress.com\n\n\nThese commands will add three tasks to the `fetches` queue. To process them, start the MRQ dashboard and a worker:\n\n1.  **Start Dashboard**: `mrq-dashboard &` (then visit `http://localhost:5555/#jobs`)\n2.  **Start Worker**: `mrq-worker fetches`\n\nThe worker will pick up and execute the queued jobs in parallel, and you can monitor their status through the dashboard.\n\n## Why Use MRQ?\n\nMRQ offers a rich set of features that make it a compelling choice for distributed task management:\n\n*   **Simple Code**: Designed for ease of understanding and extension, avoiding the complexity found in some other queues.\n*   **Great Dashboard**: Provides comprehensive visibility and control over queued jobs, current jobs, and worker status.\n*   **Per-Job Logs**: Access individual log outputs for each task directly from the dashboard.\n*   **Gevent Worker**: Optimizes throughput for I/O-bound tasks by running them in parallel within the same UNIX process.\n*   **Supervisord Integration**: Easily distribute CPU-bound tasks across multiple UNIX processes.\n*   **Job Management**: Offers robust capabilities to retry, requeue, or cancel jobs programmatically or via the dashboard.\n*   **Performance**: Includes features like bulk job queuing and easy job profiling.\n*   **Easy Configuration**: Every aspect of MRQ is configurable through command-line flags or a dedicated configuration file.\n*   **Job Routing**: Supports defining default queues, timeouts, and TTL values for jobs, similar to Celery.\n*   **Built-in Scheduler**: Schedule tasks by interval or specific times of the day.\n*   **Strategies**: Choose between sequential or parallel dequeue orders, and a burst mode for batch jobs.\n*   **Subqueues**: Simple command-line patterns for dequeuing multiple sub-queues with auto-discovery.\n*   **Thorough Testing**: Robustly tested against edge cases like worker interrupts and Redis failures using Docker.\n*   **Greenlet Tracing**: Debug CPU-intensive jobs by seeing time spent in each greenlet.\n*   **Integrated Memory Leak Debugger**: Identify and track down memory leaks within jobs using `objgraph`.\n\n## Links\n\n*   **Official Documentation**: [http://mrq.readthedocs.org/en/latest/](http://mrq.readthedocs.org/en/latest/){:target=\"_blank\"}\n*   **GitHub Repository**: [https://github.com/pricingassistant/mrq](https://github.com/pricingassistant/mrq){:target=\"_blank\"}","metrics":{"detailViews":3,"githubClicks":0},"dates":{"published":null,"modified":"2026-08-05T11:21:50.000Z"}}