# MRQ: A Distributed Python Task Queue with Redis and Gevent

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/pricingassistant-mrq
Generated for open source discovery and AI-assisted research.

MRQ is a distributed worker task queue written in Python, leveraging Redis and gevent for efficient task management. It aims to combine the simplicity of RQ with the performance capabilities of Celery, offering a robust solution for handling heterogeneous jobs. Developed by Pricing Assistant, it provides a comprehensive dashboard, per-job logs, and flexible job management features.

GitHub: https://github.com/pricingassistant/mrq
OSRepos URL: https://osrepos.com/repo/pricingassistant-mrq

## Summary

MRQ is a distributed worker task queue written in Python, leveraging Redis and gevent for efficient task management. It aims to combine the simplicity of RQ with the performance capabilities of Celery, offering a robust solution for handling heterogeneous jobs. Developed by Pricing Assistant, it provides a comprehensive dashboard, per-job logs, and flexible job management features.

## Topics

- Python
- Task Queue
- Redis
- Gevent
- Distributed Systems
- Worker
- Asynchronous
- MongoDB

## Repository Information

Last analyzed by OSRepos: Wed Aug 05 2026 12:21:50 GMT+0100 (Western European Summer Time)
Detail views: 3
GitHub clicks: 0

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

MRQ, or Mr. Queue, is a powerful and opinionated distributed worker task queue built in Python. It utilizes Redis for queue management and gevent for concurrent task execution, aiming to strike a balance between the simplicity of RQ and the high performance of Celery. Originally developed at Pricing Assistant, MRQ is designed to handle a wide range of jobs, from I/O-bound to CPU-bound tasks, efficiently and reliably.

## Installation

Getting started with MRQ is straightforward. You'll need Redis and MongoDB installed and running as dependencies.

1.  **Install Dependencies**: Ensure you have Redis and MongoDB running.
2.  **Install MRQ**: Use pip to install the library.

    bash
    pip install mrq
    

3.  **Start Servers**: Launch your MongoDB and Redis instances.

    bash
    mongod &
    redis-server &
    

## Examples

Let's walk through a quick example to demonstrate MRQ's capabilities.

### Write your first task

Create a directory and a `tasks.py` file with a simple task:

python
from mrq.task import Task
import urllib.request # Changed from urllib2 for Python 3 compatibility


class Fetch(Task):

    def run(self, params):

        with urllib.request.urlopen(params["url"]) as f:
          t = f.read()
          return len(t)


### Run it synchronously

You can execute tasks directly using `mrq-run`:

bash
mrq-run tasks.Fetch url http://www.google.com


This will execute the task immediately and print its result.

### Run it asynchronously

To queue tasks for asynchronous processing, use `mrq-run` with a queue name:

bash
mrq-run --queue fetches tasks.Fetch url http://www.google.com && \
  mrq-run --queue fetches tasks.Fetch url http://www.yahoo.com && \
  mrq-run --queue fetches tasks.Fetch url http://www.wordpress.com


These commands will add three tasks to the `fetches` queue. To process them, start the MRQ dashboard and a worker:

1.  **Start Dashboard**: `mrq-dashboard &` (then visit `http://localhost:5555/#jobs`)
2.  **Start Worker**: `mrq-worker fetches`

The worker will pick up and execute the queued jobs in parallel, and you can monitor their status through the dashboard.

## Why Use MRQ?

MRQ offers a rich set of features that make it a compelling choice for distributed task management:

*   **Simple Code**: Designed for ease of understanding and extension, avoiding the complexity found in some other queues.
*   **Great Dashboard**: Provides comprehensive visibility and control over queued jobs, current jobs, and worker status.
*   **Per-Job Logs**: Access individual log outputs for each task directly from the dashboard.
*   **Gevent Worker**: Optimizes throughput for I/O-bound tasks by running them in parallel within the same UNIX process.
*   **Supervisord Integration**: Easily distribute CPU-bound tasks across multiple UNIX processes.
*   **Job Management**: Offers robust capabilities to retry, requeue, or cancel jobs programmatically or via the dashboard.
*   **Performance**: Includes features like bulk job queuing and easy job profiling.
*   **Easy Configuration**: Every aspect of MRQ is configurable through command-line flags or a dedicated configuration file.
*   **Job Routing**: Supports defining default queues, timeouts, and TTL values for jobs, similar to Celery.
*   **Built-in Scheduler**: Schedule tasks by interval or specific times of the day.
*   **Strategies**: Choose between sequential or parallel dequeue orders, and a burst mode for batch jobs.
*   **Subqueues**: Simple command-line patterns for dequeuing multiple sub-queues with auto-discovery.
*   **Thorough Testing**: Robustly tested against edge cases like worker interrupts and Redis failures using Docker.
*   **Greenlet Tracing**: Debug CPU-intensive jobs by seeing time spent in each greenlet.
*   **Integrated Memory Leak Debugger**: Identify and track down memory leaks within jobs using `objgraph`.

## Links

*   **Official Documentation**: [http://mrq.readthedocs.org/en/latest/](http://mrq.readthedocs.org/en/latest/){:target="_blank"}
*   **GitHub Repository**: [https://github.com/pricingassistant/mrq](https://github.com/pricingassistant/mrq){:target="_blank"}