MRQ: A Distributed Python Task Queue with Redis and Gevent

Introduction

MRQ, or Mr. Queue, is a powerful and opinionated distributed worker task queue built in Python. It utilizes Redis for queue management and gevent for concurrent task execution, aiming to strike a balance between the simplicity of RQ and the high performance of Celery. Originally developed at Pricing Assistant, MRQ is designed to handle a wide range of jobs, from I/O-bound to CPU-bound tasks, efficiently and reliably.

Installation

Getting started with MRQ is straightforward. You'll need Redis and MongoDB installed and running as dependencies.

1. Install Dependencies: Ensure you have Redis and MongoDB running.
2. Install MRQ: Use pip to install the library.

pip install mrq

3. Start Servers: Launch your MongoDB and Redis instances.

mongod &
redis-server &

Examples

Let's walk through a quick example to demonstrate MRQ's capabilities.

Write your first task

Create a directory and a tasks.py file with a simple task:

from mrq.task import Task
import urllib.request # Changed from urllib2 for Python 3 compatibility


class Fetch(Task):

    def run(self, params):

        with urllib.request.urlopen(params["url"]) as f:
          t = f.read()
          return len(t)

Run it synchronously

You can execute tasks directly using mrq-run:

mrq-run tasks.Fetch url http://www.google.com

This will execute the task immediately and print its result.

Run it asynchronously

To queue tasks for asynchronous processing, use mrq-run with a queue name:

mrq-run --queue fetches tasks.Fetch url http://www.google.com && \
  mrq-run --queue fetches tasks.Fetch url http://www.yahoo.com && \
  mrq-run --queue fetches tasks.Fetch url http://www.wordpress.com

These commands will add three tasks to the fetches queue. To process them, start the MRQ dashboard and a worker:

1. Start Dashboard: mrq-dashboard & (then visit http://localhost:5555/#jobs)
2. Start Worker: mrq-worker fetches

The worker will pick up and execute the queued jobs in parallel, and you can monitor their status through the dashboard.

Why Use MRQ?

MRQ offers a rich set of features that make it a compelling choice for distributed task management:

• Simple Code: Designed for ease of understanding and extension, avoiding the complexity found in some other queues.
• Great Dashboard: Provides comprehensive visibility and control over queued jobs, current jobs, and worker status.
• Per-Job Logs: Access individual log outputs for each task directly from the dashboard.
• Gevent Worker: Optimizes throughput for I/O-bound tasks by running them in parallel within the same UNIX process.
• Supervisord Integration: Easily distribute CPU-bound tasks across multiple UNIX processes.
• Job Management: Offers robust capabilities to retry, requeue, or cancel jobs programmatically or via the dashboard.
• Performance: Includes features like bulk job queuing and easy job profiling.
• Easy Configuration: Every aspect of MRQ is configurable through command-line flags or a dedicated configuration file.
• Job Routing: Supports defining default queues, timeouts, and TTL values for jobs, similar to Celery.
• Built-in Scheduler: Schedule tasks by interval or specific times of the day.
• Strategies: Choose between sequential or parallel dequeue orders, and a burst mode for batch jobs.
• Subqueues: Simple command-line patterns for dequeuing multiple sub-queues with auto-discovery.
• Thorough Testing: Robustly tested against edge cases like worker interrupts and Redis failures using Docker.
• Greenlet Tracing: Debug CPU-intensive jobs by seeing time spent in each greenlet.
• Integrated Memory Leak Debugger: Identify and track down memory leaks within jobs using objgraph.

Links

• Official Documentation: http://mrq.readthedocs.org/en/latest/
• GitHub Repository: https://github.com/pricingassistant/mrq