Repository navigation
DOC-7314 Add RQ and Dramatiq integration pages #4602
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
andy-stark-redis
merged 1 commit into
main
from
DOC-7314-add-rq-and-dramatiq-integration-docs
Nov 17, 2026
+261
−2
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| --- | ||
| LinkTitle: Dramatiq | ||
| Title: Redis with Dramatiq | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - integrate | ||
| - oss | ||
| - rs | ||
| - rc | ||
| description: Run Python background tasks with Dramatiq and Redis. | ||
| group: library | ||
| hideListLinks: false | ||
| stack: true | ||
| summary: Run Dramatiq actors on Redis with retries, delayed messages, and stored results, and learn how recovery of a killed worker's messages works. | ||
| type: integration | ||
| weight: 16 | ||
| --- | ||
|
|
||
| [Dramatiq](https://dramatiq.io/) is a Python library for background task processing. You | ||
| define *actors*, which are functions that run in worker processes, and send them messages. | ||
| Dramatiq supports Redis and RabbitMQ as brokers. See the | ||
| [job queue use case](/content/develop/use-cases/job-queue/_index.md) for when to choose | ||
| this pattern. | ||
|
|
||
| ## Requirements | ||
|
|
||
| The examples on this page were run with Dramatiq 2.2 and redis-py against Redis 7.2. | ||
| Dramatiq 2.2 requires Python 3.10 or later. Install it with its Redis dependencies: | ||
|
|
||
| ```bash | ||
| pip install "dramatiq[redis]" | ||
| ``` | ||
|
|
||
| You also need a running Redis server. You can run one locally with | ||
| [Redis Open Source](/content/operate/oss_and_stack/install/archive/install-redis/_index.md), | ||
| use [Docker](https://hub.docker.com/_/redis), or connect to | ||
| [Redis Cloud](https://redis.io/cloud/). | ||
|
|
||
| ## Configure the Redis broker | ||
|
|
||
| Create a `RedisBroker` and register it with Dramatiq. To store the return values of | ||
| actors, add the `Results` middleware with a Redis backend. This example puts results in a | ||
| different [logical database](/content/commands/select.md) from the queues: | ||
|
|
||
| ```python | ||
| # tasks.py | ||
| import dramatiq | ||
| from dramatiq.brokers.redis import RedisBroker | ||
| from dramatiq.results import Results | ||
| from dramatiq.results.backends import RedisBackend | ||
|
|
||
| broker = RedisBroker(url="redis://<your-redis-host>:6379/0") | ||
| broker.add_middleware(Results(backend=RedisBackend(url="redis://<your-redis-host>:6379/1"))) | ||
| dramatiq.set_broker(broker) | ||
| ``` | ||
|
|
||
| ## Define actors and send messages | ||
|
|
||
| Decorate a function with `@dramatiq.actor`. Set `store_results=True` to keep its return | ||
| value, and `max_retries` with a backoff range in milliseconds to retry failures: | ||
|
|
||
| ```python | ||
| @dramatiq.actor(store_results=True) | ||
| def add(x, y): | ||
| return x + y | ||
|
|
||
| @dramatiq.actor(max_retries=3, min_backoff=500, max_backoff=2000) | ||
| def send_email(): | ||
| ... # raise an exception to trigger a retry | ||
| ``` | ||
|
|
||
| Call `send()` to queue a message, and `send_with_options()` to set a delay in | ||
| milliseconds. A message object lets you wait for the stored result: | ||
|
|
||
| ```python | ||
| message = add.send(2, 3) | ||
| print(message.get_result(block=True, timeout=10000)) # 5 | ||
|
|
||
| late = add.send_with_options(args=(10, 5), delay=2000) | ||
| print(late.get_result(block=True, timeout=15000)) # 15, after about 3 seconds | ||
| ``` | ||
|
|
||
| Start a worker in a separate terminal, in the directory that contains `tasks.py`: | ||
|
|
||
| ```bash | ||
| dramatiq tasks --processes 1 --threads 2 | ||
| ``` | ||
|
|
||
| In a test run, an actor that raised an error twice logged `Retrying message` for each | ||
| failure and succeeded on the third attempt. | ||
|
|
||
| ## What Dramatiq stores in Redis | ||
|
|
||
| | Key | Type | Contents | | ||
| |-----|------|----------| | ||
| | `dramatiq:default` | list | Internal IDs of the messages that are waiting for a worker (`default` is the queue name). | | ||
| | `dramatiq:default.msgs` | hash | The message payloads, keyed by those internal IDs. | | ||
| | `dramatiq:__heartbeats__` | sorted set | The last heartbeat time of each worker process. | | ||
| | `dramatiq:__acks__.<worker-id>.default` | set | The internal IDs of messages that a worker has taken but not yet acknowledged. | | ||
|
|
||
| Results go to the database that you gave the results backend, one key for each message, | ||
| with a time to live (TTL) of about 600 seconds by default. | ||
|
|
||
| ## What happens to a message when a worker is killed | ||
|
|
||
| A worker takes a message, runs the actor, and acknowledges the message only afterward, so | ||
| a message that a worker was running when it died stays in that worker's `__acks__` set. | ||
| Another worker puts it back on the queue when two conditions hold. The dead worker's | ||
| heartbeat must be older than the broker's `heartbeat_timeout` (60 seconds by default), and | ||
| a worker must run the broker's maintenance, which Dramatiq triggers for a small | ||
| random fraction of the broker's Redis commands (`maintenance_chance`, 1,000 in a million by | ||
| default). | ||
|
|
||
| The test results show how much those defaults matter. After a worker running a | ||
| 100-second job was stopped with `kill -9`: | ||
|
|
||
| - With the defaults, a new worker did not run the job again in 204 seconds. | ||
| - With `RedisBroker(..., heartbeat_timeout=5000, maintenance_chance=1000000)`, a new worker | ||
| started the job 6 seconds after the kill. | ||
|
|
||
| Plan for this delay and for double execution: write actors so that running one twice is | ||
| safe, and set `heartbeat_timeout` to a value that suits how quickly you need recovery. | ||
|
|
||
| ## More information | ||
|
|
||
| - [Dramatiq documentation](https://dramatiq.io/) | ||
| - [Dramatiq on GitHub](https://github.com/Bogdanp/dramatiq) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| --- | ||
| LinkTitle: RQ | ||
| Title: Redis with RQ | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - integrate | ||
| - oss | ||
| - rs | ||
| - rc | ||
| description: Run Python background jobs with RQ (Redis Queue). | ||
| group: library | ||
| hideListLinks: false | ||
| stack: true | ||
| summary: Run RQ background jobs on Redis with retries, scheduled jobs, and result storage, and learn what happens to a running job when a worker is killed. | ||
| type: integration | ||
| weight: 15 | ||
| --- | ||
|
|
||
| [RQ](https://python-rq.org/) (Redis Queue) is a Python library for queueing jobs and | ||
| processing them in the background with workers. Redis is its only data store. RQ jobs are | ||
| ordinary Python functions, so there is no task decorator to add. See the | ||
| [job queue use case](/content/develop/use-cases/job-queue/_index.md) for when to choose | ||
| this pattern. | ||
|
|
||
| ## Requirements | ||
|
|
||
| The examples on this page were run with RQ 3.0 and redis-py against Redis 7.2. RQ 3.0 | ||
| requires Python 3.10 or later and redis-py 5.0.1 or later. Install it with pip: | ||
|
|
||
| ```bash | ||
| pip install rq | ||
| ``` | ||
|
|
||
| You also need a running Redis server. You can run one locally with | ||
| [Redis Open Source](/content/operate/oss_and_stack/install/archive/install-redis/_index.md), | ||
| use [Docker](https://hub.docker.com/_/redis), or connect to | ||
| [Redis Cloud](https://redis.io/cloud/). | ||
|
|
||
| ## Enqueue and run jobs | ||
|
|
||
| Put the function that does the work in a module that both the code that enqueues the job | ||
| and the worker can import. This example uses a `tasks.py` module: | ||
|
|
||
| ```python | ||
| # tasks.py | ||
| def add(x, y): | ||
| return x + y | ||
| ``` | ||
|
|
||
| Create a `Queue` with a redis-py connection and enqueue the function with its arguments: | ||
|
|
||
| ```python | ||
| from redis import Redis | ||
| from rq import Queue | ||
|
|
||
| from tasks import add | ||
|
|
||
| queue = Queue("default", connection=Redis(host="<your-redis-host>", port=6379)) | ||
|
|
||
| job = queue.enqueue(add, 2, 3) | ||
| print(job.get_status()) # JobStatus.QUEUED | ||
| ``` | ||
|
|
||
| Start a worker in a separate terminal, in the directory that contains `tasks.py`: | ||
|
|
||
| ```bash | ||
| rq worker --url redis://<your-redis-host>:6379 --with-scheduler | ||
| ``` | ||
|
|
||
| After the worker runs the job, `job.return_value()` returns `5`. The `--with-scheduler` | ||
| option starts the scheduler that the next two sections need. | ||
|
|
||
| ## Retry and schedule jobs | ||
|
|
||
| Pass a `Retry` object to retry a job that raises an error, with the wait in seconds before | ||
| each attempt. In a test run, a job that failed twice and succeeded on the third attempt | ||
| finished with its return value: | ||
|
|
||
| ```python | ||
| from rq import Retry | ||
|
|
||
| queue.enqueue(flaky, retry=Retry(max=3, interval=[1, 1])) | ||
| ``` | ||
|
|
||
| Use `enqueue_in` to run a job after a delay: | ||
|
|
||
| ```python | ||
| from datetime import timedelta | ||
|
|
||
| queue.enqueue_in(timedelta(seconds=2), add, 10, 5) | ||
| ``` | ||
|
|
||
| A scheduled job runs only while a worker started with `--with-scheduler` is running. | ||
|
|
||
| ## What RQ stores in Redis | ||
|
|
||
| | Key | Type | Contents | | ||
| |-----|------|----------| | ||
| | `rq:queue:<name>` | list | The IDs of jobs that are waiting to run. | | ||
| | `rq:scheduled:<name>` | sorted set | The jobs scheduled with `enqueue_in`, scored by run time. | | ||
| | `rq:job:<job-id>` | hash | The job: `data`, `status`, `enqueued_at`, `started_at`, `ended_at`, `worker_name`, and so on. | | ||
| | `rq:results:<job-id>` | stream | The job's results. | | ||
| | `rq:finished:<name>` | sorted set | The IDs of finished jobs. | | ||
| | `rq:queues` | set | The names of known queues. | | ||
| | `rq:workers` and `rq:worker:<id>` | set, hash | The registered workers and their state. | | ||
|
|
||
| A job's hash expires after its result time to live (TTL). The default is about 500 | ||
| seconds, so finished jobs disappear on their own. Set `result_ttl` when you enqueue a job | ||
| to keep them longer. | ||
|
|
||
| ## What happens to a job when a worker is killed | ||
|
|
||
| RQ doesn't remove a running job from Redis. It records the job in a *started* registry, | ||
| the sorted set `rq:wip:<name>`, and the job's status stays `STARTED`. After a worker and | ||
| its work-horse process were stopped with `kill -9` partway through a job with a | ||
| 6-second timeout: | ||
|
|
||
| - The queue was empty and the started registry held the job, so the job was not lost, but | ||
| it also did not run again. A new worker started right afterward left it in the started | ||
| registry for the 24 seconds that the test ran. | ||
| - After the registry entry's expiry passed, RQ's registry cleanup (`StartedJobRegistry.cleanup()`) | ||
| moved the job to the failed registry with an `AbandonedJobError`. | ||
|
|
||
| In these runs RQ did not re-run an abandoned job. It ends up in the failed registry, and | ||
| your code decides whether to requeue it. Write jobs so that running one twice is safe. | ||
|
|
||
| ## More information | ||
|
|
||
| - [RQ documentation](https://python-rq.org/) | ||
| - [RQ on GitHub](https://github.com/rq/rq) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
\0/