Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions content/develop/use-cases/job-queue/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,8 @@ Redis provides the following features that make it a good fit for background job
The following libraries implement reliable job-queue patterns on Redis:

- **Python**: [Celery](/content/integrate/celery/_index.md),
[RQ](https://python-rq.org/),
[Dramatiq](https://dramatiq.io/)
[RQ](/content/integrate/rq/_index.md),
[Dramatiq](/content/integrate/dramatiq/_index.md)
- **Ruby**: [Sidekiq](/content/integrate/sidekiq/_index.md),
[Resque](https://github.com/resque/resque)
- **Node.js**: [Bull](https://github.com/OptimalBits/bull),
Expand Down
128 changes: 128 additions & 0 deletions content/integrate/dramatiq/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
---
LinkTitle: Dramatiq
Title: Redis with Dramatiq
alwaysopen: false
categories:
- docs
- integrate
- oss
- rs
- rc
description: Run Python background tasks with Dramatiq and Redis.
group: library
hideListLinks: false
stack: true
summary: Run Dramatiq actors on Redis with retries, delayed messages, and stored results, and learn how recovery of a killed worker's messages works.
type: integration
weight: 16
---

[Dramatiq](https://dramatiq.io/) is a Python library for background task processing. You
define *actors*, which are functions that run in worker processes, and send them messages.
Dramatiq supports Redis and RabbitMQ as brokers. See the
[job queue use case](/content/develop/use-cases/job-queue/_index.md) for when to choose
this pattern.

## Requirements

The examples on this page were run with Dramatiq 2.2 and redis-py against Redis 7.2.
Dramatiq 2.2 requires Python 3.10 or later. Install it with its Redis dependencies:

```bash
pip install "dramatiq[redis]"
```

You also need a running Redis server. You can run one locally with
[Redis Open Source](/content/operate/oss_and_stack/install/archive/install-redis/_index.md),
use [Docker](https://hub.docker.com/_/redis), or connect to
[Redis Cloud](https://redis.io/cloud/).

## Configure the Redis broker

Create a `RedisBroker` and register it with Dramatiq. To store the return values of
actors, add the `Results` middleware with a Redis backend. This example puts results in a
different [logical database](/content/commands/select.md) from the queues:

```python
# tasks.py
import dramatiq
from dramatiq.brokers.redis import RedisBroker
from dramatiq.results import Results
from dramatiq.results.backends import RedisBackend

broker = RedisBroker(url="redis://<your-redis-host>:6379/0")
broker.add_middleware(Results(backend=RedisBackend(url="redis://<your-redis-host>:6379/1")))
dramatiq.set_broker(broker)
```

## Define actors and send messages

Decorate a function with `@dramatiq.actor`. Set `store_results=True` to keep its return
value, and `max_retries` with a backoff range in milliseconds to retry failures:

```python
@dramatiq.actor(store_results=True)
def add(x, y):
return x + y

@dramatiq.actor(max_retries=3, min_backoff=500, max_backoff=2000)
def send_email():
... # raise an exception to trigger a retry
```

Call `send()` to queue a message, and `send_with_options()` to set a delay in
milliseconds. A message object lets you wait for the stored result:

```python
message = add.send(2, 3)
print(message.get_result(block=True, timeout=10000)) # 5

late = add.send_with_options(args=(10, 5), delay=2000)
print(late.get_result(block=True, timeout=15000)) # 15, after about 3 seconds
```

Start a worker in a separate terminal, in the directory that contains `tasks.py`:

```bash
dramatiq tasks --processes 1 --threads 2
```

In a test run, an actor that raised an error twice logged `Retrying message` for each
failure and succeeded on the third attempt.

## What Dramatiq stores in Redis

| Key | Type | Contents |
|-----|------|----------|
| `dramatiq:default` | list | Internal IDs of the messages that are waiting for a worker (`default` is the queue name). |
| `dramatiq:default.msgs` | hash | The message payloads, keyed by those internal IDs. |
| `dramatiq:__heartbeats__` | sorted set | The last heartbeat time of each worker process. |
| `dramatiq:__acks__.<worker-id>.default` | set | The internal IDs of messages that a worker has taken but not yet acknowledged. |

Results go to the database that you gave the results backend, one key for each message,
with a time to live (TTL) of about 600 seconds by default.

## What happens to a message when a worker is killed

A worker takes a message, runs the actor, and acknowledges the message only afterward, so
a message that a worker was running when it died stays in that worker's `__acks__` set.
Another worker puts it back on the queue when two conditions hold. The dead worker's
heartbeat must be older than the broker's `heartbeat_timeout` (60 seconds by default), and
a worker must run the broker's maintenance, which Dramatiq triggers for a small
random fraction of the broker's Redis commands (`maintenance_chance`, 1,000 in a million by
default).

The test results show how much those defaults matter. After a worker running a
100-second job was stopped with `kill -9`:

- With the defaults, a new worker did not run the job again in 204 seconds.
- With `RedisBroker(..., heartbeat_timeout=5000, maintenance_chance=1000000)`, a new worker
started the job 6 seconds after the kill.

Plan for this delay and for double execution: write actors so that running one twice is
safe, and set `heartbeat_timeout` to a value that suits how quickly you need recovery.

## More information

- [Dramatiq documentation](https://dramatiq.io/)
- [Dramatiq on GitHub](https://github.com/Bogdanp/dramatiq)
131 changes: 131 additions & 0 deletions content/integrate/rq/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
---
LinkTitle: RQ
Title: Redis with RQ

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\0/

alwaysopen: false
categories:
- docs
- integrate
- oss
- rs
- rc
description: Run Python background jobs with RQ (Redis Queue).
group: library
hideListLinks: false
stack: true
summary: Run RQ background jobs on Redis with retries, scheduled jobs, and result storage, and learn what happens to a running job when a worker is killed.
type: integration
weight: 15
---

[RQ](https://python-rq.org/) (Redis Queue) is a Python library for queueing jobs and
processing them in the background with workers. Redis is its only data store. RQ jobs are
ordinary Python functions, so there is no task decorator to add. See the
[job queue use case](/content/develop/use-cases/job-queue/_index.md) for when to choose
this pattern.

## Requirements

The examples on this page were run with RQ 3.0 and redis-py against Redis 7.2. RQ 3.0
requires Python 3.10 or later and redis-py 5.0.1 or later. Install it with pip:

```bash
pip install rq
```

You also need a running Redis server. You can run one locally with
[Redis Open Source](/content/operate/oss_and_stack/install/archive/install-redis/_index.md),
use [Docker](https://hub.docker.com/_/redis), or connect to
[Redis Cloud](https://redis.io/cloud/).

## Enqueue and run jobs

Put the function that does the work in a module that both the code that enqueues the job
and the worker can import. This example uses a `tasks.py` module:

```python
# tasks.py
def add(x, y):
return x + y
```

Create a `Queue` with a redis-py connection and enqueue the function with its arguments:

```python
from redis import Redis
from rq import Queue

from tasks import add

queue = Queue("default", connection=Redis(host="<your-redis-host>", port=6379))

job = queue.enqueue(add, 2, 3)
print(job.get_status()) # JobStatus.QUEUED
```

Start a worker in a separate terminal, in the directory that contains `tasks.py`:

```bash
rq worker --url redis://<your-redis-host>:6379 --with-scheduler
```

After the worker runs the job, `job.return_value()` returns `5`. The `--with-scheduler`
option starts the scheduler that the next two sections need.

## Retry and schedule jobs

Pass a `Retry` object to retry a job that raises an error, with the wait in seconds before
each attempt. In a test run, a job that failed twice and succeeded on the third attempt
finished with its return value:

```python
from rq import Retry

queue.enqueue(flaky, retry=Retry(max=3, interval=[1, 1]))
```

Use `enqueue_in` to run a job after a delay:

```python
from datetime import timedelta

queue.enqueue_in(timedelta(seconds=2), add, 10, 5)
```

A scheduled job runs only while a worker started with `--with-scheduler` is running.

## What RQ stores in Redis

| Key | Type | Contents |
|-----|------|----------|
| `rq:queue:<name>` | list | The IDs of jobs that are waiting to run. |
| `rq:scheduled:<name>` | sorted set | The jobs scheduled with `enqueue_in`, scored by run time. |
| `rq:job:<job-id>` | hash | The job: `data`, `status`, `enqueued_at`, `started_at`, `ended_at`, `worker_name`, and so on. |
| `rq:results:<job-id>` | stream | The job's results. |
| `rq:finished:<name>` | sorted set | The IDs of finished jobs. |
| `rq:queues` | set | The names of known queues. |
| `rq:workers` and `rq:worker:<id>` | set, hash | The registered workers and their state. |

A job's hash expires after its result time to live (TTL). The default is about 500
seconds, so finished jobs disappear on their own. Set `result_ttl` when you enqueue a job
to keep them longer.

## What happens to a job when a worker is killed

RQ doesn't remove a running job from Redis. It records the job in a *started* registry,
the sorted set `rq:wip:<name>`, and the job's status stays `STARTED`. After a worker and
its work-horse process were stopped with `kill -9` partway through a job with a
6-second timeout:

- The queue was empty and the started registry held the job, so the job was not lost, but
it also did not run again. A new worker started right afterward left it in the started
registry for the 24 seconds that the test ran.
- After the registry entry's expiry passed, RQ's registry cleanup (`StartedJobRegistry.cleanup()`)
moved the job to the failed registry with an `AbandonedJobError`.

In these runs RQ did not re-run an abandoned job. It ends up in the failed registry, and
your code decides whether to requeue it. Write jobs so that running one twice is safe.

## More information

- [RQ documentation](https://python-rq.org/)
- [RQ on GitHub](https://github.com/rq/rq)
Loading