Skip to content

[6.x] Recover rows claimed by workers that died - #20792

Open
riasvdv wants to merge 5 commits into
6.xfrom
rias/stale-claims
Open

riasvdv wants to merge 5 commits into
6.xfrom
rias/stale-claims

Conversation

@riasvdv

@riasvdv riasvdv commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Description

Rows that a worker marks as taken (jobprogress Reserved, imagetransformindex.inProgress, searchindexqueue.reserved, asset index entries in Processing) had no way out when the worker died, so they blocked the queue runner, transforms and indexing for good. They're now taken with a compare-and-set through Database\Claim, and a row held longer than its timeout can be taken again.

 Reserved job        -> blocks QueueController::run() until the 7-day prune
+Reserved job        -> ignored once older than the queue's retry_after
 inProgress transform -> wait loop, then generate anyway (twice)
+inProgress transform -> claimed atomically, re-queued after 30 s
 reserved search row -> never indexed again
+reserved search row -> re-indexed after 5 min (new dateReserved column)

The job progress statuses also get a transition table, so late worker events can't move a Done or Failed job back:

upsert(status)
  if not current->canTransitionTo(status): keep current

Also in this PR:

  • RunQueue only waits for running jobs, so a failed job no longer stops the queue from running automatically. Delayed jobs count as waiting once they're due, jobs with a delay of 0 show in the sidebar, and cancelled rows are pruned.
  • A successful transform clears error. The transform index's flags are set together through markGenerating(), markGenerated() and markFailed().
  • The legacy QueueComponent::status() (and so isFailed()) reads job progress and failed_jobs, instead of the Yii queue table that 6.x never creates.

Evidence

  • Before: a dead worker's Reserved row made JobProgress::hasReservedJobs() true until it was pruned. Craft::$app->getQueue()->isFailed('abc') threw no such table: queue. A late setProgress() moved a Done job back to Reserved.
    After: new tests in JobProgressServiceTest, RunQueueMiddlewareTest, ImageTransformerTest, SearchTest, AssetIndexerTest and the adapter's QueueTest cover each case, and each fails on the old code.

riasvdv and others added 5 commits November 17, 2026 15:07
Add a Database\Claim helper for rows one worker claims at a time. Claiming is
one conditional update, and a claim older than its TTL counts as abandoned, so
another worker can claim the row again.

- Search index queue: reserve jobs atomically and pick up jobs reserved more
  than 5 minutes ago. Adds a searchindexqueue.dateReserved column.
- Image transforms: claim inProgress atomically instead of checking and then
  writing it, and queue transforms abandoned for more than 30 seconds.
- Asset index: AssetIndexData::claim() takes pending entries and entries
  abandoned while processing for more than 5 minutes.
- Queue: hasReservedJobs() ignores jobs reserved for longer than the queue's
  retry_after, so a dead worker no longer stops the queue from running.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4
Co-authored-by: Amp <amp@ampcode.com>
JobStatus::canTransitionTo() defines which status changes are allowed, and
the job progress upsert only changes a row whose current status can move to
the new one. That replaces the cancelled-only guard, and it stops late worker
events from moving Done or Failed jobs back to Reserved or swapping Done and
Failed. Queuing or retrying a job still starts it over from any status.

clear() and updateStatus() read the same table.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4
Co-authored-by: Amp <amp@ampcode.com>
- RunQueue only skips the queue runner while a worker is running a job.
  Before, any pending, delayed or failed progress row stopped it, so one
  failed job turned off automatic queue running for good.
- hasPendingJobs() counts delayed jobs whose delay has passed, so the
  control panel and queue/run start a worker for them.
- Jobs queued with a delay of 0 are stored without a delay, so the sidebar
  shows them.
- Cancelled rows are pruned a week after the job's delay ran out. Until
  then they still stop a delayed job that a worker hasn't picked up yet.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4
Co-authored-by: Amp <amp@ampcode.com>
A transform that failed once kept error = 1 after it later generated, so
if its file was lost, GeneratePendingTransforms skipped it for good.

ImageTransformIndex now has markGenerating(), markGenerated() and
markFailed(), which set fileExists, inProgress and error together, and
ImageTransformer uses them for every generation write.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4
QueueComponent didn't override status(), so isFailed() queried the Yii
queue table, which 6.x never creates, and threw. status() now reads job
progress, then failed_jobs.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4
@github-actions

Copy link
Copy Markdown

📚 Storybook previews

@craftcms/ui — open Storybook

No changed components detected in this Storybook.

resources/js — open Storybook

No changed components detected in this Storybook.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant