Repository navigation
Conversation
Add a Database\Claim helper for rows one worker claims at a time. Claiming is one conditional update, and a claim older than its TTL counts as abandoned, so another worker can claim the row again. - Search index queue: reserve jobs atomically and pick up jobs reserved more than 5 minutes ago. Adds a searchindexqueue.dateReserved column. - Image transforms: claim inProgress atomically instead of checking and then writing it, and queue transforms abandoned for more than 30 seconds. - Asset index: AssetIndexData::claim() takes pending entries and entries abandoned while processing for more than 5 minutes. - Queue: hasReservedJobs() ignores jobs reserved for longer than the queue's retry_after, so a dead worker no longer stops the queue from running. Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4 Co-authored-by: Amp <amp@ampcode.com>
JobStatus::canTransitionTo() defines which status changes are allowed, and the job progress upsert only changes a row whose current status can move to the new one. That replaces the cancelled-only guard, and it stops late worker events from moving Done or Failed jobs back to Reserved or swapping Done and Failed. Queuing or retrying a job still starts it over from any status. clear() and updateStatus() read the same table. Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4 Co-authored-by: Amp <amp@ampcode.com>
- RunQueue only skips the queue runner while a worker is running a job. Before, any pending, delayed or failed progress row stopped it, so one failed job turned off automatic queue running for good. - hasPendingJobs() counts delayed jobs whose delay has passed, so the control panel and queue/run start a worker for them. - Jobs queued with a delay of 0 are stored without a delay, so the sidebar shows them. - Cancelled rows are pruned a week after the job's delay ran out. Until then they still stop a delayed job that a worker hasn't picked up yet. Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4 Co-authored-by: Amp <amp@ampcode.com>
A transform that failed once kept error = 1 after it later generated, so if its file was lost, GeneratePendingTransforms skipped it for good. ImageTransformIndex now has markGenerating(), markGenerated() and markFailed(), which set fileExists, inProgress and error together, and ImageTransformer uses them for every generation write. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4
QueueComponent didn't override status(), so isFailed() queried the Yii queue table, which 6.x never creates, and threw. status() now reads job progress, then failed_jobs. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-01a1e8d1-9981-7090-9108-69125aec9ad4
📚 Storybook previews@craftcms/ui — open Storybook No changed components detected in this Storybook. resources/js — open Storybook No changed components detected in this Storybook. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Rows that a worker marks as taken (
jobprogressReserved,imagetransformindex.inProgress,searchindexqueue.reserved, asset index entries in Processing) had no way out when the worker died, so they blocked the queue runner, transforms and indexing for good. They're now taken with a compare-and-set throughDatabase\Claim, and a row held longer than its timeout can be taken again.The job progress statuses also get a transition table, so late worker events can't move a Done or Failed job back:
Also in this PR:
RunQueueonly waits for running jobs, so a failed job no longer stops the queue from running automatically. Delayed jobs count as waiting once they're due, jobs with a delay of 0 show in the sidebar, and cancelled rows are pruned.error. The transform index's flags are set together throughmarkGenerating(),markGenerated()andmarkFailed().QueueComponent::status()(and soisFailed()) reads job progress andfailed_jobs, instead of the Yiiqueuetable that 6.x never creates.Evidence
JobProgress::hasReservedJobs()true until it was pruned.Craft::$app->getQueue()->isFailed('abc')threwno such table: queue. A latesetProgress()moved a Done job back to Reserved.After: new tests in
JobProgressServiceTest,RunQueueMiddlewareTest,ImageTransformerTest,SearchTest,AssetIndexerTestand the adapter'sQueueTestcover each case, and each fails on the old code.