fix: retry failed latest only#410
Draft
mdiapenabc wants to merge 2 commits into
Draft
Conversation
Exclude RETRY_FAILED from the retry queue and dedupe failed cron jobs by job type so hourly retries do not replay the full historical backlog.
Only retry cron failures newer than the last successful run for that job type, and align hasStuckOrFailedJobs with the same actionable set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes the
csa-retry-failedcron replaying the full historicaljob_runsbacklog (e.g. 83 failed rows / ~2+ hours on dev) instead of only retrying jobs that still need attention.Root cause:
getFailedJobs()returned every top-level failed cron job with no deduplication, andRETRY_FAILEDitself could be retried recursively.Changes:
RETRY_FAILEDfrom the retry queue to prevent recursionjob_type(not the full backlog)completedAt <= lastSuccess)hasStuckOrFailedJobs()with the same actionable set so the hourly cron exits early when there is nothing to doEnd-user
SEND_CRA_FILEfailures are unchanged: latest failure per type is still retried regardless of later successes.Fixes # (issue)
How Has This Been Tested?
Ran
npm testfor:jobs.service.spec.tsjob-runner.service.spec.tsretry-failed.handler.spec.tsManual verification on dev (post-deploy):
csa-retry-failedrun logs a small retry count (≈ number of job types with current failures, not 80+)INGEST_DATAfailure still retries while ICM is downFAILEDrows remain injob_runsFollow UP
FAILEDrows remain injob_runs(harmless; they are just no longer retried)