Skip to content

fix: retry failed latest only#410

Draft
mdiapenabc wants to merge 2 commits into
devfrom
fix/retry-failed-latest-only
Draft

fix: retry failed latest only#410
mdiapenabc wants to merge 2 commits into
devfrom
fix/retry-failed-latest-only

Conversation

@mdiapenabc

Copy link
Copy Markdown
Collaborator

Description

Fixes the csa-retry-failed cron replaying the full historical job_runs backlog (e.g. 83 failed rows / ~2+ hours on dev) instead of only retrying jobs that still need attention.

Root cause: getFailedJobs() returned every top-level failed cron job with no deduplication, and RETRY_FAILED itself could be retried recursively.

Changes:

  • Exclude RETRY_FAILED from the retry queue to prevent recursion
  • Retry only the latest failure per job_type (not the full backlog)
  • Skip cron failures superseded by a later success (completedAt <= lastSuccess)
  • Align hasStuckOrFailedJobs() with the same actionable set so the hourly cron exits early when there is nothing to do

End-user SEND_CRA_FILE failures are unchanged: latest failure per type is still retried regardless of later successes.

Fixes # (issue)

How Has This Been Tested?

Ran npm test for:

  • jobs.service.spec.ts
  • job-runner.service.spec.ts
  • retry-failed.handler.spec.ts

Manual verification on dev (post-deploy):

  1. Confirm next csa-retry-failed run logs a small retry count (≈ number of job types with current failures, not 80+)
  2. Confirm latest INGEST_DATA failure still retries while ICM is down
  3. Confirm cron exits early when only stale FAILED rows remain in job_runs

Follow UP

  • Delete - Historical stale FAILED rows remain in job_runs (harmless; they are just no longer retried)

Exclude RETRY_FAILED from the retry queue and dedupe failed cron jobs
by job type so hourly retries do not replay the full historical backlog.
Only retry cron failures newer than the last successful run for that
job type, and align hasStuckOrFailedJobs with the same actionable set.
@mdiapenabc mdiapenabc changed the title fix/retry failed latest only fix: retry failed latest only Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant