Skip to content

Stalled jobs exceeding maxStalledCount are deleted instead of moved to failed #30

Description

@Kds-JS

Summary

When a stalled job exceeds maxStalledCount, the Redis adapter deletes the job data entirely instead of moving it to the failed set. No failed entry is created and application-level failure hooks never run, so the job disappears silently.

This contradicts the library's own documentation: the JSDoc for recoverStalledJobs states "Jobs exceeding maxStalledCount are failed permanently" — but the implementation calls delete_job_data instead of failing the job.

For comparison, BullMQ in the same situation moves the job to failed with the error "job stalled more than allowable limit", keeping it visible and replayable.

Affected version

@boringnode/queue 0.6.0 (via @adonisjs/queue 0.6.2), Redis adapter.

Where

In the stalled-recovery Lua script (build/chunk-S37X3CBO.js, recoverStalledJobs path):

-- Check if job has exceeded max stalled count
if current_stalled_count >= max_stalled_count then
  -- Job failed permanently, remove data + dedup key (only if pointer still ours)
  ...
  delete_job_data(data_key, overlay_key, job_id)
else
  -- Recover: increment stalledCount ...

The comment says "failed permanently", but delete_job_data removes the hash entries: the job ends up in no set at all — not pending, not active, not failed.

Reproduction

With default worker config (maxStalledCount: 1, stalledThreshold: 30s):

  1. Dispatch a job; kill the worker hard (SIGKILL / OOM) while the job is active so the heartbeat stops.
  2. Wait past stalledThreshold; a fresh worker recovers the job → stalledCount = 1, job re-queued. ✅
  3. Kill the worker hard again while the same job is active; wait past the threshold.
  4. On the next recovery pass: stalledCount (1) >= maxStalledCount (1) → job data is deleted.

Observed Redis state after step 4: job = null, failed = 0, pending = 0, active = 0. The job's failed() hook is never invoked, so retry/alerting logic built on it never fires.

Expected behavior

A job exceeding maxStalledCount should be moved to the failed set (with a "stalled more than allowable limit"-style error) and go through the normal terminal-failure path so failed() hooks and retention rules apply — matching both the JSDoc and BullMQ's semantics.

Impact

Two brutal worker interruptions on the same job (e.g. consecutive deploys, OOM kills) permanently and silently lose the job. In production queue systems this is indistinguishable from "everything is fine".

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions