Skip to content

ci: let staging jobs wait for a slot, skip them for docs changes - #255

Merged
rfay merged 1 commit into
mainfrom
20261117_rfay_ci_queue_no_giveup
Nov 17, 2026
Merged

rfay merged 1 commit into
mainfrom
20261117_rfay_ci_queue_no_giveup

Conversation

@rfay

@rfay rfay commented Nov 17, 2026

Copy link
Copy Markdown
Member

Short Summary (TL;DR)

Integration jobs now wait for a staging lock slot as long as it takes, not 45 minutes, and docs-only changes no longer start them. When several pushes ran CI together, many jobs failed only because they waited too long.

The Issue

The staging workflows share a few lock slots on staging-coder.ddev.com. A job gave up after waiting 45 minutes for a slot. The wait also counted against the 75-minute job timeout. A passing job holds its slot for up to 45 minutes, so a job that waited long could not finish in time.

One push to main starts about 9 heavy jobs, and one PR push starts 6 to 9. Whenever two pushes ran CI together, the jobs that waited longest failed. Days with 9 to 39 jobs passed almost completely. On 2026-11-05, 2026-11-16 and 2026-11-17, there were 77 to 152 jobs a day, and only 49% to 63% passed. Most failures were all 3 staging box lock slot(s) still busy after 2700s -- giving up, or a timeout during or right after the wait.

Docs-only PRs started every heavy workflow too, although docs-check.yml is the only workflow that tests them.

How This PR Solves The Issue

Change Before After
MAX_WAIT_SECONDS default in ci-acquire-staging-lock.sh 2700 (45 min) 0, which means no limit
timeout-minutes of each integration job 75 360, the maximum for GitHub-hosted runners
VERSION_AGE_MINUTES default in ci-reap-staging.sh 60 380
paths-ignore for docs-only changes none integration-test.yml, drupal-integration-test.yml, drupal-contrib-integration-test.yml, staging-push.yml

A stuck job still cannot hold a slot for long. The janitor deletes workspaces older than 80 minutes, and a waiting job reclaims a slot older than 80 minutes. The longest slot hold among 128 passing jobs since 2026-11-15 was 44.5 minutes.

The janitor archives unused template versions older than VERSION_AGE_MINUTES. A job pushes its template version before it waits for a slot. With no wait limit, 60 minutes could archive a version that a waiting job still needs, so the limit is now above the job timeout.

The docs patterns are docs/**, top-level *.md, */README.md, and the docs tool configuration files. Markdown files that templates copy into workspaces, such as modules/*/files/*.md, still trigger the tests. A change with any non-docs file still runs every workflow. schedule runs are not filtered.

Manual Testing Instructions

  1. Open a PR that changes only a file under docs/. Expect Check docs and Validate Templates, and no integration or Staging Push Validation run.
  2. Push to two PRs that change templates within a minute. Expect jobs to log All N staging box lock slot(s) busy; waiting 30s (...s so far)... and then continue, with no giving up failure.
  3. Run the janitor in dry-run mode while a job waits for a slot. Expect no archive line for that job's template version:
gh workflow run ci-reap.yml -R ddev/coder-ddev -f dry_run=true

Automated Testing Overview

No tests cover these workflows or scripts. Claude Code ran bash -n and shellcheck on both scripts on Linux, with no findings. It also ran actionlint on all workflows, which reported the same 19 findings before and after the change. Each workflow file parses as YAML with the expected triggers. Nobody has run the new queueing on staging yet.

Release/Deployment Notes

Raise the CI_LOCK_SLOTS repository variable to allow more jobs at once. Staging has 64 GB of memory, and these tests use about 1 CPU each. Watch the box load after a change, because starts are the heaviest part.


Generated by Claude Code

## Short Summary (TL;DR)

Integration jobs now wait for a staging lock slot as long as it takes, not 45 minutes, and docs-only changes no longer start them. When several pushes ran CI together, many jobs failed only because they waited too long.

## The Issue

The staging workflows share a few lock slots on staging-coder.ddev.com. A job gave up after waiting 45 minutes for a slot. The wait also counted against the 75-minute job timeout. A passing job holds its slot for up to 45 minutes, so a job that waited long could not finish in time.

One push to `main` starts about 9 heavy jobs, and one PR push starts 6 to 9. Whenever two pushes ran CI together, the jobs that waited longest failed. Days with 9 to 39 jobs passed almost completely. On 2026-11-05, 2026-11-16 and 2026-11-17, there were 77 to 152 jobs a day, and only 49% to 63% passed. Most failures were `all 3 staging box lock slot(s) still busy after 2700s -- giving up`, or a timeout during or right after the wait.

Docs-only PRs started every heavy workflow too, although `docs-check.yml` is the only workflow that tests them.

## How This PR Solves The Issue

| Change | Before | After |
| --- | --- | --- |
| `MAX_WAIT_SECONDS` default in `ci-acquire-staging-lock.sh` | 2700 (45 min) | 0, which means no limit |
| `timeout-minutes` of each integration job | 75 | 360, the maximum for GitHub-hosted runners |
| `VERSION_AGE_MINUTES` default in `ci-reap-staging.sh` | 60 | 380 |
| `paths-ignore` for docs-only changes | none | `integration-test.yml`, `drupal-integration-test.yml`, `drupal-contrib-integration-test.yml`, `staging-push.yml` |

A stuck job still cannot hold a slot for long. The janitor deletes workspaces older than 80 minutes, and a waiting job reclaims a slot older than 80 minutes. The longest slot hold among 128 passing jobs since 2026-11-15 was 44.5 minutes.

The janitor archives unused template versions older than `VERSION_AGE_MINUTES`. A job pushes its template version before it waits for a slot. With no wait limit, 60 minutes could archive a version that a waiting job still needs, so the limit is now above the job timeout.

The docs patterns are `docs/**`, top-level `*.md`, `*/README.md`, and the docs tool configuration files. Markdown files that templates copy into workspaces, such as `modules/*/files/*.md`, still trigger the tests. A change with any non-docs file still runs every workflow. `schedule` runs are not filtered.

## Manual Testing Instructions

1. Open a PR that changes only a file under `docs/`. Expect `Check docs` and `Validate Templates`, and no integration or `Staging Push Validation` run.
2. Push to two PRs that change templates within a minute. Expect jobs to log `All N staging box lock slot(s) busy; waiting 30s (...s so far)...` and then continue, with no `giving up` failure.
3. Run the janitor in dry-run mode while a job waits for a slot. Expect no `archive` line for that job's template version:

```bash
gh workflow run ci-reap.yml -R ddev/coder-ddev -f dry_run=true
```

## Automated Testing Overview

No tests cover these workflows or scripts. Claude Code ran `bash -n` and `shellcheck` on both scripts on Linux, with no findings. It also ran `actionlint` on all workflows, which reported the same 19 findings before and after the change. Each workflow file parses as YAML with the expected triggers. Nobody has run the new queueing on staging yet.

## Release/Deployment Notes

Raise the `CI_LOCK_SLOTS` repository variable to allow more jobs at once. Staging has 64 GB of memory, and these tests use about 1 CPU each. Watch the box load after a change, because starts are the heaviest part.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01T3Cc2F1jt7n3f7sxKHo6p1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3Cc2F1jt7n3f7sxKHo6p1
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Previews

The start.coder.ddev.com landing-page preview has been removed. Read the Docs keeps or removes its own docs preview.

@rfay
rfay merged commit 4e928dc into main Nov 17, 2026
19 checks passed
@rfay
rfay deleted the 20261117_rfay_ci_queue_no_giveup branch November 17, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants