Repository navigation
ci: let staging jobs wait for a slot, skip them for docs changes - #255
Merged
Merged
Conversation
## Short Summary (TL;DR) Integration jobs now wait for a staging lock slot as long as it takes, not 45 minutes, and docs-only changes no longer start them. When several pushes ran CI together, many jobs failed only because they waited too long. ## The Issue The staging workflows share a few lock slots on staging-coder.ddev.com. A job gave up after waiting 45 minutes for a slot. The wait also counted against the 75-minute job timeout. A passing job holds its slot for up to 45 minutes, so a job that waited long could not finish in time. One push to `main` starts about 9 heavy jobs, and one PR push starts 6 to 9. Whenever two pushes ran CI together, the jobs that waited longest failed. Days with 9 to 39 jobs passed almost completely. On 2026-11-05, 2026-11-16 and 2026-11-17, there were 77 to 152 jobs a day, and only 49% to 63% passed. Most failures were `all 3 staging box lock slot(s) still busy after 2700s -- giving up`, or a timeout during or right after the wait. Docs-only PRs started every heavy workflow too, although `docs-check.yml` is the only workflow that tests them. ## How This PR Solves The Issue | Change | Before | After | | --- | --- | --- | | `MAX_WAIT_SECONDS` default in `ci-acquire-staging-lock.sh` | 2700 (45 min) | 0, which means no limit | | `timeout-minutes` of each integration job | 75 | 360, the maximum for GitHub-hosted runners | | `VERSION_AGE_MINUTES` default in `ci-reap-staging.sh` | 60 | 380 | | `paths-ignore` for docs-only changes | none | `integration-test.yml`, `drupal-integration-test.yml`, `drupal-contrib-integration-test.yml`, `staging-push.yml` | A stuck job still cannot hold a slot for long. The janitor deletes workspaces older than 80 minutes, and a waiting job reclaims a slot older than 80 minutes. The longest slot hold among 128 passing jobs since 2026-11-15 was 44.5 minutes. The janitor archives unused template versions older than `VERSION_AGE_MINUTES`. A job pushes its template version before it waits for a slot. With no wait limit, 60 minutes could archive a version that a waiting job still needs, so the limit is now above the job timeout. The docs patterns are `docs/**`, top-level `*.md`, `*/README.md`, and the docs tool configuration files. Markdown files that templates copy into workspaces, such as `modules/*/files/*.md`, still trigger the tests. A change with any non-docs file still runs every workflow. `schedule` runs are not filtered. ## Manual Testing Instructions 1. Open a PR that changes only a file under `docs/`. Expect `Check docs` and `Validate Templates`, and no integration or `Staging Push Validation` run. 2. Push to two PRs that change templates within a minute. Expect jobs to log `All N staging box lock slot(s) busy; waiting 30s (...s so far)...` and then continue, with no `giving up` failure. 3. Run the janitor in dry-run mode while a job waits for a slot. Expect no `archive` line for that job's template version: ```bash gh workflow run ci-reap.yml -R ddev/coder-ddev -f dry_run=true ``` ## Automated Testing Overview No tests cover these workflows or scripts. Claude Code ran `bash -n` and `shellcheck` on both scripts on Linux, with no findings. It also ran `actionlint` on all workflows, which reported the same 19 findings before and after the change. Each workflow file parses as YAML with the expected triggers. Nobody has run the new queueing on staging yet. ## Release/Deployment Notes Raise the `CI_LOCK_SLOTS` repository variable to allow more jobs at once. Staging has 64 GB of memory, and these tests use about 1 CPU each. Watch the box load after a change, because starts are the heaviest part. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01T3Cc2F1jt7n3f7sxKHo6p1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3Cc2F1jt7n3f7sxKHo6p1
PreviewsThe start.coder.ddev.com landing-page preview has been removed. Read the Docs keeps or removes its own docs preview. |
1 task done
1 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Summary (TL;DR)
Integration jobs now wait for a staging lock slot as long as it takes, not 45 minutes, and docs-only changes no longer start them. When several pushes ran CI together, many jobs failed only because they waited too long.
The Issue
The staging workflows share a few lock slots on staging-coder.ddev.com. A job gave up after waiting 45 minutes for a slot. The wait also counted against the 75-minute job timeout. A passing job holds its slot for up to 45 minutes, so a job that waited long could not finish in time.
One push to
mainstarts about 9 heavy jobs, and one PR push starts 6 to 9. Whenever two pushes ran CI together, the jobs that waited longest failed. Days with 9 to 39 jobs passed almost completely. On 2026-11-05, 2026-11-16 and 2026-11-17, there were 77 to 152 jobs a day, and only 49% to 63% passed. Most failures wereall 3 staging box lock slot(s) still busy after 2700s -- giving up, or a timeout during or right after the wait.Docs-only PRs started every heavy workflow too, although
docs-check.ymlis the only workflow that tests them.How This PR Solves The Issue
MAX_WAIT_SECONDSdefault inci-acquire-staging-lock.shtimeout-minutesof each integration jobVERSION_AGE_MINUTESdefault inci-reap-staging.shpaths-ignorefor docs-only changesintegration-test.yml,drupal-integration-test.yml,drupal-contrib-integration-test.yml,staging-push.ymlA stuck job still cannot hold a slot for long. The janitor deletes workspaces older than 80 minutes, and a waiting job reclaims a slot older than 80 minutes. The longest slot hold among 128 passing jobs since 2026-11-15 was 44.5 minutes.
The janitor archives unused template versions older than
VERSION_AGE_MINUTES. A job pushes its template version before it waits for a slot. With no wait limit, 60 minutes could archive a version that a waiting job still needs, so the limit is now above the job timeout.The docs patterns are
docs/**, top-level*.md,*/README.md, and the docs tool configuration files. Markdown files that templates copy into workspaces, such asmodules/*/files/*.md, still trigger the tests. A change with any non-docs file still runs every workflow.scheduleruns are not filtered.Manual Testing Instructions
docs/. ExpectCheck docsandValidate Templates, and no integration orStaging Push Validationrun.All N staging box lock slot(s) busy; waiting 30s (...s so far)...and then continue, with nogiving upfailure.archiveline for that job's template version:Automated Testing Overview
No tests cover these workflows or scripts. Claude Code ran
bash -nandshellcheckon both scripts on Linux, with no findings. It also ranactionlinton all workflows, which reported the same 19 findings before and after the change. Each workflow file parses as YAML with the expected triggers. Nobody has run the new queueing on staging yet.Release/Deployment Notes
Raise the
CI_LOCK_SLOTSrepository variable to allow more jobs at once. Staging has 64 GB of memory, and these tests use about 1 CPU each. Watch the box load after a change, because starts are the heaviest part.Generated by Claude Code