Bound per-matrix concurrency in integration test workflows - #2231
Open
olowosulu wants to merge 2 commits into
Open
Bound per-matrix concurrency in integration test workflows#2231olowosulu wants to merge 2 commits into
olowosulu wants to merge 2 commits into
Conversation
With 600+ matrix jobs hitting STS AssumeRoleWithWebIdentity concurrently against the same OIDC role, contention on the endpoint may cause throttled requests. The credential action fetches a short-lived JWT (~5 min + 5 min clock-skew grace) and retries AssumeRole with that same token. If retries are delayed long enough, the token can expire before a successful call. Setting max-parallel: 100 limits concurrent STS calls per matrix. The value is not calibrated against measured throttle data; it is a conservative cap based on the reasoning above. The platform concurrency limit for the account may already cap below 100.
musa-asad
reviewed
Aug 6, 2026
| strategy: | ||
| fail-fast: false | ||
| # Cap concurrent jobs per matrix to limit simultaneous STS AssumeRoleWithWebIdentity calls. | ||
| # 100 is a conservative value, not calibrated against throttle data; account concurrency limits may cap lower. |
Contributor
There was a problem hiding this comment.
nit - "account concurrency limits" reads as an AWS quota here; worth naming the GitHub runner concurrency limit explicitly?
| fail-fast: false | ||
| # Cap concurrent jobs per matrix to limit simultaneous STS AssumeRoleWithWebIdentity calls. | ||
| # 100 is a conservative value, not calibrated against throttle data; account concurrency limits may cap lower. | ||
| max-parallel: 100 |
Contributor
There was a problem hiding this comment.
nit - this matrix resolves to a single row today, so the cap is inert here — worth saying so in the comment?
musa-asad
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
max-parallel: 100to the 13 matrix job definitions in the integration testworkflows, bounding how many jobs from any single matrix run concurrently.
This is defensive hardening rather than a fix for a diagnosed failure. I want to be
upfront about the evidence, because the strength of the argument varies by claim.
What the change does
Twelve sites in
test-artifacts.ymland one inec2-integration-test.yml. No otherchanges, no deletions.
The argument I am confident in
The workflow dispatches on the order of 600 jobs.
max-parallelis scoped per matrixdefinition, so today any one matrix can request as much concurrency as it has jobs, and
a single large matrix can consume the account's shared runner pool at the expense of
others.
This is not hypothetical for the largest matrices. Public run history for run
30584015978shows theEC2Linux-0matrix generating 200 jobs andEC2Linux-1generating 176. Those come from
ec2-integration-test.yml, which is called nine timesfrom
test-artifacts.yml. Capping at 100 measurably constrains those matrices and givesfairer scheduling across the rest.
For matrices that generate fewer than 100 jobs the setting is inert, so the effect is
partial by design.
Verification
Both workflow files parse with
yaml.safe_load, and 13 jobs carry the new key(12 in
test-artifacts.yml, 1 inec2-integration-test.yml). The diff touches onlythose two files.