Skip to content

prevent cross-architectural leakeage in matrix_job_re - #76

Open
cgoea wants to merge 2 commits into
mainfrom
users/cgoea/group_by_arch
Open

prevent cross-architectural leakeage in matrix_job_re#76
cgoea wants to merge 2 commits into
mainfrom
users/cgoea/group_by_arch

Conversation

@cgoea

@cgoea cgoea commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Motivation

status.json's matrix-cell key for PyTorch/JAX fan-out builds is parsed purely from job names as (py, ref) — it carries no GPU architecture. When a single build cell (e.g. py 3.12 | torch release/2.10) nests test jobs for multiple architectures (e.g. gfx942 and gfx1101), all of those jobs collapse into one shared, worst-of variant.

That conflated variant then gets broadcast onto every architecture's test leaf by _refresh_same_run_fanout_tests, so one architecture's failure silently drags down another architecture's otherwise-passing result (and vice versa) in status.json, even though the two GPUs' actual test outcomes are independent.

closes #75

Technical Details

  • Added _TEST_ARCH_JOB_RE / _job_matches_arch to detect which architecture (if any) a job names in its own "Test | <arch>" segment, so a job can be classified as either arch-agnostic (the cell's shared build step) or scoped to one specific architecture.
  • _variants_from_jobs, _derive_variants, and _create_leaf now accept an optional arch parameter that filters the job list down to that architecture's own jobs plus any arch-agnostic jobs before deriving matrix-cell variants.
  • _refresh_same_run_fanout_tests now re-derives each existing test leaf's variants scoped to its own architecture, instead of reusing the build leaf's arch-blind variant list wholesale. It also rolls the leaf's status up from those scoped variants (previously it copied the raw run conclusion directly, which doesn't reflect the arch-specific outcome either).
  • _merge_run_into_document now re-derives a per-architecture leaf when a single event legitimately reports more than one target architecture, closing the same conflation risk (and a related object-aliasing hazard) on the primary write path, not just the same-run refresh path.

Test Plan

  • Added test_completed_fanout_build_does_not_leak_status_across_architectures, which sets up two architectures (gfx942 passing, gfx1101 failing) nested under the same (py, torch) build cell and asserts each architecture's test leaf reflects only its own outcome.
  • Verified the new test fails against the pre-fix code (gfx942's leaf incorrectly flips to failure) and passes with the fix applied.
  • Ran the full scripts/receive_therock test suite and checked for regressions against develop.

Test Result

  • pytest scripts/receive_therock/tests/therock_update_status_json_test.py: 99 passed
  • pytest scripts/receive_therock/tests/: 401 passed

Submission Checklist

@cgoea
cgoea requested a review from a team August 12, 2026 18:34
@cgoea
cgoea changed the base branch from main to develop August 12, 2026 18:35
…by_arch

# Conflicts:
#	scripts/receive_therock/tests/therock_update_status_json_test.py
#	scripts/receive_therock/therock_update_status_json.py
@cgoea
cgoea changed the base branch from develop to main August 13, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant