Skip to content

fix(analysis): keep matrix rounds distinct in bug-event charts#34

Draft
grandizzy wants to merge 1 commit into
mainfrom
fix/matrix-run-identity-from-log-prefix
Draft

fix(analysis): keep matrix rounds distinct in bug-event charts#34
grandizzy wants to merge 1 commit into
mainfrom
fix/matrix-run-identity-from-log-prefix

Conversation

@grandizzy

Copy link
Copy Markdown

Summary

Matrix/aggregated benchmark runs (the report-matrix path used for foundry PR benchmarks) are flattened by the workflows collector (modules/tests.nu scfuzzbench-collect-all-logs) into a single per-target directory. Each round's log files are renamed to <run_id>-<name>, where <run_id> is the safe-labeled path of that round's logs.zip (so it always ends in logs.zip).

scfuzzbench's analysis only keyed run identity on rel.parts[0] (the directory) and used a single global run_id (infer_run_id finds nothing → "unknown"). As a result every round of a target collapsed into one shared unknown run.

Impact (graphs/tables only — not verdicts)

  • bugs_over_time / final_distribution / plateau charts grouped by run_id reported a single run per series → flat/median-of-one charts and "unknown" labels.
  • Zero-bug targets (e.g. liquity-v2-governance) were seeded a single unknown:unknown run via events_to_cumulative inventory (which iterated directories, not rounds).
  • Verdicts are unaffected: the showmap / relscore / relcov path keys on unique trial files, so rounds stay distinct there and the regression/inconclusive verdicts are computed correctly.

Fix (scfuzzbench only)

  • analyze.py: discover_log_files recovers the per-round run_id from the flattened filename prefix; event/throughput/progress parsing use it (falling back to the previous global behaviour for native layouts).
  • events_to_cumulative.py: zero-event seeding now walks *.log files (recovering the same per-round prefix) instead of iterating directories, so each round seeds a distinct run and aligns with the event run keys.

Native (non-matrix) and workflows single-run layouts are unchanged (no logs.zip- prefix → same fallback as before).

Tests

  • New analysis/tests/test_matrix_run_identity.py pins the contract end-to-end: matrix-flattened layout yields distinct rounds in discover_log_files, parse_logs, and events_to_cumulative (including the zero-bug target), and event/inventory run keys align.
  • Full analysis suite passes (3 pre-existing failures in test_runner_command_sanitization are unrelated bash-env issues that also fail on main).

Co-Authored-By: grandizzy 38490174+grandizzy@users.noreply.github.com

Prompted by: grandizzy

Matrix/aggregated runs are flattened by the workflows collector into one
per-target directory, with each round's logs renamed to
"<run_id>-<name>" (<run_id> ends in "logs.zip"). discover_log_files only
keyed on the directory and used a single global run_id, so every round
collapsed into one shared "unknown" run. This made bugs-over-time and
final-distribution charts report a single run per series (and zero-bug
targets seeded a single "unknown:unknown" run via events_to_cumulative).

Recover the per-round run id from the flattened filename prefix in both
analyze.py (event/throughput/progress parsing) and events_to_cumulative.py
(zero-event seeding now walks log files instead of directories). Verdicts
are unaffected: the showmap/relscore path already keys on unique trial
files.

Co-authored-by: grandizzy <38490174+grandizzy@users.noreply.github.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019eef3d-2e1d-7381-8ef3-4a459426d3ef
Co-authored-by: Amp <amp@ampcode.com>
@grandizzy grandizzy self-assigned this Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant