fix(analysis): keep matrix rounds distinct in bug-event charts#34
Draft
grandizzy wants to merge 1 commit into
Draft
fix(analysis): keep matrix rounds distinct in bug-event charts#34grandizzy wants to merge 1 commit into
grandizzy wants to merge 1 commit into
Conversation
Matrix/aggregated runs are flattened by the workflows collector into one per-target directory, with each round's logs renamed to "<run_id>-<name>" (<run_id> ends in "logs.zip"). discover_log_files only keyed on the directory and used a single global run_id, so every round collapsed into one shared "unknown" run. This made bugs-over-time and final-distribution charts report a single run per series (and zero-bug targets seeded a single "unknown:unknown" run via events_to_cumulative). Recover the per-round run id from the flattened filename prefix in both analyze.py (event/throughput/progress parsing) and events_to_cumulative.py (zero-event seeding now walks log files instead of directories). Verdicts are unaffected: the showmap/relscore path already keys on unique trial files. Co-authored-by: grandizzy <38490174+grandizzy@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019eef3d-2e1d-7381-8ef3-4a459426d3ef Co-authored-by: Amp <amp@ampcode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Matrix/aggregated benchmark runs (the
report-matrixpath used for foundry PR benchmarks) are flattened by the workflows collector (modules/tests.nuscfuzzbench-collect-all-logs) into a single per-target directory. Each round's log files are renamed to<run_id>-<name>, where<run_id>is the safe-labeled path of that round'slogs.zip(so it always ends inlogs.zip).scfuzzbench's analysis only keyed run identity on
rel.parts[0](the directory) and used a single globalrun_id(infer_run_idfinds nothing →"unknown"). As a result every round of a target collapsed into one sharedunknownrun.Impact (graphs/tables only — not verdicts)
bugs_over_time/final_distribution/plateaucharts grouped byrun_idreported a single run per series → flat/median-of-one charts and "unknown" labels.liquity-v2-governance) were seeded a singleunknown:unknownrun viaevents_to_cumulativeinventory (which iterated directories, not rounds).Fix (scfuzzbench only)
analyze.py:discover_log_filesrecovers the per-roundrun_idfrom the flattened filename prefix; event/throughput/progress parsing use it (falling back to the previous global behaviour for native layouts).events_to_cumulative.py: zero-event seeding now walks*.logfiles (recovering the same per-round prefix) instead of iterating directories, so each round seeds a distinct run and aligns with the event run keys.Native (non-matrix) and workflows single-run layouts are unchanged (no
logs.zip-prefix → same fallback as before).Tests
analysis/tests/test_matrix_run_identity.pypins the contract end-to-end: matrix-flattened layout yields distinct rounds indiscover_log_files,parse_logs, andevents_to_cumulative(including the zero-bug target), and event/inventory run keys align.test_runner_command_sanitizationare unrelated bash-env issues that also fail onmain).Co-Authored-By: grandizzy 38490174+grandizzy@users.noreply.github.com
Prompted by: grandizzy