Fix remaining flaky CodSpeed microbenchmarks by connortsui20 · Pull Request #8861 · vortex-data/vortex

connortsui20 · 2026-07-20T15:33:26Z

Rationale for this change

Closes: Remove remaining codspeed flakes #8807

Three benchmarks stayed flaky after #8742, flipping between the same two values on PRs that can't affect them. Same root causes and fixes as #8742:

Benchmark	Seen flaky on	Why	Fix
`true_count_vortex_buffer[128]`	±11.17% on 9 unrelated PRs (#8805, #8811, #8812, #8820, #8843, #8803, …)	a 128-bit popcount measures harness overhead and code layout, not the count	drop the 128 size
runend `compress[(100000, 4)]`	±11.9% on #8805, #8750, #8856	allocates in the timed region; glibc malloc differs across runner images	mimalloc as global allocator
`cast_decimal` `copy_*[65536]`	identical flags on #8838 and #8724	same glibc-malloc cause (512 KB alloc per iteration)	mimalloc as global allocator

Left alone: compact_sliced[(4096, 90)] (single sighting) and the CUDA walltime benches (hosted-runner walltime noise, a runner config issue).

The allocator swap shifts every benchmark in the two touched binaries once — see the comment below. Needs a one-time CodSpeed acknowledgment, like #8742.

What changes are included in this PR?

One commit per benchmark; bench files only. Ran cargo check + clippy on the three bench targets, smoke-ran the binaries, cargo +nightly fmt.

`true_count_vortex_buffer[128]` flipped between the same two values (580.6ns <-> 522.2ns, +/-11.17%) on at least nine unrelated PRs since #8742, including java-only (#8811, #8812), Cargo.toml-only (#8820), website-removal-only (#8805), zstd-only (#8843), and geo-only (#8803) changes. A 128-bit true_count is a popcount over two u64 words, so the measurement is fixed dispatch/harness overhead and binary code layout rather than the count itself. Keep only the sizes where the popcount loop dominates, mirroring the bitwise_not_vortex_buffer_mut fix from #8742. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P

`copy_nullable[65536]` and `copy_non_nullable[65536]` were on the #8742 watch list and have now been flagged with byte-identical values on two unrelated open PRs (#8838 union scalars, #8724 decimal add/sub kernels; -24.3% / -16.59% each). The copy casts allocate their output buffer in the timed region, so glibc malloc differences across CodSpeed runner images show up as benchmark changes. Install vendored mimalloc as the global allocator, as #8742 did for the other alloc-heavy vortex-array bench files. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P

`compress[(100000, 4)]` flipped between the same two values (1.8ms <-> 1.6ms, ~+/-11.9%) on three unrelated PRs since #8742: #8805 (deletes a JS website), #8750, and #8856 (list min/max expressions). runend_encode allocates its output buffers inside the timed region and the runend bench binaries still use glibc malloc, whose code differs across CodSpeed runner images. Install vendored mimalloc as the global allocator, matching the #8742 fix for the flaky vortex-array benches. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P

github-actions · 2026-07-20T15:36:18Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`1902072`		1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-07-20T15:37:11Z

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +1.9%
Engines: DataFusion No clear signal (+5.3%, low confidence) · DuckDB No clear signal (-1.5%, low confidence)
Vortex (geomean): 0.974x ➖
Parquet (geomean): 0.966x ➖
Shifts: Parquet (control) -3.4% · Median polish -1.6%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (0.987x ➖, 0↑ 0↓)

name	PR `1902072` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	9615865	9831831	0.98
vortex_q01/datafusion:vortex-file-compressed	6190978	6216077	1.00

datafusion / parquet (0.937x ➖, 0↑ 0↓)

name	PR `1902072` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	19834449	21543706	0.92
vortex_q01/datafusion:parquet	4496080	4713514	0.95

duckdb / vortex-file-compressed (0.980x ➖, 0↑ 0↓)

name	PR `1902072` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9973448	9900408	1.01
vortex_q01/duckdb:vortex-file-compressed	5688309	5965326	0.95

duckdb / parquet (0.995x ➖, 0↑ 0↓)

name	PR `1902072` (ns)	base `b7b01d3` (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23287826	23497829	0.99
vortex_q01/duckdb:parquet	9394154	9407972	1.00

No file size changes detected.

codspeed-hq · 2026-07-20T15:37:58Z

Merging this PR will degrade performance by 16.43%

❌ 6 regressed benchmarks
✅ 1662 untouched benchmarks
⏩ 46 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`decompress[u64, (1000, 16)]`	22.6 µs	31.1 µs	-27.3%
❌	Simulation	`decompress[u8, (1000, 4)]`	24.5 µs	31.3 µs	-21.55%
❌	Simulation	`decompress[u8, (10000, 4)]`	96 µs	111.6 µs	-13.93%
❌	Simulation	`decompress_utf8[(1000, 4)]`	37.5 µs	43.5 µs	-13.71%
❌	Simulation	`take_indices[(10000, 256)]`	57.3 µs	64.1 µs	-10.63%
❌	Simulation	`take_indices[(1000, 4)]`	49.2 µs	54.7 µs	-10.03%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing claude/codspeed-flaky-benchmarks-4vapbm (1902072) with develop (b7b01d3)}

46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 · 2026-07-20T15:41:00Z

On the 6 "regressions" in the CodSpeed report: they are the expected one-time step change from the allocator swap, not real regressions.

All 6 are in run_end_compress.rs — the binary that switched to mimalloc. CodSpeed's Simulation mode counts instructions for the whole trace, allocator included, and the decompress/take benchmarks also allocate inside the timed region; their traces now include mimalloc's allocation path (more instructions on these sizes) instead of glibc's. No library code changed, so nothing actually got slower — the values just move to a new baseline once.

#8742 had the same ±10–20% level shifts on benchmarks sharing files with the fixed ones and was resolved by a one-time acknowledgment on the dashboard. After that, every PR compares mimalloc-vs-mimalloc, and these benchmarks measure only Vortex changes — that stability is the point of the swap.

Generated by Claude Code

One divan bench_refs iteration carries a fixed ~146 instructions of harness. Where the measured operation costs less than that, the reported figure tracks the harness rather than the operation, and a small change in harness cost swings it past CodSpeed s 10% flag threshold with the measured code untouched. Per-iteration instruction counts (callgrind, the same instrument class as CodSpeed Simulation) confirm the flagged benchmarks are exactly the overhead-dominated ones: true_count_vortex_buffer 1024: 187 Ir (78% harness) <- flagged 2048: 228 Ir (64% harness) <- flagged 16384: 802 Ir (18% harness) 65536: 2770 Ir (5% harness) The fit is linear at ~0.04 Ir/bit with a ~146 Ir intercept, and no report ever flagged 16384 or 65536. This also explains why #8861 did not settle it: dropping 128 (97% harness) just promoted 1024 to worst. - true_count: keep 16384 and 65536, where the popcount dominates. - slice: BitBuffer::slice only adjusts an offset, a length and a refcount, so cost is independent of length -- every size measured exactly 125 Ir, which is why CodSpeed reported identical values for [1024] and [16384] on #8749. Measure one length with enough slices per iteration to dominate; varying the offset keeps them from folding together. Per-slice cost is 144 Ir, 1.3% harness. - binary_search: one search over 65536 elements is ~16 comparisons, and at ~71% harness the two implementations measured 203 vs 206 Ir -- indistinguishable, defeating the comparison the benchmark exists for. Searching 64 targets per iteration gives 1.0-1.5% harness and separates them: 15133 Ir vortex vs 10010 Ir std. varbinview_compact was left alone: compact[(4096, 90)] is 1069 Ir, only 14% harness, so its reports belong to the comparison-base problem fixed in the preceding commit rather than to harness overhead. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P

One divan bench_refs iteration carries a fixed ~146 instructions of harness. Where the measured operation costs less than that, the reported figure tracks the harness rather than the operation, and a small change in harness cost swings it past CodSpeed's 10% flag threshold with the measured code untouched. Per-iteration instruction counts (callgrind, the same instrument class as CodSpeed Simulation) confirm the flagged benchmarks are exactly the overhead-dominated ones: true_count_vortex_buffer 1024: 187 Ir (78% harness) <- flagged 2048: 228 Ir (64% harness) <- flagged 16384: 802 Ir (18% harness) 65536: 2770 Ir (5% harness) The fit is linear at ~0.04 Ir/bit with a ~146 Ir intercept, and no report ever flagged 16384 or 65536. This also explains why #8861 did not settle it: dropping 128 (97% harness) just promoted 1024 to worst offender. - true_count: keep 16384 and 65536, where the popcount dominates. - slice: BitBuffer::slice only adjusts an offset, a length and a refcount, so its cost is independent of length -- every size measured exactly 125 Ir, which is why #8749 reported identical values for [1024] and [16384]. Measure one length with enough slices per iteration to dominate, varying the offset so they cannot be folded together. No black_box is needed: the slice clones an Arc, and that refcount traffic is a side effect the optimiser cannot remove. - binary_search: one search over 65536 elements is ~16 comparisons, and at ~71% harness the two implementations measured 203 vs 206 Ir -- indistinguishable, defeating the comparison the benchmark exists for. Searching 64 targets per iteration separates them: 15133 Ir for vortex against 10010 for std. varbinview_compact was left alone: compact[(4096, 90)] is 1069 Ir, only 14% harness, so its reports belong to the comparison-base problem fixed in the preceding commit rather than to harness overhead. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P

claude added 3 commits July 20, 2026 15:30

connortsui20 added the changelog/ci label Jul 20, 2026 — with Claude

connortsui20 enabled auto-merge (squash) July 20, 2026 15:44

connortsui20 requested review from joseph-isaacs and robert3005 July 20, 2026 15:44

robert3005 approved these changes Jul 20, 2026

View reviewed changes

connortsui20 merged commit 5b688a9 into develop Jul 20, 2026
80 of 82 checks passed

connortsui20 deleted the claude/codspeed-flaky-benchmarks-4vapbm branch July 20, 2026 22:14

connortsui20 mentioned this pull request Jul 27, 2026

Fix the two root causes of CodSpeed flakes: substituted baselines and harness-dominated benchmarks #9011

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix remaining flaky CodSpeed microbenchmarks - #8861

Fix remaining flaky CodSpeed microbenchmarks#8861
connortsui20 merged 3 commits into
developfrom
claude/codspeed-flaky-benchmarks-4vapbm

connortsui20 commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

codspeed-hq Bot commented Jul 20, 2026

Uh oh!

connortsui20 commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

connortsui20 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Jul 20, 2026

Benchmarks: Vortex queries 📖

Uh oh!

codspeed-hq Bot commented Jul 20, 2026

Merging this PR will degrade performance by 16.43%

Performance Changes

Footnotes

Uh oh!

connortsui20 commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

connortsui20 commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading