Skip to content

Fix flaky CodSpeed microbenchmarks#8742

Merged
robert3005 merged 10 commits into
developfrom
claude/flaky-microbenchmarks-cleanup-u68he8
Jul 14, 2026
Merged

Fix flaky CodSpeed microbenchmarks#8742
robert3005 merged 10 commits into
developfrom
claude/flaky-microbenchmarks-cleanup-u68he8

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 13, 2026

Copy link
Copy Markdown
Member

A small pool of microbenchmarks flips ±10-35% between two fixed values on unrelated PRs (including docs-only and lockfile-only changes), spamming every CodSpeed report. This PR fixes them (and removes only 1 benchmark). No #[cfg(codspeed)] gating; CI and local runs stay identical. One commit per benchmark.

Changes

  • mimalloc as global allocator in the 5 flaky vortex-array bench files: chunk_array_builder, dict_compress, varbinview_compact, compare, binary_ops
  • bitwise_not_vortex_buffer_mut: drop the 128/1024/2048 sizes (measured only harness overhead)
  • slice_empty_vortex: rewrite as a 1024-iteration tight loop, renamed slice_empty_tight_loop_vortex
  • rebuild_naive (vortex-zstd): the one benchmark removed instead of fixed — its cost is dominated by zstd-internal copies (glibc ifunc-resolved memcpy) that no bench-level change can stabilize, its fixture is degenerate (a 4-string dictionary with all-zero offsets), and ListView rebuild is already benchmarked across element types and list shapes in vortex-array/benches/listview_rebuild.rs. The crate's now-unused divan dev-dependency goes with it.
Which benchmarks were flaky, and the evidence

Identified by reading the CodSpeed comments on the ~47 PRs merged since June 25 (post-#8490). The tell: the same benchmark flipping between the same two values, in both directions, on PRs that can't have affected it — including deny.toml-only (#8712, #8716), uv.lock-only (#8732), docs-only (#8737, #8728, #8685), and CI-YAML-only (#8660, #8683) changes.

Benchmark Evidence
bitwise_not_vortex_buffer_mut[128/1024/2048] ~half of all PRs — worst offender
chunked_varbinview_* ×4 ~20 PRs, both directions
chunked_bool_canonical_into[(1000,10)] ~2× flips (16µs ↔ 35µs) on 4 PRs
encode_varbin, encode_varbinview ~12 PRs; encode_varbinview[(10000,4)] also flipped on an earlier revision of this PR
compact, compact_sliced (90%-utilization args) ~8 PRs
compare_int_constant ±11.1% verbatim on ≥9 PRs
eq_i64_constant same ±11% signature incl. docs-only PR
slice_empty_vortex -14.66% verbatim on ~13 PRs
rebuild_naive (vortex-zstd) ~10 PRs, both directions

Watch list (left alone, below the ≥3-independent-sightings bar): copy_nullable/copy_non_nullable[65536] in cast_decimal.rs, true_count_vortex_buffer[128].

Root causes and why each fix matches
  • Allocation in the timed region → glibc malloc's code differs across CodSpeed runner images, so alloc-heavy benchmarks trace differently for byte-identical Vortex code. Vendored mimalloc removes glibc malloc from the trace. Empirical support: the only three bench files already using mimalloc (single_encoding_throughput, common_encoding_tree_throughput, row_encode) are the most alloc-heavy suites in the repo and were never flagged once in the 47-PR window.
  • Sub-microsecond work → the measurement is fixed harness overhead plus binary code layout, which shifts with any unrelated change. Fix by making the operation dominate: the in-place NOT (no alloc, no copy) keeps only sizes where the loop dominates; the empty slice runs 1024× per iteration, mirroring the neighboring slice_tight_loop_vortex.
  • Environment-bound and low-signalrebuild_naive, per the justification above: unfixable at the bench level and redundant with the vortex-array ListView rebuild suite, so removed.
Validation: A/A reruns and a stacked canary PR
  • Expected one-time step changes on this PR: swapping the allocator changes the trace of every benchmark in the touched files, so this PR's report shows a few ±10-20% level shifts (including on never-flaky encode_primitives, which just shares a file). These need a one-time acknowledgment on the CodSpeed dashboard; after merge every PR compares mimalloc-vs-mimalloc.
  • A/A reruns (same bench binaries measured three times, on separate runners and commits): every value reproduced exactly — 211.5µs, 137.1µs, 14.6µs, 26.3µs — with zero new flags across 1656 benchmarks.
  • Canary test: canary for the flaky-benchmark fixes in #8742 (do not merge) #8743 (a Better error message in case of unknown encoding id #8681-style cold-string change stacked on this branch — the class of change that used to collect five false flags): Performance Gate Passed, ✅ 1660 untouched, zero changes reported.

No public API changes; benchmark-only.

https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC

@codspeed-hq

codspeed-hq Bot commented Jul 13, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 1656 untouched benchmarks
🆕 1 new benchmark
⏩ 47 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_into_canonical[(1000, 10)] 169.5 µs 211.5 µs -19.86%
Simulation encode_primitives[i64, (1000, 512)] 117.3 µs 137.1 µs -14.47%
Simulation chunked_bool_canonical_into[(1000, 10)] 16.4 µs 14.6 µs +12.43%
🆕 Simulation slice_empty_tight_loop_vortex N/A 26.3 µs N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/flaky-microbenchmarks-cleanup-u68he8 (1a06040) with develop (002e40a)

Open in CodSpeed

Footnotes

  1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

claude added 2 commits July 13, 2026 18:08
bitwise_not_vortex_buffer_mut[128/1024/2048] was the noisiest benchmark on
CodSpeed after #8490: it flipped between the same two values in both
directions (+27.1%/-10.66% etc.) on roughly half of ~47 recently merged PRs,
including PRs changing only docs, uv.lock, or workflow YAML.

The Not impl for an owned BitBufferMut runs bitwise_unary_op_mut in place -
no allocation, no memcpy. At 128-2048 bits that is a handful of word ops, so
the reported number is fixed divan harness overhead plus binary code layout,
both of which shift with any unrelated code change. Those sizes never
measured the operation at all, so remove them outright; 16384 and 65536,
where the loop dominates, were never flagged and remain.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
slice_empty_vortex measured a single Buffer::slice(8..8) - a few nanoseconds
of real work - so the reported ~340ns was dominated by fixed harness overhead
and it flipped -14.66%/-15.83% verbatim on at least a dozen recently merged
PRs that never touched vortex-buffer (docs-only, CI-YAML-only, issue-template
PRs among them).

Fix it the same way the file already handles cheap slices: run the operation
in a tight loop (1024 empty slices per iteration, mirroring
slice_tight_loop_vortex) so the slice cost dominates the measurement. Renamed
to slice_empty_tight_loop_vortex to match its sibling and so CodSpeed treats
it as a new benchmark rather than reporting a bogus step change on the old
name.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
@connortsui20 connortsui20 force-pushed the claude/flaky-microbenchmarks-cleanup-u68he8 branch from e74ee1a to daeb6c0 Compare July 13, 2026 18:19
@connortsui20 connortsui20 changed the title Gate flaky benchmarks from CodSpeed CI Fix or remove flaky CodSpeed microbenchmarks Jul 13, 2026
claude added 6 commits July 13, 2026 18:22
The chunked_varbinview_{canonical_into,into_canonical,opt_canonical_into,
opt_into_canonical} benches were flagged on ~20 of ~47 recently merged PRs
and chunked_bool_canonical_into[(1000, 10)] flipped ~2x (16us <-> 35us) on
four more, always between the same two values and always on PRs that never
touched canonicalization (deny.toml-only and uv.lock-only PRs included).

These benches allocate and grow builder buffers inside the timed region, so
the simulated trace runs through glibc malloc, whose code differs across
CodSpeed runner images. Routing allocation through mimalloc removes glibc
malloc from the trace entirely: the allocator is vendored and identical in
every build. The three bench files that already use mimalloc
(single_encoding_throughput, common_encoding_tree_throughput, row_encode)
are the most alloc-heavy suites in the repo and were never flagged once in
the same 47-PR window, which is strong evidence this is the right fix. Also
what docs/developer-guide/benchmarking.md already prescribes for
allocation-heavy benchmarks.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
encode_varbin[(1000, *)] flipped between ~138us and ~155-160us on about a
dozen recently merged PRs (including docs-only #8737 and uv.lock-only #8732),
encode_varbinview[(10000, 2)] on several more, and encode_varbinview
[(10000, 4)] flipped -10.1% on an earlier revision of this very PR - which
does not touch dict encoding.

String dict encoding is hash-table growth plus builder-buffer growth, i.e.
allocation-dominated, so the trace inherits glibc malloc's per-runner-image
code differences. Route it through vendored mimalloc, as the never-flagged
alloc-heavy bench suites already do.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
compact[(4096, 90)] and compact_sliced[(4096, 90)] (occasionally the 16384
variants) flipped +/-10-16% between the same two values on ~8 recently merged
PRs, including CI-YAML-only and FFI-only changes. At 90% buffer utilization
compaction copies almost nothing, so the trace is mostly the allocation of
the compacted buffers - glibc malloc code that differs per runner image.
Route allocation through vendored mimalloc like the never-flagged
throughput suites.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
compare_int_constant reported a near-identical +/-11.1% flip on at least nine
recently merged PRs spanning docs-only, JNI-packaging, file-layout, and
DuckDB-binding changes, none of which touch the compare kernel. The timed
region allocates the output BoolArray buffer through glibc malloc, whose code
differs across CodSpeed runner images; the 65k-element compare loop itself is
compute-bound and stable. Route allocation through vendored mimalloc.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
eq_i64_constant showed the same ~+11% two-state flip as compare_int_constant,
most conclusively on docs-only #8737. Like compare.rs, the timed region
allocates the output array through glibc malloc while the kernel itself is
compute-bound, so route allocation through vendored mimalloc.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
rebuild_naive flipped between ~91us and ~105us (reported as -10% to -16%
regressions or +15% improvements depending on the baseline) on about ten
recently merged PRs that never touched vortex-zstd or ListView rebuild,
including a docs-only Spark PR (#8728) and a build.rs-only CUDA PR (#8708).

The benchmark cannot be stabilized: its trace is dominated by zstd's internal
window/output copies (glibc memcpy resolved per runner image via ifunc), not
by Vortex code. It is also of little value: the fixture is degenerate - a
4-string dictionary with all-zero offsets - and the rebuild paths it nominally
covers are benchmarked properly across element types and list shapes in
vortex-array/benches/listview_rebuild.rs. Remove it rather than keep a noise
source; the divan dev-dependency goes with it since this was the crate's only
benchmark.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
@connortsui20 connortsui20 force-pushed the claude/flaky-microbenchmarks-cleanup-u68he8 branch from daeb6c0 to a97990f Compare July 13, 2026 18:22
@connortsui20 connortsui20 marked this pull request as draft July 13, 2026 18:51
claude added 2 commits July 13, 2026 19:38
mimalloc added as a vortex-array dev-dependency; divan removed from
vortex-zstd along with its only benchmark.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
Same tree as the previous commit. If the mimalloc fix is sound, this run must
reproduce the previous run's values for the benchmarks it touched (e.g.
chunked_varbinview_into_canonical[(1000, 10)] at 211.5us); a flip would mean
the bimodality survives mimalloc and the benchmark should be removed instead.
This commit can be dropped before the PR leaves draft.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6PPcdrqcNeUkfi1EGd4oC
@connortsui20 connortsui20 force-pushed the claude/flaky-microbenchmarks-cleanup-u68he8 branch from 4efeb0d to 1a06040 Compare July 13, 2026 19:38
@connortsui20 connortsui20 changed the title Fix or remove flaky CodSpeed microbenchmarks Fix flaky CodSpeed microbenchmarks Jul 13, 2026
@connortsui20 connortsui20 marked this pull request as ready for review July 13, 2026 20:14

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's try this

@robert3005 robert3005 merged commit 5abaf98 into develop Jul 14, 2026
73 of 74 checks passed
@robert3005 robert3005 deleted the claude/flaky-microbenchmarks-cleanup-u68he8 branch July 14, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants