feat(btrblocks): compress binary arrays with VarBin offsets and FSST

[joseph-isaacs]

Summary

Binary columns had exactly one scheme in ALL_SCHEMES: BinaryDictScheme. Once the dictionary declined a column, nothing else was eligible, so the array fell through to a canonical VarBinViewArray written as payload plus a fixed 16 bytes per element of opaque views buffer — regardless of content.

Two consequences, both measurable:

• Size became insensitive to the data. A binary column that zlib -9 shrinks 6.8x was written at exactly the same size as an incompressible one, because no scheme ran on either.
• The per-element overhead was a flat 16.0 B/value independent of width, so a 16-byte column doubled (2.00x payload) and a 4-byte column quintupled (5.01x).

For comparison, the same values typed as Utf8 compressed to 713,324 bytes where Binary produced 3,204,100 — a 4.5x gap decided purely by dtype.

This PR gives binary two schemes that already existed in spirit elsewhere in the compressor, and adds the tests to pin their behavior.

Changes

VarBinScheme (new, schemes/binary/varbin.rs) re-encodes the canonical VarBinViewArray as a VarBinArray, replacing the opaque views buffer with an offsets child array. That child is then compressed by the ordinary integer schemes, so a fixed-width column's constant-stride offsets collapse to a sequence and the per-element overhead disappears. This mirrors what FSSTScheme already does for strings, which builds its codes as a VarBinArray with compressed offsets.

FSSTScheme now matches binary as well as utf8. Its matches gated on is_utf8(), but the compress path never validates UTF-8 — it trains and compresses over the raw array_as_varbinview() bytes. The gate excluded binary from a scheme that already worked on it, leaving any binary payload with intra-value structure (shared prefixes, zero padding, common framing) uncompressed once the dictionary declined it.

The two are complementary rather than competing, and selection picks correctly between them: FSST wins where the payload has structure, VarBinScheme wins on incompressible data where FSST's symbol table is pure overhead.

Measured at 100k rows, compressed nbytes against the same compressor with these schemes excluded:

case	before	after	selected
shared prefix	3,200,000	734,685	FSST
nulls every 7th	2,966,827	690,230	FSST
random 16B	3,200,000	1,600,000	VarBin
random 256B	27,200,000	25,600,000	VarBin

The random 16B case is now at its payload floor (1,600,000 bytes of incompressible data) with no metadata overhead.

Tests (tests/varbin_scheme.rs) cover both schemes with roundtrip assertions through assert_arrays_eq!, including a nullable case, plus two invariants: enabling VarBinScheme must never grow the output, and identical values must compress identically whether typed binary or utf8 (FSST compresses bytes, not codepoints).

One golden snapshot moves, and it improves: binary_low_cardinality dictionary values go from 96 to 52 bytes as VarBinScheme cascades into the dictionary's values child.

Review notes

• VarBinScheme is registered in ALL_SCHEMES, so it affects every binary column on a default write. Worth confirming that blast radius is intended rather than gating it behind with_compact.
• On the FSST gate: compress demonstrably does not validate UTF-8, but I have not read through fsst_train_compressor/fsst_compress. The byte-identical binary/utf8 assertion is evidence the widening is safe, not proof. If the original restriction was deliberate for a reason deeper in the FSST implementation, that is the part to push back on.

Checks

• cargo test -p vortex-btrblocks — all pass
• cargo test -p vortex-file — all pass
• cargo clippy -p vortex-btrblocks --all-targets --all-features — clean
• cargo +nightly fmt --all

Not measured: decode throughput for either scheme, and the file-level (rather than in-memory nbytes) size effect. Both are worth a follow-up before leaning on these numbers for anything latency-sensitive.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GJdgPga43u5rXYwv6R5b19

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 2184 untouched benchmarks
⏩ 218 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	decompress[u64, (4000, 1024)]	70.8 µs	86.3 µs	-17.87%
⚡	WallTime	arrow_checked_add_u32_neon[16384]	20.4 µs	12.4 µs	+64.99%
⚡	Simulation	take[duplicates/repeated/primitive/nonnull/chunks=16/indices=1000]	240.3 µs	199.3 µs	+20.58%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/binary-compression-schemes (bac0b7f) with develop (bffdca1)}

Footnotes

1. 218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

[joseph-isaacs]

Brought this up to date with develop (clean merge, no conflicts) and fixed the three CI failures from the last run, all in vortex-btrblocks tests:

• golden_compact / golden_unstable: the binary_low_cardinality snapshots for those two variants were never regenerated. Both now pick vortex.varbin for the dictionary values child and both get smaller (compact 6199 → 6196 replacing zstd, unstable 6240 → 6196 replacing varbinview).
• fsst_versus_varbin_on_identical_bytes: the binary-vs-utf8 equality assertion failed because OnPairScheme (utf8-only) wins the utf8 column under unstable_encodings, not because FSST behaves differently. The test now excludes OnPair so both dtypes go through FSST and the bytes-not-codepoints invariant is what's actually checked. Sizes are identical for both cases with that change.

Also resolved the earlier per-element copy thread, which 54fb31b already addressed via VarBinBuilder.

Verified locally: cargo test -p vortex-btrblocks with and without --all-features, cargo clippy -p vortex-btrblocks --all-targets --all-features -D warnings, and nightly rustfmt on the changed files. The CodSpeed regression is on words_gather_dispatch_avx2 in vortex-buffer, which this PR doesn't touch.

---

Generated by Claude Code

[joseph-isaacs]

Two more pushes since the last update:

• Rust (docs) was failing on an unresolved intra-doc link to VarBinViewArray in the VarBinScheme module docs (-D rustdoc::broken_intra_doc_links). Fixed by linking the full path; RUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc -p vortex-btrblocks --no-deps --document-private-items now passes locally.
• DCO flagged the four original commits: three because the sign-off name was quoted ("Claude" vs Claude), and 54fb31b because it is authored by Joe Isaacs but signed off as Claude. I added the DCO remediation commit for the three Claude-authored ones.

Remaining blocker (needs the author): 54fb31b still needs a sign-off from its author. Pushing this remediation commit to the branch will clear it without rewriting history:

DCO Remediation Commit for Joe Isaacs <joe.isaacs@live.co.uk>

I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 54fb31beee8c3cfe06633e7cb6f0e4ab1a568924

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Everything else is CI on the new head, which I'm watching.

---

Generated by Claude Code