Skip to content

[wip] PatchedArray wiring: Part 2#7145

Draft
a10y wants to merge 19 commits intoaduffy/patched-arrayfrom
aduffy/lazy-patches
Draft

[wip] PatchedArray wiring: Part 2#7145
a10y wants to merge 19 commits intoaduffy/patched-arrayfrom
aduffy/lazy-patches

Conversation

@a10y
Copy link
Contributor

@a10y a10y commented Mar 24, 2026

Summary

Part of #6535 epic and successor to #7002

This wires PatchedArray through to BitPackedArray, replacing the interior Patches entirely. This change is backward compatible.

API Changes

  1. Updates VTable::build to return ArrayRef instead of V::Array. This allows us to lazily deserialize a previously written value as a new type in a backward-compatible way
  2. Adds a new LazyPatchedArray which is just a dummy node that holds on to an array and some Patches, has no reductions and can just be executed into a transposed PatchedArray
  3. Removes the patches child from BitPackedArray
  4. Removes the BitPackedArray::encode() and a mess of other functions, instead moving everything into a new BitPackedEncoder fluent API

Testing

Reuses our existing and extensive test suite for BP

@codspeed-hq
Copy link

codspeed-hq bot commented Mar 24, 2026

Merging this PR will degrade performance by 18.62%

⚡ 1 improved benchmark
❌ 4 regressed benchmarks
✅ 423 untouched benchmarks
⏩ 2200 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[alp_for_bp_f64] 2.4 ms 1.7 ms +45.72%
Simulation compress_rd[f32, 10000] 378.7 µs 464.6 µs -18.5%
Simulation compress_rd[f64, 100000] 4.6 ms 5.5 ms -15.59%
Simulation compress_rd[f32, 100000] 3.6 ms 4.4 ms -18.62%
Simulation compress_rd[f64, 10000] 489.8 µs 578.5 µs -15.32%

Comparing aduffy/lazy-patches (e658811) with aduffy/patched-array (e372d98)

Open in CodSpeed

Footnotes

  1. 2200 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@a10y a10y force-pushed the aduffy/lazy-patches branch 2 times, most recently from 1d28e78 to 65d8c10 Compare March 24, 2026 19:06
@a10y a10y force-pushed the aduffy/patched-array branch from 860cc64 to 042394b Compare March 24, 2026 19:17
@a10y a10y force-pushed the aduffy/lazy-patches branch 3 times, most recently from c76f6bd to 93e2785 Compare March 24, 2026 19:30
@a10y a10y changed the title [wip] BP remove interior Patches [wip] PatchedArray wiring: Part 2 Mar 24, 2026
@a10y a10y force-pushed the aduffy/lazy-patches branch from 93e2785 to dd20d70 Compare March 24, 2026 19:37
@a10y a10y force-pushed the aduffy/patched-array branch from 042394b to aa8f05f Compare March 24, 2026 19:48
@a10y a10y force-pushed the aduffy/lazy-patches branch 2 times, most recently from b7c14f1 to 69ce351 Compare March 25, 2026 20:26
a10y added 10 commits March 26, 2026 15:17
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
We call append_to_builder on the inner first, then just do a single pass
and overwrite what it just wrote.

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
@a10y a10y force-pushed the aduffy/patched-array branch from 5d0a954 to df8dbf5 Compare March 26, 2026 19:50
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
@a10y a10y force-pushed the aduffy/patched-array branch from df8dbf5 to 862d814 Compare March 26, 2026 19:54
a10y added 5 commits March 26, 2026 15:58
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
this lets us deserialize BPArray with Patches without eagerly
transposing

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
This lets us return something other than the original array encoding at
read time.

Currently we'll want this so that BitPacked::build returns a
LazyPatched, but this is applicable for pretty much any back-compat
preserving encoding rewrites.

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
removes both the patches field as well as all code for handling patches.
this is safe to do now that we have updated the VTable build function to
always read methods.

note that we need to leave the metadata as-is.

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
a10y added 2 commits March 26, 2026 16:06
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
@a10y a10y force-pushed the aduffy/lazy-patches branch from 69ce351 to 0d703d8 Compare March 26, 2026 20:29
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
@a10y a10y force-pushed the aduffy/lazy-patches branch from 0d703d8 to e658811 Compare March 26, 2026 20:59
@a10y a10y force-pushed the aduffy/patched-array branch from daf258f to 578d83d Compare March 27, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant