Skip to content

Support reporting statistics and currently written byte size from writers#8726

Open
robert3005 wants to merge 3 commits into
developfrom
rk/writeapi
Open

Support reporting statistics and currently written byte size from writers#8726
robert3005 wants to merge 3 commits into
developfrom
rk/writeapi

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Allow users to extract the whole file stats from the write instead of having to
read the files back

@codspeed-hq

codspeed-hq Bot commented Jul 12, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚡ 4 improved benchmarks
❌ 4 regressed benchmarks
✅ 1656 untouched benchmarks
⏩ 42 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation encode_varbin[(1000, 8)] 138.9 µs 155.6 µs -10.79%
Simulation encode_varbin[(1000, 4)] 138.7 µs 154.6 µs -10.3%
Simulation encode_varbin[(1000, 32)] 143.9 µs 160.4 µs -10.29%
Simulation encode_varbin[(1000, 2)] 138.3 µs 154 µs -10.24%
Simulation bitwise_not_vortex_buffer_mut[128] 244.4 ns 186.1 ns +31.34%
Simulation bitwise_not_vortex_buffer_mut[1024] 304.7 ns 246.4 ns +23.68%
Simulation bitwise_not_vortex_buffer_mut[2048] 398.6 ns 340.3 ns +17.14%
Simulation eq_i64_constant 298 µs 268.7 µs +10.9%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/writeapi (9211c32) with develop (002e40a)

Open in CodSpeed

Footnotes

  1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

robert3005 and others added 3 commits July 13, 2026 20:28
Expose currently written byte size to writers and through vortex-jni

Signed-off-by: Robert Kruszewski <github@robertk.io>
Replace the top-level-only column size computation with a single walk
of the layout tree that attributes each segment to the deepest field
path referencing it, exposed as Footer::compressed_field_sizes.
WriteSummary::compressed_column_sizes is now a thin schema-order
wrapper over it.

Signed-off-by: "Robert Kruszewski" <github@robertk.io>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JE2aKEzyogTh1dP126gudc
Construct the counting write before spawning the JNI write task and
retrieve the shared counter directly via counter(), instead of
threading a pre-allocated counter through with_counter. Sink creation
now happens synchronously in NativeWriter.create, surfacing IO errors
there instead of at close.

Signed-off-by: "Robert Kruszewski" <github@robertk.io>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JE2aKEzyogTh1dP126gudc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant