Skip to content

Use PiecewiseSequence indices for FSL take#8800

Draft
danking wants to merge 7 commits into
codex/piecewise-sequential-arrayfrom
codex/piecewise-sequential-take
Draft

Use PiecewiseSequence indices for FSL take#8800
danking wants to merge 7 commits into
codex/piecewise-sequential-arrayfrom
codex/piecewise-sequential-take

Conversation

@danking

@danking danking commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Ports the TakeSlices-style optimization to Take(elements, PiecewiseSequenceArray) instead of introducing a value-wrapper API.

  • FSL take now builds one start per selected list, a constant list length, and a constant multiplier of 1, then calls elements.take(piecewise_indices).
  • Primitive, bool, decimal, VarBinView, and VarBin take paths detect PiecewiseSequenceArray indices and gather contiguous ranges only when multipliers is a constant unsigned 1; non-unit multipliers fall back through normal materialized-index take.
  • Shared PiecewiseSequence execution handles general start + j * multiplier sequences, while the current optimized consumers stay explicitly contiguous.

Validation

  • cargo fmt -p vortex-array
  • cargo test -p vortex-array piecewise_sequence
  • cargo check -p vortex-array
  • cargo check -p vortex-array --benches
  • git diff --check

@codspeed-hq

codspeed-hq Bot commented Jul 16, 2026

Copy link
Copy Markdown

Merging this PR will regress 2 benchmarks

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 19 improved benchmarks
❌ 2 regressed benchmarks
✅ 1649 untouched benchmarks
🆕 128 new benchmarks
⏩ 52 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decode_varbin[(1000, 4)] 26.1 µs 34.1 µs -23.32%
Simulation true_count_vortex_buffer[128] 522.2 ns 580.6 ns -10.05%
Simulation take_fsl_random[1024, 1000] 15.2 ms 6 ms ×2.5
Simulation take_fsl_nullable_random[1024, 1000] 14.8 ms 6 ms ×2.5
Simulation take_fsl_random[256, 1000] 3.5 ms 1.6 ms ×2.2
Simulation take_fsl_nullable_random[256, 1000] 3.4 ms 1.6 ms ×2.1
Simulation take_fsl_random[1024, 100] 1,567.8 µs 799.3 µs +96.14%
Simulation take_fsl_nullable_random[1024, 100] 1,535.6 µs 811.9 µs +89.12%
Simulation copy_nullable[65536] 1,027.9 µs 573.6 µs +79.2%
Simulation take_fsl_random[256, 100] 418.1 µs 239 µs +74.92%
Simulation take_fsl_random[64, 1000] 815.6 µs 466.6 µs +74.79%
Simulation take_fsl_nullable_random[64, 1000] 811.9 µs 477.3 µs +70.11%
Simulation take_fsl_nullable_random[256, 100] 415.5 µs 252.1 µs +64.82%
Simulation take_fsl_nullable_random[16, 1000] 257.6 µs 199.3 µs +29.28%
Simulation take_fsl_random[16, 1000] 237.7 µs 189.4 µs +25.55%
Simulation take_fsl_random[4096, 1000] 67.3 ms 54.2 ms +24.16%
Simulation take_fsl_random[64, 100] 123 µs 99.3 µs +23.8%
Simulation take_fsl_nullable_random[4096, 1000] 65.6 ms 54.3 ms +20.89%
Simulation take_fsl_nullable_random[64, 100] 131.5 µs 111.9 µs +17.51%
Simulation copy_non_nullable[65536] 908.7 µs 794.3 µs +14.4%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing codex/piecewise-sequential-take (d694338) with codex/piecewise-sequential-array (2b6fa3e)

Open in CodSpeed

Footnotes

  1. 52 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@danking danking force-pushed the codex/piecewise-sequential-array branch from aba7e4d to 3edf6c8 Compare July 16, 2026 18:06
@danking danking force-pushed the codex/piecewise-sequential-take branch from 07fde63 to 3ecd955 Compare July 16, 2026 18:06
@danking danking force-pushed the codex/piecewise-sequential-array branch from 3edf6c8 to 1748e31 Compare July 16, 2026 18:20
@danking danking force-pushed the codex/piecewise-sequential-take branch from 3ecd955 to 98a9180 Compare July 16, 2026 18:20
@danking danking force-pushed the codex/piecewise-sequential-take branch from 01e95cb to eb83451 Compare July 16, 2026 19:19
@danking danking changed the title Use PiecewiseSequential indices for FSL take Use PiecewiseSequence indices for FSL take Jul 16, 2026
@danking danking force-pushed the codex/piecewise-sequential-take branch from eb83451 to e79ec4b Compare July 16, 2026 20:29
S: crate::dtype::UnsignedPType,
L: crate::dtype::UnsignedPType,
{
validate_index_ranges(source.len(), starts, lengths, output_len)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the slice will validate them

array.buffer::<D>().as_slice(),
indices_ref.len(),
)?;
let validity = array.validity()?.take(indices_ref)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eh, we need my refactor pr to extract the ref from view

L: crate::dtype::UnsignedPType,
T: NativeDecimalType,
{
validate_index_ranges(values.len(), starts, lengths, output_len)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is already validated by the result slice

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so imho you either keep this and convert the loop below all into unsafe accesses or remove this and use the optional getters on slice

for (&start, &length) in starts.iter().zip_eq(lengths) {
let start = start.as_();
let length = length.as_();
result.extend_from_slice(&values[start..start + length]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be better if you had the mutable result and then wrote directly into the next available range result[curr..][..length]


let (new_elements, new_len) =
take_non_empty_non_degenerate_elements::<I>(array, indices_array, ctx)?;
let new_validity = if array.dtype().is_nullable() || indices_nullability.is_nullable() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think validity take already handles this

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have to audit the validation from the compute and verify that we do not do the work twice. We have to audit exactly but ideally we'd not validate and figure out how to make operations safe by default even given bad input.

@danking danking force-pushed the codex/piecewise-sequential-array branch from 6eb0c92 to 16ca5dc Compare July 16, 2026 23:36
@danking danking force-pushed the codex/piecewise-sequential-take branch from b615895 to 6dcc52a Compare July 16, 2026 23:41
@danking danking added the changelog/performance A performance improvement label Jul 16, 2026
@danking danking force-pushed the codex/piecewise-sequential-take branch from 6dcc52a to a2cbdf0 Compare July 17, 2026 00:15
danking added 7 commits July 16, 2026 20:17
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking danking force-pushed the codex/piecewise-sequential-take branch from a2cbdf0 to d694338 Compare July 17, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants