Skip to content

Speed up binary kernels (30% faster and and or), add BooleanBuffer::from_bitwise_binary_op #9090

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
alamb wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Speed up binary kernels (30% faster and and or), add BooleanBuffer::from_bitwise_binary_op #9090

alamb wants to merge 6 commits into from
+169 −20

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Jan 2, 2026 •
edited
Loading

Which issue does this PR close?

This is the next step after

Rationale for this change

Also, it is hard to find the code to create new Buffers by applying bitwise unary operations.

What changes are included in this PR?

  • Introduce optimized BooleanBuffer::from_bitwise_binary
  • Migrate several kernels that use bitwise_bin_op_helper to use the new BooleanBuffer

Are these changes tested?

Yes new tests are added

Performance results show 30% performance improvement for the and and or kernels for aligned buffers (common case)

Are there any user-facing changes?

A new API

@github-actions github-actions bot added the arrow Changes to the arrow crate label Jan 2, 2026
alamb
alamb commented Jan 2, 2026
View reviewed changes
arrow-buffer/src/buffer/boolean.rs
/// );
/// assert_eq!(result.inner().as_slice(), &[0b10101110u8, 0b00001111u8]);
/// ```
pub fn from_bitwise_binary_op<F>(
Copy link
Contributor Author

@alamb alamb Jan 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the new API

@alamb alamb force-pushed the alamb/boolean_kernel branch from 9d87a11 to 577dfa8 Compare January 2, 2026 14:30
alamb
alamb commented Jan 2, 2026
View reviewed changes
arrow-buffer/src/buffer/ops.rs
/// Apply a bitwise not to one input and return the result as a Buffer.
/// The input is treated as a bitmap, meaning that offset and length are specified in number of bits.
pub fn buffer_unary_not(left: &Buffer, offset_in_bits: usize, len_in_bits: usize) -> Buffer {
// TODO: should we deprecate this function in favor of the Buffer ! impl ?
Copy link
Contributor Author

@alamb alamb Jan 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no Buffer impl of ! (there is an implementation in BooleanBuffer)

@alamb
Copy link
Contributor Author

alamb commented Jan 2, 2026

run benchmark boolean_kernels

@alamb-ghbot
Copy link

alamb-ghbot commented Jan 2, 2026

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/boolean_kernel (577dfa8) to 44d4c90 diff
BENCH_NAME=boolean_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench boolean_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_boolean_kernel
Results will be posted here when complete

This was referenced Jan 2, 2026
Closed
Closed
Closed
Closed
@alamb-ghbot
Copy link

alamb-ghbot commented Jan 2, 2026

🤖: Benchmark completed

Details 

group            alamb_boolean_kernel                   main
-----            --------------------                   ----
and              1.00    208.4±2.61ns        ? ?/sec    1.31    273.8±4.63ns        ? ?/sec
and_sliced_1     1.01   1101.0±3.25ns        ? ?/sec    1.00   1094.0±7.12ns        ? ?/sec
and_sliced_24    1.05    250.9±1.84ns        ? ?/sec    1.00    239.4±2.17ns        ? ?/sec
not              1.02    148.5±3.30ns        ? ?/sec    1.00    145.6±0.72ns        ? ?/sec
not_slice_24     1.02    198.9±7.14ns        ? ?/sec    1.00    194.4±4.51ns        ? ?/sec
not_sliced_1     1.01    623.8±9.34ns        ? ?/sec    1.00    620.0±6.84ns        ? ?/sec
or               1.00    197.8±1.30ns        ? ?/sec    1.28    252.9±2.20ns        ? ?/sec
or_sliced_1      1.00  1106.9±50.01ns        ? ?/sec    1.11  1228.1±13.41ns        ? ?/sec
or_sliced_24     1.02    248.3±1.07ns        ? ?/sec    1.00    242.4±4.02ns        ? ?/sec

@alamb alamb changed the title Speed up binary kernels, add BooleanBuffer::from_bitwise_binary_op Speed up binary kernels (30% faster and and or), add BooleanBuffer::from_bitwise_binary_op Jan 2, 2026
@alamb alamb marked this pull request as ready for review January 5, 2026 12:01
@alamb alamb marked this pull request as draft January 5, 2026 12:01
@alamb alamb marked this pull request as ready for review January 5, 2026 19:54
Dandandan
Dandandan reviewed Jan 7, 2026
View reviewed changes
arrow-buffer/src/buffer/boolean.rs Outdated
.iter()
.zip(right_chunks.iter())
.map(|(left, right)| op(left, right));
// Soundness: `BitChunks` is a `BitChunks` iterator which
Copy link
Contributor

@Dandandan Dandandan Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trusted Len ?

Copy link
Contributor Author

@alamb alamb Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in a22ad8d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Dandandan Dandandan Dandandan approved these changes

Assignees

No one assigned

Labels

arrow Changes to the arrow crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alamb @alamb-ghbot @Dandandan