perf: improve calculating length performance for GenericByteArray in row conversion

[rluvaton]

Which issue does this PR close?

N/A

Rationale for this change

Making the row length calculation faster which result in faster row conversion

What changes are included in this PR?

Instead of iterating over the items in the array and getting the length from the byte slice, we use the offsets directly and zip with nulls if necessary

Are these changes tested?

Existing tests

Are there any user-facing changes?

Faster encoding

---

Split to 2 more PRs as the other 2 add a change to the public API

Related to:

• perf: improve calculating length performance for nested arrays in row conversion #9079
• perf: improve calculating length performance for view byte array in row conversion #9080

[rluvaton]

run benchmark row_format

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing improve-row-lengths-for-binary (ff5b8ab) to 843bee2 diff
BENCH_NAME=row_format
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench row_format
BENCH_FILTER=
BENCH_BRANCH_NAME=improve-row-lengths-for-binary
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                                                                                                         improve-row-lengths-for-binary         main
-----                                                                                                                         ------------------------------         ----
append_rows 10 large_list(0) of u64(0)                                                                                        1.00    626.3±6.97ns        ? ?/sec    1.04   651.6±12.20ns        ? ?/sec
append_rows 10 list(0) of u64(0)                                                                                              1.00    669.9±4.76ns        ? ?/sec    1.05   702.8±10.88ns        ? ?/sec
append_rows 4096 4096 string_dictionary(20, 0.5), string_dictionary(30, 0), string_dictionary(100, 0), i64(0)                 1.00    351.3±8.17µs        ? ?/sec    1.05    368.9±2.34µs        ? ?/sec
append_rows 4096 bool(0, 0.5)                                                                                                 1.41     12.1±0.12µs        ? ?/sec    1.00      8.6±0.04µs        ? ?/sec
append_rows 4096 bool(0.3, 0.5)                                                                                               1.00     14.9±0.66µs        ? ?/sec    1.14     17.0±0.15µs        ? ?/sec
append_rows 4096 i64(0)                                                                                                       1.00      7.7±0.18µs        ? ?/sec    1.00      7.7±0.20µs        ? ?/sec
append_rows 4096 i64(0.3)                                                                                                     1.00     11.1±0.08µs        ? ?/sec    1.37     15.3±0.14µs        ? ?/sec
append_rows 4096 large_list(0) of u64(0)                                                                                      1.00    158.2±1.60µs        ? ?/sec    1.03    163.1±1.13µs        ? ?/sec
append_rows 4096 large_list(0) sliced to 10 of u64(0)                                                                         1.00   897.6±30.55ns        ? ?/sec    1.02   912.3±11.26ns        ? ?/sec
append_rows 4096 list(0) of u64(0)                                                                                            1.00    150.3±2.54µs        ? ?/sec    1.10    165.6±1.29µs        ? ?/sec
append_rows 4096 list(0) sliced to 10 of u64(0)                                                                               1.00    979.8±7.02ns        ? ?/sec    1.08   1059.4±7.95ns        ? ?/sec
append_rows 4096 string view(1..100, 0)                                                                                       1.03    117.6±0.93µs        ? ?/sec    1.00    114.5±0.50µs        ? ?/sec
append_rows 4096 string view(1..100, 0.5)                                                                                     1.01    103.4±0.74µs        ? ?/sec    1.00    102.6±0.93µs        ? ?/sec
append_rows 4096 string view(10, 0)                                                                                           1.00     44.4±0.67µs        ? ?/sec    1.17     51.9±0.23µs        ? ?/sec
append_rows 4096 string view(100, 0)                                                                                          1.04     78.5±1.13µs        ? ?/sec    1.00     75.6±1.01µs        ? ?/sec
append_rows 4096 string view(100, 0.5)                                                                                        1.00     85.6±0.70µs        ? ?/sec    1.01     86.2±0.38µs        ? ?/sec
append_rows 4096 string view(30, 0)                                                                                           1.00     47.9±0.43µs        ? ?/sec    1.13     54.1±0.61µs        ? ?/sec
append_rows 4096 string(10, 0)                                                                                                1.00     36.1±0.13µs        ? ?/sec    1.35     48.8±0.37µs        ? ?/sec
append_rows 4096 string(100, 0)                                                                                               1.00     68.7±2.88µs        ? ?/sec    1.04     71.3±1.64µs        ? ?/sec
append_rows 4096 string(100, 0.5)                                                                                             1.11     91.1±2.23µs        ? ?/sec    1.00     81.9±1.48µs        ? ?/sec
append_rows 4096 string(20, 0.5), string(30, 0), string(100, 0), i64(0)                                                       1.00    211.5±3.56µs        ? ?/sec    1.04    220.6±3.80µs        ? ?/sec
append_rows 4096 string(30, 0)                                                                                                1.00     39.1±0.50µs        ? ?/sec    1.27     49.5±0.47µs        ? ?/sec
append_rows 4096 string_dictionary(10, 0)                                                                                     1.00     61.6±0.57µs        ? ?/sec    1.21     74.5±0.55µs        ? ?/sec
append_rows 4096 string_dictionary(100, 0)                                                                                    1.00    142.7±8.24µs        ? ?/sec    1.01    144.5±1.45µs        ? ?/sec
append_rows 4096 string_dictionary(100, 0.5)                                                                                  1.00    106.1±2.38µs        ? ?/sec    1.02    108.6±0.95µs        ? ?/sec
append_rows 4096 string_dictionary(30, 0)                                                                                     1.00     67.0±1.32µs        ? ?/sec    1.15     77.0±0.29µs        ? ?/sec
append_rows 4096 string_dictionary_low_cardinality(10, 0)                                                                     1.00     24.7±0.16µs        ? ?/sec    1.10     27.2±0.12µs        ? ?/sec
append_rows 4096 string_dictionary_low_cardinality(100, 0)                                                                    1.00     45.1±0.42µs        ? ?/sec    1.04     46.7±1.02µs        ? ?/sec
append_rows 4096 string_dictionary_low_cardinality(30, 0)                                                                     1.00     25.2±0.45µs        ? ?/sec    1.09     27.3±0.42µs        ? ?/sec
append_rows 4096 u64(0)                                                                                                       1.00      7.5±0.10µs        ? ?/sec    1.00      7.6±0.16µs        ? ?/sec
append_rows 4096 u64(0.3)                                                                                                     1.00     13.8±0.12µs        ? ?/sec    1.06     14.6±0.15µs        ? ?/sec
convert_columns 10 large_list(0) of u64(0)                                                                                    1.00   907.2±12.04ns        ? ?/sec    1.01   913.0±10.87ns        ? ?/sec
convert_columns 10 list(0) of u64(0)                                                                                          1.00   966.2±17.56ns        ? ?/sec    1.00    965.7±6.15ns        ? ?/sec
convert_columns 4096 4096 string_dictionary(20, 0.5), string_dictionary(30, 0), string_dictionary(100, 0), i64(0)             1.00    354.8±2.64µs        ? ?/sec    1.05    372.7±7.47µs        ? ?/sec
convert_columns 4096 bool(0, 0.5)                                                                                             1.38     12.3±0.05µs        ? ?/sec    1.00      9.0±0.19µs        ? ?/sec
convert_columns 4096 bool(0.3, 0.5)                                                                                           1.00     15.1±0.42µs        ? ?/sec    1.15     17.3±0.20µs        ? ?/sec
convert_columns 4096 i64(0)                                                                                                   1.00      7.9±0.10µs        ? ?/sec    1.00      7.9±0.24µs        ? ?/sec
convert_columns 4096 i64(0.3)                                                                                                 1.00     11.5±0.08µs        ? ?/sec    1.35     15.5±0.20µs        ? ?/sec
convert_columns 4096 large_list(0) of u64(0)                                                                                  1.00    160.7±2.10µs        ? ?/sec    1.02    163.4±0.77µs        ? ?/sec
convert_columns 4096 large_list(0) sliced to 10 of u64(0)                                                                     1.00  1165.2±13.51ns        ? ?/sec    1.02  1187.7±27.45ns        ? ?/sec
convert_columns 4096 list(0) of u64(0)                                                                                        1.00    150.8±2.98µs        ? ?/sec    1.10    166.2±1.00µs        ? ?/sec
convert_columns 4096 list(0) sliced to 10 of u64(0)                                                                           1.00  1276.1±47.49ns        ? ?/sec    1.02  1296.8±24.11ns        ? ?/sec
convert_columns 4096 string view(1..100, 0)                                                                                   1.03    118.1±2.20µs        ? ?/sec    1.00    115.1±1.25µs        ? ?/sec
convert_columns 4096 string view(1..100, 0.5)                                                                                 1.00    103.8±1.39µs        ? ?/sec    1.00    103.3±0.92µs        ? ?/sec
convert_columns 4096 string view(10, 0)                                                                                       1.00     44.8±0.23µs        ? ?/sec    1.18     52.6±0.62µs        ? ?/sec
convert_columns 4096 string view(100, 0)                                                                                      1.03     79.0±2.32µs        ? ?/sec    1.00     76.5±0.92µs        ? ?/sec
convert_columns 4096 string view(100, 0.5)                                                                                    1.00     86.2±0.77µs        ? ?/sec    1.00     86.2±1.72µs        ? ?/sec
convert_columns 4096 string view(30, 0)                                                                                       1.00     48.5±0.66µs        ? ?/sec    1.11     53.9±0.20µs        ? ?/sec
convert_columns 4096 string(10, 0)                                                                                            1.00     36.5±0.34µs        ? ?/sec    1.33     48.7±0.45µs        ? ?/sec
convert_columns 4096 string(100, 0)                                                                                           1.00     69.3±0.97µs        ? ?/sec    1.04     72.0±0.92µs        ? ?/sec
convert_columns 4096 string(100, 0.5)                                                                                         1.11     91.1±0.83µs        ? ?/sec    1.00     82.0±1.25µs        ? ?/sec
convert_columns 4096 string(20, 0.5), string(30, 0), string(100, 0), i64(0)                                                   1.00    213.1±3.04µs        ? ?/sec    1.04    221.4±2.26µs        ? ?/sec
convert_columns 4096 string(30, 0)                                                                                            1.00     39.4±0.39µs        ? ?/sec    1.26     49.6±0.22µs        ? ?/sec
convert_columns 4096 string_dictionary(10, 0)                                                                                 1.00     62.8±0.66µs        ? ?/sec    1.22     76.6±0.82µs        ? ?/sec
convert_columns 4096 string_dictionary(100, 0)                                                                                1.00    142.6±1.21µs        ? ?/sec    1.03    147.0±4.41µs        ? ?/sec
convert_columns 4096 string_dictionary(100, 0.5)                                                                              1.00    107.8±0.63µs        ? ?/sec    1.01    109.4±2.44µs        ? ?/sec
convert_columns 4096 string_dictionary(30, 0)                                                                                 1.00     68.8±0.57µs        ? ?/sec    1.14     78.4±0.80µs        ? ?/sec
convert_columns 4096 string_dictionary_low_cardinality(10, 0)                                                                 1.00     25.8±0.19µs        ? ?/sec    1.09     28.1±0.13µs        ? ?/sec
convert_columns 4096 string_dictionary_low_cardinality(100, 0)                                                                1.00     47.1±0.16µs        ? ?/sec    1.01     47.3±0.23µs        ? ?/sec
convert_columns 4096 string_dictionary_low_cardinality(30, 0)                                                                 1.00     26.1±0.69µs        ? ?/sec    1.09     28.4±0.59µs        ? ?/sec
convert_columns 4096 u64(0)                                                                                                   1.00      7.8±0.16µs        ? ?/sec    1.01      7.8±0.13µs        ? ?/sec
convert_columns 4096 u64(0.3)                                                                                                 1.00     14.1±0.21µs        ? ?/sec    1.05     14.9±0.12µs        ? ?/sec
convert_columns_prepared 10 large_list(0) of u64(0)                                                                           1.00    686.0±7.06ns        ? ?/sec    1.03   708.5±20.14ns        ? ?/sec
convert_columns_prepared 10 list(0) of u64(0)                                                                                 1.00   742.7±18.78ns        ? ?/sec    1.02    760.0±9.35ns        ? ?/sec
convert_columns_prepared 4096 4096 string_dictionary(20, 0.5), string_dictionary(30, 0), string_dictionary(100, 0), i64(0)    1.00    352.8±3.02µs        ? ?/sec    1.05    368.7±3.81µs        ? ?/sec
convert_columns_prepared 4096 bool(0, 0.5)                                                                                    1.41     12.3±0.21µs        ? ?/sec    1.00      8.7±0.07µs        ? ?/sec
convert_columns_prepared 4096 bool(0.3, 0.5)                                                                                  1.00     15.0±0.17µs        ? ?/sec    1.15     17.2±0.34µs        ? ?/sec
convert_columns_prepared 4096 i64(0)                                                                                          1.00      7.7±0.07µs        ? ?/sec    1.02      7.9±0.13µs        ? ?/sec
convert_columns_prepared 4096 i64(0.3)                                                                                        1.00     11.2±0.10µs        ? ?/sec    1.37     15.4±0.16µs        ? ?/sec
convert_columns_prepared 4096 large_list(0) of u64(0)                                                                         1.00    159.1±3.82µs        ? ?/sec    1.02    162.9±0.86µs        ? ?/sec
convert_columns_prepared 4096 large_list(0) sliced to 10 of u64(0)                                                            1.00   976.0±27.13ns        ? ?/sec    1.03  1002.9±26.26ns        ? ?/sec
convert_columns_prepared 4096 list(0) of u64(0)                                                                               1.00    150.7±3.19µs        ? ?/sec    1.10    166.2±2.37µs        ? ?/sec
convert_columns_prepared 4096 list(0) sliced to 10 of u64(0)                                                                  1.00  1079.6±23.87ns        ? ?/sec    1.01   1094.0±6.79ns        ? ?/sec
convert_columns_prepared 4096 string view(1..100, 0)                                                                          1.03    117.8±2.02µs        ? ?/sec    1.00    114.7±0.83µs        ? ?/sec
convert_columns_prepared 4096 string view(1..100, 0.5)                                                                        1.00    103.6±1.46µs        ? ?/sec    1.00    103.2±0.87µs        ? ?/sec
convert_columns_prepared 4096 string view(10, 0)                                                                              1.00     44.4±0.25µs        ? ?/sec    1.18     52.2±0.73µs        ? ?/sec
convert_columns_prepared 4096 string view(100, 0)                                                                             1.04     78.2±1.24µs        ? ?/sec    1.00     75.1±0.74µs        ? ?/sec
convert_columns_prepared 4096 string view(100, 0.5)                                                                           1.00     85.8±0.30µs        ? ?/sec    1.00     86.0±0.48µs        ? ?/sec
convert_columns_prepared 4096 string view(30, 0)                                                                              1.00     48.1±0.96µs        ? ?/sec    1.13     54.5±2.11µs        ? ?/sec
convert_columns_prepared 4096 string(10, 0)                                                                                   1.00     36.2±0.46µs        ? ?/sec    1.34     48.6±0.33µs        ? ?/sec
convert_columns_prepared 4096 string(100, 0)                                                                                  1.00     68.7±0.84µs        ? ?/sec    1.04     71.8±0.83µs        ? ?/sec
convert_columns_prepared 4096 string(100, 0.5)                                                                                1.11     91.0±0.89µs        ? ?/sec    1.00     81.9±0.35µs        ? ?/sec
convert_columns_prepared 4096 string(20, 0.5), string(30, 0), string(100, 0), i64(0)                                          1.00    210.5±1.60µs        ? ?/sec    1.05    220.9±3.37µs        ? ?/sec
convert_columns_prepared 4096 string(30, 0)                                                                                   1.00     39.3±0.45µs        ? ?/sec    1.26     49.5±0.35µs        ? ?/sec
convert_columns_prepared 4096 string_dictionary(10, 0)                                                                        1.00     61.6±0.65µs        ? ?/sec    1.22     75.0±0.98µs        ? ?/sec
convert_columns_prepared 4096 string_dictionary(100, 0)                                                                       1.00    141.7±1.03µs        ? ?/sec    1.03    145.6±1.73µs        ? ?/sec
convert_columns_prepared 4096 string_dictionary(100, 0.5)                                                                     1.00    106.6±0.39µs        ? ?/sec    1.02    108.7±0.88µs        ? ?/sec
convert_columns_prepared 4096 string_dictionary(30, 0)                                                                        1.00     66.8±0.25µs        ? ?/sec    1.15     76.9±0.31µs        ? ?/sec
convert_columns_prepared 4096 string_dictionary_low_cardinality(10, 0)                                                        1.00     25.1±0.30µs        ? ?/sec    1.09     27.4±0.20µs        ? ?/sec
convert_columns_prepared 4096 string_dictionary_low_cardinality(100, 0)                                                       1.00     45.4±0.21µs        ? ?/sec    1.03     46.7±1.80µs        ? ?/sec
convert_columns_prepared 4096 string_dictionary_low_cardinality(30, 0)                                                        1.00     25.3±0.27µs        ? ?/sec    1.08     27.4±0.13µs        ? ?/sec
convert_columns_prepared 4096 u64(0)                                                                                          1.00      7.7±0.12µs        ? ?/sec    1.00      7.7±0.17µs        ? ?/sec
convert_columns_prepared 4096 u64(0.3)                                                                                        1.00     14.0±0.20µs        ? ?/sec    1.06     14.8±0.17µs        ? ?/sec
convert_rows 10 large_list(0) of u64(0)                                                                                       1.06  1621.2±17.01ns        ? ?/sec    1.00   1527.5±8.85ns        ? ?/sec
convert_rows 10 list(0) of u64(0)                                                                                             1.11  1811.2±12.24ns        ? ?/sec    1.00  1630.7±36.57ns        ? ?/sec
convert_rows 4096 4096 string_dictionary(20, 0.5), string_dictionary(30, 0), string_dictionary(100, 0), i64(0)                1.03    314.6±8.67µs        ? ?/sec    1.00   304.7±12.79µs        ? ?/sec
convert_rows 4096 bool(0, 0.5)                                                                                                1.19     19.7±0.14µs        ? ?/sec    1.00     16.5±0.23µs        ? ?/sec
convert_rows 4096 bool(0.3, 0.5)                                                                                              1.19     19.7±0.20µs        ? ?/sec    1.00     16.6±0.56µs        ? ?/sec
convert_rows 4096 i64(0)                                                                                                      1.00     33.2±0.14µs        ? ?/sec    1.23    40.7±13.84µs        ? ?/sec
convert_rows 4096 i64(0.3)                                                                                                    1.00     33.3±1.19µs        ? ?/sec    1.22    40.8±14.17µs        ? ?/sec
convert_rows 4096 large_list(0) of u64(0)                                                                                     1.01    273.4±2.37µs        ? ?/sec    1.00    269.6±4.87µs        ? ?/sec
convert_rows 4096 large_list(0) sliced to 10 of u64(0)                                                                        1.00      2.1±0.02µs        ? ?/sec    1.00      2.1±0.02µs        ? ?/sec
convert_rows 4096 list(0) of u64(0)                                                                                           1.02    272.8±3.74µs        ? ?/sec    1.00    268.3±3.44µs        ? ?/sec
convert_rows 4096 list(0) sliced to 10 of u64(0)                                                                              1.04      2.3±0.03µs        ? ?/sec    1.00      2.2±0.01µs        ? ?/sec
convert_rows 4096 string view(1..100, 0)                                                                                      1.00    168.9±0.73µs        ? ?/sec    1.04    175.3±1.26µs        ? ?/sec
convert_rows 4096 string view(1..100, 0.5)                                                                                    1.00    137.4±0.87µs        ? ?/sec    1.02    140.3±1.63µs        ? ?/sec
convert_rows 4096 string view(10, 0)                                                                                          1.00     76.0±1.68µs        ? ?/sec    1.12     85.4±7.08µs        ? ?/sec
convert_rows 4096 string view(100, 0)                                                                                         1.00    123.7±1.68µs        ? ?/sec    1.05    129.6±2.47µs        ? ?/sec
convert_rows 4096 string view(100, 0.5)                                                                                       1.00    113.5±0.31µs        ? ?/sec    1.05    119.2±4.25µs        ? ?/sec
convert_rows 4096 string view(30, 0)                                                                                          1.00     85.1±1.60µs        ? ?/sec    1.11     94.4±2.46µs        ? ?/sec
convert_rows 4096 string(10, 0)                                                                                               1.07     64.8±0.19µs        ? ?/sec    1.00     60.4±1.39µs        ? ?/sec
convert_rows 4096 string(100, 0)                                                                                              1.04    114.6±1.40µs        ? ?/sec    1.00    110.2±0.71µs        ? ?/sec
convert_rows 4096 string(100, 0.5)                                                                                            1.05    108.4±1.08µs        ? ?/sec    1.00    103.4±0.51µs        ? ?/sec
convert_rows 4096 string(20, 0.5), string(30, 0), string(100, 0), i64(0)                                                      1.04   315.1±12.59µs        ? ?/sec    1.00   304.2±13.21µs        ? ?/sec
convert_rows 4096 string(30, 0)                                                                                               1.08     78.3±3.33µs        ? ?/sec    1.00     72.6±0.96µs        ? ?/sec
convert_rows 4096 string_dictionary(10, 0)                                                                                    1.09     66.0±7.15µs        ? ?/sec    1.00     60.4±0.29µs        ? ?/sec
convert_rows 4096 string_dictionary(100, 0)                                                                                   1.04    115.1±1.09µs        ? ?/sec    1.00    110.4±1.53µs        ? ?/sec
convert_rows 4096 string_dictionary(100, 0.5)                                                                                 1.05    109.4±6.25µs        ? ?/sec    1.00    104.1±3.10µs        ? ?/sec
convert_rows 4096 string_dictionary(30, 0)                                                                                    1.08     78.3±0.40µs        ? ?/sec    1.00     72.6±0.46µs        ? ?/sec
convert_rows 4096 string_dictionary_low_cardinality(10, 0)                                                                    1.07     64.8±0.55µs        ? ?/sec    1.00     60.5±0.30µs        ? ?/sec
convert_rows 4096 string_dictionary_low_cardinality(100, 0)                                                                   1.03    114.6±1.56µs        ? ?/sec    1.00    110.8±3.98µs        ? ?/sec
convert_rows 4096 string_dictionary_low_cardinality(30, 0)                                                                    1.08     78.1±0.27µs        ? ?/sec    1.00     72.6±0.27µs        ? ?/sec
convert_rows 4096 u64(0)                                                                                                      1.01     32.3±0.75µs        ? ?/sec    1.00     32.1±0.43µs        ? ?/sec
convert_rows 4096 u64(0.3)                                                                                                    1.01     32.3±0.21µs        ? ?/sec    1.00     31.9±0.14µs        ? ?/sec
iterate rows                                                                                                                  1.00      2.6±0.10µs        ? ?/sec    1.00      2.6±0.05µs        ? ?/sec