Compute data buffer length by using start and end values in offset buffer

[viirya]

Which issue does this PR close?

Closes #5756.

Rationale for this change

Encountered an issue when importing empty variable-size binary layout array (e.g., string) from Java Arrow.

There is difference between Java Arrow and arrow-rs when computing the length of data buffer: apache/arrow#41610 (comment)

This is how Java Arrow imports an Utf8 array:

try (ArrowBuf offsets = importOffsets(type, VarCharVector.OFFSET_WIDTH)) {
      final int start = offsets.getInt(0);
      final int end = offsets.getInt(fieldNode.getLength() * (long) VarCharVector.OFFSET_WIDTH);
      final int len = end - start;
      ...
}

So even the offset buffer is not initialized, for empty array with one element offset buffer, end - start is always 0 that is the length of data buffer. That is why the added roundtrip tests are passed.

But in arrow-rs, it takes the last value of the offset buffer as the length of data buffer, i.e., end. If the value is not initialized to zero, the computed length of data buffer is incorrect.

That is what I found for the first offset value from the spec:

Generally the first slot in the offsets array is 0, and the last slot is the length of the values array.
When serializing this layout, we recommend normalizing the offsets to start at 0.

It looks like the first value doesn't have to be 0, although generally it is. So seems Java Arrow's approach is (more) correct.

What changes are included in this PR?

Changing the way to compute the length of data buffer when importing an empty variable-size binary layout array.

Are there any user-facing changes?

No

[jhorstmann]

There are comments a few lines above the changes about the len of the data buffer, which should probably be updated.

Other than that the change looks good to me, for slices of StringArray/BinaryArray the first offset would be != 0.

[viirya]

Thank you @jhorstmann. I've updated the code comment.

[viirya]

Hmm, our CI pipeline (Archery test With other arrows) seems broken:

  = note: /opt/conda/envs/arrow/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d2ef02247056996e.rlib(std-d2ef02247056996e.std.e4dfbc2c3f4b09f1-cgu.0.rcgu.o): in function `std::sys::pal::unix::stack_overflow::imp::sigstack_size':
          /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/stack_overflow.rs:216: undefined reference to `getauxval'
          /opt/conda/envs/arrow/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/stack_overflow.rs:216: undefined reference to `getauxval'
          collect2: error: ld returned 1 exit status

Created #5742

[alamb]

The Ci failure is covered by #5719

[alamb]

Is it possible to write a test for this fix? Otherwise I worry we may end up breaking it in future releases without knowing

[viirya]

Is it possible to write a test for this fix? Otherwise I worry we may end up breaking it in future releases without knowing

I added a unit test that simulates an empty string array with a non-zero offset from a producer and checks the buffer_len call returns correct data buffer length.

[alamb]

Makes sense to me -- thank you @viirya and @jhorstmann

[viirya]

Merged. Thanks @jhorstmann @alamb

[tustvold]

This PR breaks handling of non-empty arrays, in particular consider the case of a sliced offset buffer consisting of 3, 5. The length of the data buffer is 5 not 2.

[alamb]

Proposed fix in #5964

[viirya]

Yea, this copied how to calculate the length of data buffer from Arrow Java and it is incorrect.