Improve IPCTests

[willtemperley]

Describe the enhancement requested

In #95 a fix for the Block metadata length was added, however this is difficult to test because there is no API to access Blocks in the footer.

A temporary fix might be to just read the footer in IPC tests and ensure that record batches can be accessed with the block metadata. Alternatively, perhaps a random access API could be added to ArrowReader?

There is also no test to ensure other implementations can read the files created with ArrowWriter. @kou mentioned Archery should be used for this.

[kou]

The current our reader implementation accepts both of data that have padding and data that don't have padding, right?

Should our reader implementation rejects data that don't have padding as the first step?

We should use Archery based integration test eventually.

[willtemperley]

Should our reader implementation rejects data that don't have padding as the first step?

Yes agreed that should happen. However without testing against other implementations, the potential for matching bugs in reader and writer implementations remains.

[willtemperley]

Maybe we could add arrow-testing as a git submodule?

[kou]

We can do it. Do you want to use files generated by OSS-Fuzz, right?

[willtemperley]

I'd like to use the gold files in arrow-ipc-stream/integration/cpp-21.0.0, e.g.

generated_binary.arrow_file
generated_binary.json.gz
generated_binary.stream

My initial thought was to set up a test target to read the JSON files, however I now realise these are part of Archery.

Maybe it's worth thinking about setting up executable targets to support Archery? These could probably be referenced in a Swift test target.

[kou]

Maybe it's worth thinking about setting up executable targets to support Archery? These could probably be referenced in a Swift test target.

If we want to use gold files, yes.