Thread Safety for parquet::arrow::FileReader::GetRecordBatch

[pjones]

I posted the following on the Zulip chat and Antoine asked me to repost it here:

Is it safe for multiple threads to use the same parquet::arrow::FileReader object and each call GetRecordBatchReader with different row group IDs separately?

Also, how can I figure this out for myself in the future so I don't have to bug others?

Thank you!

[pitrou]

@wgtmac @adamreeve @HuaHuaY @mapleFU If you have a more precise idea than I have :)

[wgtmac]

I don't think we can do this blindly. Codex has spotted evidence like below:

---

With the current default settings, concurrent calls to GetRecordBatchReader() on the same FileReader are not safe. ArrowReaderProperties::pre_buffer defaults to true, and each call invokes ParquetFileReader::PreBuffer(). That method replaces SerializedFile::cached_source_ and clears/rebuilds prebuffered_column_chunks_, while GetRowGroup() reads both without synchronization. Different row-group IDs do not avoid the race because this state is file-wide.

If pre-buffering is disabled before constructing the FileReader, then separate RecordBatchReader instances for different row groups appear safe to consume concurrently in the current implementation, provided that:

• the FileReader outlives all returned readers;
• its properties are not modified concurrently;
• each RecordBatchReader is consumed by only one thread; and
• the underlying RandomAccessFile follows the thread-safe ReadAt() contract.

[HuaHuaY]

It sounds to me like this has nothing to do with whether the row group IDs of multiple RecordBatchReaders overlap. I suspect it depends on the actual capabilities supported by the InputStream. If we were to make a thread-safety guarantee regarding the interface, it would likely be: "Whether the result of GetRecordBatchReader is thread-safe depends on the safety of certain underlying parameters."

[pjones]

Thank you to everyone who took the time to respond to this discussion.

For now it seems like the best option is to have each thread open its own copy of the Parquet file and fetch separate row groups.