Compact ViewArrays on IPCWrite

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The IPC writer typically tries to avoid writing unreferenced data, e.g. re-encoding offsets for StringArray, ListArray, etc... see write_array_data.

Currently the IPC writer writes view data types as is, potentially including significant unreferenced data.

Describe the solution you'd like

I think by default the IPC writer should at least invoke the GC API, and potentially deduplicate them. We could potentially make this configurable.

Describe alternatives you've considered

Additional context

Potentially some overlap with

• Minify Kernel #7186
• Support interleave kernel with deep copy buffers for StringViewArray #7184

FYI @alamb @XiangpengHao

[alamb]

I agree this would be a good feature

[davidhewitt]

I think we've encountered performance trouble for this at Pydantic at service boundaries where we pass IPC streams and will either add a mitigation in our own code or contribute here.

Also, I was wondering if this problem will be relevant for the parquet writer too?

[tustvold]

Also, I was wondering if this problem will be relevant for the parquet writer too?

The parquet writer won't run into this precise issue, although I'm not saying there couldn't be some other ViewArray related issues.

[davidhewitt]

Good to hear, of course I'd expect there may be other things to improve!