Error loading parquet file

[kletts]

I have a parquet file with data generated in R for which I'm receiving a load error. I have one version of the file generated by the R function arrow::write_parquet which works and a second generated by a newer library called nanoparquet::write_parquet which fails with the following error when loading in Observable Framework.

Loading as:

const df = await  FileAttachment("data/test-nanoparquet.parquet").parquet();

I get the following error:

Error: Arrow: Unable to get root as message stored in ARROW:schema: Unaligned { position: 60, unaligned_type: "arrow_ipc::gen::Schema::Feature", error_trace: ErrorTrace([TableField { field_name: "features", position: 44 }, UnionVariant { variant: "MessageHeader::Schema", position: 20 }, TableField { field_name: "header", position: 20 }]) }

The same file loaded using the DuckDB client does not generate an error:

const db = DuckDBClient.of({test:  FileAttachment("data/test-nanoparquet.parquet") });  

Both files, the arrow and nanoparquet generated files are loaded here, where both version are working: https://observablehq.com/d/fe3170f7c940438b

I have tried updating Observable Framework version (I'm on v1.9.0) and run npm update, is there something else I should update? Although I'm happy with the work arounds, the simplicity of loading the file directly is very convenient.

Any thoughts?
Christian

[Fil]

You can get the same error in your notebook by using the most recent version of parquet-wasm:

parquetWasm = import("https://cdn.jsdelivr.net/npm/[email protected]/esm/parquet_wasm.js")

This is probably due to parquet-wasm dropping the arrow2 and parquet2 formats in https://github.com/kylebarron/parquet-wasm/releases/tag/v0.6.0 (EDIT: no)

pinging @kylebarron because this goes a bit above my head :)

[kylebarron]

It seems likely this is a non compliant Parquet file, especially given the fact that nanoparquet is a recent library. I don't think this is related to dropping support for arrow2/parquet2 on the rust side, because arrow/parquet is the canonical parquet implementation in rust and should be correct.

[Fil]

In fact you get the same error with parquetWasm = import("https://cdn.jsdelivr.net/npm/[email protected]/web.js"), so the difference happens between 0.1.1 and 0.2.0 — my assumption above was wrong.

[kylebarron]

I don't think it's anything I'm doing then; more likely it's from a version bump of the underlying rust library. I know that union fields in arrow IPC changed layout slightly at some point, so maybe nanoparquet is using the old layout?

[kletts]

Thanks for your help folks, I'll post an issue with nanoparquet as I can see some differences in the file metadata