Fix: Use arrow's casting logic to cast file stats when reporting them to DF - #9818
Conversation
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Merging this PR will improve performance by 18.73%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | decompress[u64, (4000, 1024)] |
86.1 µs | 70.8 µs | +21.62% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.4 µs | 17.6 µs | +21.29% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing adamg/cast-dt-ts-stats (f37ca4b) with develop (2a5b1a7)2
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(9d1b103) during the generation of this report, so 2a5b1a7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
robert3005
left a comment
There was a problem hiding this comment.
I would really like to plumb the custom behaviour through vortex execution
|
@joseph-isaacs and I talked it earlier this week, will require some bigger changes to make cast pushdown for extensions more extendible |
Polar Signals Profiling ResultsLatest Run
Previous Runs (3)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.974x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.919x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.116x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.040x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.028x ➖, 1↑ 2↓)
datafusion / parquet / ns (0.967x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed / ns (1.068x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.048x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.979x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 3↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.027x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.985x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 1↑ 5↓)
duckdb / parquet / ns (1.006x ➖, 1↑ 5↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.015x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.984x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 41↑ 59↓)
Totals:
|
Summary
Delegate stat casting to Arrow's casting that cover more cases and adhere to the semantics that DF expects.
This also fixes a bug where our TS casting only changes the metadata but doesn't touch the data itself (in stats, fixing the casting behavior for arrays is a different issue).