Bug: rust panic when reading simple table

[eeroel]

Hi,

I'm getting a crash with duckdb 1.2.0 and duckdb-delta efc4522 when reading a simple test table written with Polars. This used to work with an older version. I'm running on Mac with Apple Silicon.

Script to reproduce, requires polars. Note that the crash seems to depend on the number of files generated, with a smaller number it works.

import duckdb
import time

import polars as pl

## This works with smaller values, like range(50)
for file in range(100):
    df = pl.DataFrame(
        [
            pl.Series("foo", range(10), dtype=pl.Int64),
            pl.Series("bar", range(0+file, 10+file), dtype=pl.Int64),
        ]
    )
    df.write_delta("test_table", mode="append")


duckdb.execute("force install delta from core_nightly; UPDATE EXTENSIONS; load delta;").fetchall()
duckdb.execute("ATTACH './test_table' AS my_delta_table (TYPE delta)")
t=time.time()
res = duckdb.query("""SELECT COUNT(*) from my_delta_table""")
print(str(res))
print(time.time()-t)

Full backtrace:

thread '<unnamed>' panicked at core/src/panicking.rs:223:5:
panic in a function that cannot unwind
stack backtrace:
   0:        0x1627260a4 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hadba1856081fe8dc
   1:        0x1627460e8 - core::fmt::write::h5358bd20891469bc
   2:        0x162722ce0 - std::io::Write::write_fmt::hbf0611cc5d72cc91
   3:        0x162725f58 - std::sys::backtrace::BacktraceLock::print::he2302a8c253c9a13
   4:        0x162726ea4 - std::panicking::default_hook::{{closure}}::hec1f77a77d7e7ffc
   5:        0x162726cd8 - std::panicking::default_hook::hdd59ab537dd27efb
   6:        0x162727694 - std::panicking::rust_panic_with_hook::h533a16f5f89e4278
   7:        0x1627272e4 - std::panicking::begin_panic_handler::{{closure}}::h168c3a4362c8e4df
   8:        0x162726568 - std::sys::backtrace::__rust_end_short_backtrace::h601e3529ca2053df
   9:        0x162726fc4 - _rust_begin_unwind
  10:        0x16277fca4 - core::panicking::panic_nounwind_fmt::h6251928767c378a3
  11:        0x16277fd1c - core::panicking::panic_nounwind::he5e0e92598e09d63
  12:        0x16277fe08 - core::panicking::panic_cannot_unwind::hc39e9acf1ae7ca4b
  13:        0x1619d2214 - _visit_scan_data
  14:        0x1619d1a94 - _kernel_scan_data_next
  15:        0x16091adf4 - __ZN6duckdb13DeltaSnapshot15GetFileInternalEy
  16:        0x160920964 - __ZN6duckdb13DeltaSnapshot17GetTotalFileCountEv
  17:        0x1609209dc - __ZN6duckdb13DeltaSnapshot14GetCardinalityERNS_13ClientContextE
  18:        0x108e8d200 - __ZN6duckdb19ParquetScanFunction18ParquetCardinalityERNS_13ClientContextEPKNS_12FunctionDataE
  19:        0x10a53b5b4 - __ZN6duckdb10LogicalGet19EstimateCardinalityERNS_13ClientContextE
  20:        0x10a381668 - __ZN6duckdb24RelationStatisticsHelper15ExtractGetStatsERNS_10LogicalGetERNS_13ClientContextE
  21:        0x10a37f4ec - __ZN6duckdb15RelationManager20ExtractJoinRelationsERNS_18JoinOrderOptimizerERNS_15LogicalOperatorERNS_6vectorINSt3__117reference_wrapperIS3_EELb1EEENS_12optional_ptrIS3_Lb1EEE
  22:        0x10a374fa8 - __ZN6duckdb17QueryGraphManager5BuildERNS_18JoinOrderOptimizerERNS_15LogicalOperatorE
  23:        0x10a3749c0 - __ZN6duckdb18JoinOrderOptimizer8OptimizeENS_10unique_ptrINS_15LogicalOperatorENSt3__114default_deleteIS2_EELb1EEENS_12optional_ptrINS_13RelationStatsELb1EEE
  24:        0x10a37effc - __ZN6duckdb15RelationManager20ExtractJoinRelationsERNS_18JoinOrderOptimizerERNS_15LogicalOperatorERNS_6vectorINSt3__117reference_wrapperIS3_EELb1EEENS_12optional_ptrIS3_Lb1EEE
  25:        0x10a374fa8 - __ZN6duckdb17QueryGraphManager5BuildERNS_18JoinOrderOptimizerERNS_15LogicalOperatorE
  26:        0x10a3749c0 - __ZN6duckdb18JoinOrderOptimizer8OptimizeENS_10unique_ptrINS_15LogicalOperatorENSt3__114default_deleteIS2_EELb1EEENS_12optional_ptrINS_13RelationStatsELb1EEE
  27:        0x10a37ed98 - __ZN6duckdb15RelationManager20ExtractJoinRelationsERNS_18JoinOrderOptimizerERNS_15LogicalOperatorERNS_6vectorINSt3__117reference_wrapperIS3_EELb1EEENS_12optional_ptrIS3_Lb1EEE
  28:        0x10a374fa8 - __ZN6duckdb17QueryGraphManager5BuildERNS_18JoinOrderOptimizerERNS_15LogicalOperatorE
  29:        0x10a3749c0 - __ZN6duckdb18JoinOrderOptimizer8OptimizeENS_10unique_ptrINS_15LogicalOperatorENSt3__114default_deleteIS2_EELb1EEENS_12optional_ptrINS_13RelationStatsELb1EEE
  30:        0x10a369ebc - __ZNSt3__110__function6__funcIZN6duckdb9Optimizer20RunBuiltInOptimizersEvE4$_25NS_9allocatorIS4_EEFvvEEclEv
  31:        0x10a3524b0 - __ZN6duckdb9Optimizer12RunOptimizerENS_13OptimizerTypeERKNSt3__18functionIFvvEEE
  32:        0x10a352864 - __ZN6duckdb9Optimizer20RunBuiltInOptimizersEv
  33:        0x10a3530e0 - __ZN6duckdb9Optimizer8OptimizeENS_10unique_ptrINS_15LogicalOperatorENSt3__114default_deleteIS2_EELb1EEE
  34:        0x10a2120ec - __ZN6duckdb13ClientContext31CreatePreparedStatementInternalERNS_17ClientContextLockERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEENS_10unique_ptrINS_12SQLStatementENS3_14default_deleteISD_EELb1EEENS_12optional_ptrINS3_13unordered_mapIS9_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityENS7_INS3_4pairISA_SJ_EEEEEELb1EEE
  35:        0x10a212d5c - __ZN6duckdb13ClientContext23CreatePreparedStatementERNS_17ClientContextLockERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEENS_10unique_ptrINS_12SQLStatementENS3_14default_deleteISD_EELb1EEENS_12optional_ptrINS3_13unordered_mapIS9_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityENS7_INS3_4pairISA_SJ_EEEEEELb1EEENS_21PreparedStatementModeE
  36:        0x10a217688 - __ZN6duckdb13ClientContext24PendingStatementInternalERNS_17ClientContextLockERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEENS_10unique_ptrINS_12SQLStatementENS3_14default_deleteISD_EELb1EEERKNS_22PendingQueryParametersE
  37:        0x10a218ffc - __ZN6duckdb13ClientContext35PendingStatementOrPreparedStatementERNS_17ClientContextLockERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEENS_10unique_ptrINS_12SQLStatementENS3_14default_deleteISD_EELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE
  38:        0x10a216f10 - __ZN6duckdb13ClientContext43PendingStatementOrPreparedStatementInternalERNS_17ClientContextLockERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEENS_10unique_ptrINS_12SQLStatementENS3_14default_deleteISD_EELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE
  39:        0x10a217f60 - __ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockENS_10unique_ptrINS_12SQLStatementENSt3__114default_deleteIS4_EELb1EEERKNS_22PendingQueryParametersEb
  40:        0x10a21bf30 - __ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockERKNS_10shared_ptrINS_8RelationELb1EEEb
  41:        0x10a21c454 - __ZN6duckdb13ClientContext12PendingQueryERKNS_10shared_ptrINS_8RelationELb1EEEb
  42:        0x10a7b7db8 - __ZN6duckdbL17PyExecuteRelationERKNS_10shared_ptrINS_8RelationELb1EEEb
  43:        0x10a7c2090 - __ZN6duckdb16DuckDBPyRelation16ToStringInternalERKNS_17BoxRendererConfigEb
  44:        0x10a7c2364 - __ZN6duckdb16DuckDBPyRelation8ToStringEv
  45:        0x10a7e9014 - __ZZN8pybind1112cpp_function10initializeIZNS0_C1INSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEN6duckdb16DuckDBPyRelationEJEJNS_4nameENS_9is_methodENS_7siblingEEEEMT0_FT_DpT1_EDpRKT2_EUlPSB_E_S9_JSP_EJSC_SD_SE_EEEvOSG_PFSF_SI_ESO_ENKUlRNS_6detail13function_callEE_clESW_
  46:        0x10a7e8f8c - __ZZN8pybind1112cpp_function10initializeIZNS0_C1INSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEN6duckdb16DuckDBPyRelationEJEJNS_4nameENS_9is_methodENS_7siblingEEEEMT0_FT_DpT1_EDpRKT2_EUlPSB_E_S9_JSP_EJSC_SD_SE_EEEvOSG_PFSF_SI_ESO_ENUlRNS_6detail13function_callEE_8__invokeESW_
  47:        0x10a6d2670 - __ZN8pybind1112cpp_function10dispatcherEP7_objectS2_S2_
  48:        0x1054f5a20 - _cfunction_call
  49:        0x105491738 - __PyObject_MakeTpCall
  50:        0x105495bd0 - _method_vectorcall
  51:        0x105523110 - _vectorcall_method
  52:        0x10551d194 - _slot_tp_str
  53:        0x1054fafb4 - _PyObject_Str
  54:        0x1055b5e18 - __PyEval_EvalFrameDefault
  55:        0x1055a6814 - _PyEval_EvalCode
  56:        0x10561fbcc - _pyrun_file
  57:        0x10561f39c - __PyRun_SimpleFileObject
  58:        0x10561e91c - __PyRun_AnyFileObject
  59:        0x105647e64 - _pymain_run_file_obj
  60:        0x105647648 - _pymain_run_file
  61:        0x105646e5c - _Py_RunMain
  62:        0x105647fe0 - _pymain_main
  63:        0x105648818 - _Py_BytesMain

[kapteinkaald]

We have the same issue. We are dumping data to a google bucket using deltalake. This only appears however after many many data dumps. Before that reading from the bucket works fine.

[kapteinkaald]

After some further investigation I suspect this comes from the delta-kernel-rs package since this is all C++. Or am I mistaken? Then it also could be a problem with the delta-kernel-rs? I also get this while using duckdb version 1.1.3.

[kapteinkaald]

If I delete the checkpoints that are created then I can read the data.

[samansmink]

Thanks for reporting, this appears to be related this issue: delta-io/delta-kernel-rs#669

[kapteinkaald]

Regarding this issue. @samansmink I saw that you pushed a fix for this. It seems to work now for my mac (arm64). I deleted the old extension file in ~/.duckdb and installed the extension again. However it does not seem to work when building docker images. In the github action you build the extension files. But when this is installed in a linux environment, is the latest version pulled from extentions.duckdb.org as it seemingly was for my mac?

[samansmink]

The fix should be in the nightly extension repository (install delta from core_nightly) are you sure you are pulling from there?

[dmunch]

Been hitting the same issue, and more than happy to see that it's already addressed! (the extension from nightly did work for me, thanks a lot!)

What I've been a but puzzled about is how writing that table with Polars, which certainly uses delta-rs under the hood, actually generates a deletion vector, when apparently delta-rs doesn't support them? It might be my lack of understanding regarding deletion vectors, but something feels inconsistent. Maybe it's related to delta-io/delta-rs#3211 ?

[samansmink]

I think the problem is that it writes a metadata field in the checkpoint file incorrectly, hence the error message Deletion Vector error: Unknown storage format: ''

[scovich]

I don't think this is due to JSON null that delta-io/delta-rs#3211 trips over. I strongly suspect it's actually caused by a behavior change from the arrow-53.3 upgrade, which broke kernel's NULL handling: delta-io/delta-kernel-rs#691. A fix is up but not yet merged.