Skip to content

FFI: no way to construct extension-typed scalar literals for scan predicates #8702

Description

@balicat

Context

I am building a .NET binding over vortex-ffi (https://github.com/balicat/vortex-dotnet). Predicate pushdown through vx_scan_options.filter works well for string columns — vx_expression_list_contains over a utf8 list scalar successfully prunes chunks.

The remaining gap is predicates over extension-typed date columns. My files have a period column of type vortex.date[days](i32), and I could not find a way to express period >= <date> through the FFI.

What I tried

Building the literal from the storage value:

vx_expression* root = vx_expression_root();
vx_expression* period_col = vx_expression_get_item(vx_view_from_cstr("period"), root);
vx_scalar* s = vx_scalar_new_i32(16983, false);            // days since Unix epoch
vx_expression* lit = vx_expression_literal(s, &err);
vx_expression* cmp = vx_expression_binary(VX_OPERATOR_GTE, period_col, lit);

The scan rejects it at evaluation time:

Array dtype vortex.date[days](i32?) does not match lower dtype i32

That behavior makes sense — the types really do differ. The problem is the FFI offers no way to construct a scalar of the extension type:

  • vx_scalar_new_* covers primitives, utf8, binary, decimal, list, struct and typed null, but there is no date/timestamp constructor and no generic "extension scalar from storage scalar plus dtype" constructor. The FFI exposes extension dtypes (via the schema accessors and vx_dtype_is_date), but there is no corresponding way to construct a scalar of such a dtype.
  • There is no cast expression in the FFI. The Python API already exposes both vortex.date(...) literals and expr.cast, so this appears to be an FFI surface gap rather than an engine limitation.

Checked against commit b9779d6, and re-checked against HEAD (aff0bd3, 2026-07-09) — scalar.rs still has no date, timestamp, or extension constructor.

Workaround

I push down the series predicate (which does the real pruning for sorted time-series files) and apply date bounds after decode. That preserves correctness, but it decodes rows the engine could otherwise prune, and any consumer of ext-typed columns through the FFI will hit the same wall.

Possible API additions

  1. vx_scalar_new_date(int32_t days, uint8_t unit, bool nullable) and a timestamp sibling, mirroring the Python literals (unit encoded as in vx_dtype_time_unit).
  2. More generally: vx_scalar_new_extension(const vx_dtype* ext_dtype, const vx_scalar* storage, vx_error**), which would cover future extension types without new constructors. The extension dtype itself needs no new constructor — callers can already obtain it from the file's schema via vx_data_source_dtype and the struct field accessors.

Happy to test a patch against my binding and its corpus — it includes date32 files from 252 rows up to 10.3M rows.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions