Lance Format Cache Objects

[maxweriz]

Is your feature request related to a problem? Please describe.

The cache formats right now don’t schema enforce their content and pickle and pq are relatively expensive to write and read. The most efficient is likely json but that also has its downsides and can be extremely slow for point lookups. This is extremely important for serialization and deserialization in vector settings which is going to be, imo, most use cases in ML now-a-days (or should be). None of these are optimized for zero copy or non contiguous vector use cases.

Describe the solution you'd like

I’d be interested if there is a way to on the fly schematize data into Arrow and write it to Lance format for faster lookups. The cost here is additional storage space (compared to pq and json) for faster deserialization, point lookups, and mem mapping. I’m particularly interested in zero copy i/o of objects like numpy arrays.

Describe alternatives you've considered

I’ve also considered that https://fory.apache.org/blog/fury_blazing_fast_multiple_language_serialization_framework/ could be a good option. That multi language support would be great to have given defining the following workflow

1. Define DAG in python with Hamilton
2. Have DAG steps call PyO3 rust bindings to optimized feature generation
3. Have DAG cache the outputs via Lance or Fury
4. Potentially read from that cache in Rust, cpp, etc…. for features needed during inference/deployment

Additional context
Add any other context or screenshots about the feature request here.

[maxweriz]

I’d be happy to implement this as a feature add if it sounds reasonable to support one of these vector optimized formats. My main use case right now is passing cache objects between python and Rust on different nodes but using the Hamilton DAG as the orchestration and api layer.

To be clear, I’m not suggesting using it as a feature store but rather an intermediary format when you create something that is vectorized but only need a subset of that vectorized output in different child tasks. A good example is temporal aggregation where we can form features on top of a dataset to create a new feature vector

{
   “timestamps”: [...],
   “data”: [...],
   “features”: [...]
}

Then in our child tasks we want to only operate on subsets of that data ie: even rows or odd rows in different functions, to create windows.

Right now that entire structure would have to be serialized and deserialized from disc for each of these child tasks but I don’t think that’s needed using a more optimized file format.