don't write deletion vector entry in the log

[djouallah]

getting errors from snowflake when I try to read a delta table there, it seems delta_rs add an entry "deletionVector": null

which get misinterpreted as there are some deletion vectors in the file, I notice spark don't write that entry in the log hence it works just fine.

[ion-elgreco]

Seems like easy for fix snowflake, if it's null, then obviously they can ignore it

[scovich]

JSON null is a weird beast... some systems treat it as non-NULL (because something is there), and the spark/parquet variant spec mandates that it is NOT SQL NULL.

That said, regardless of whether it's a "proper" SQL NULL or not, it's clearly not a DV and clients should ignore it.

[Zan-L]

I asked the same question before (#3055) but Delta protocol does allow null deletion vector field. I guess delta-rs is just not going to be compatible with Snowflake unless someone asks Snowflake to update their code.

[ion-elgreco]

I asked the same question before (#3055) but Delta protocol does allow null deletion vector field. I guess delta-rs is just not going to be compatible with Snowflake unless someone asks Snowflake to update their code.

You mean "snowflake is not going to be compatible with delta-rs and the delta protocol".

[Zan-L]

I asked the same question before (#3055) but Delta protocol does allow null deletion vector field. I guess delta-rs is just not going to be compatible with Snowflake unless someone asks Snowflake to update their code.

You mean "snowflake is not going to be compatible with delta-rs and the delta protocol".

I see what you mean. Yes, the fault is not at delta-rs. However, the fact that Snowflake claims to support Delta implies that the native Delta implementation (Databricks?) must have chosen one of the two routes specified in the protocol, which is to not have deletion vector field when it's not enabled, and Snowflake must have blindly followed Databricks. Unfortunately, both Snowflake and Databricks are the more dominant players.

[ion-elgreco]

Nothing is keeping Snowflake from fixing this issue though 🤷 . They are a billion dollar company with enough money and devs to address this on their end.