[Feature] Discussion: Can we use Bitmap File Index to update DV when delete where by non-primary column?

[fafacao86]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

https://cwiki.apache.org/confluence/display/PAIMON/PIP-23+Introduce+bitmap+file+index
BitmapIndex tracks row positions of a specific value.
Let's say we have a primarykey table, a non-primary-key column: type. And we enable bitmap file index on the type column. For Spark, when DELETE FROM table WHERE type IN 'type1', it seems that it'll first scan parquet files to get FilePath and Position of rows, then shuffle by FilePath to update DV. I'm curious if we can directly use the bitmap index to update DV, skipping the file scan and shuffling.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[steFaiz]

Hello! Now global index is designed for append table only. PK tables could be frequently updated, which is not suitable for current global index design.

[fafacao86]

Hello! Now global index is designed for append table only. PK tables could be frequently updated, which is not suitable for current global index design.

I'm using Bitmap File Index, which is not a global index probably? I'm trying to figure out how to speed up Delete by a non-primary column for PrimaryKey Table. If the delete-by non-primary column has a file index, we might be able to directly utilize the file index to update DV?