Support backfilling null value vector from the default null value#19072
Open
Jackie-Jiang wants to merge 1 commit into
Open
Support backfilling null value vector from the default null value#19072Jackie-Jiang wants to merge 1 commit into
Jackie-Jiang wants to merge 1 commit into
Conversation
Add an opt-in, per-column reload-time backfill that reconstructs a column's null value vector for segments that predate null handling being enabled, by treating every stored value equal to the column's default null value as null (single-value: value == default; multi-value: single-element array holding the default null value, mirroring how ingestion records a null). The opt-in lives under the null value vector index config (`indexes.null.backfill`), backed by a new `NullValueVectorConfig` whose `enabled` state is still derived from null handling while `backfill` is user-set. Because the reconstruction is lossy (a genuine value equal to the default is also marked null), it is per-column and only enabled columns are touched; `MAP`/complex types are rejected at table-config validation time. A new `nonNull` column-metadata flag records the "null handling enabled, no nulls" case so no empty bitmap file is written and the column is not re-scanned on every reload. Segment creation and the reload backfill set it symmetrically.
Jackie-Jiang
added
feature
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in, per-column reload-time mechanism to reconstruct missing null value vectors for legacy segments by treating stored default-null sentinel values as null, and introduces a column-metadata flag to represent the “null handling enabled but no nulls” case without writing empty bitmap files.
Changes:
- Introduces
NullValueVectorConfigwith abackfillflag and updates null-vector config resolution to deriveenabledfrom null-handling while honoringbackfillfromfieldConfig.indexes.null. - Adds
isNonNullcolumn-metadata flag and wires it into both segment creation and reload-time backfill to keep behavior symmetric and make backfill idempotent. - Adds a reload-time
NullValueVectorHandlerplus validation/tests covering SV/MV, dict/raw, selectivity, idempotency, and rejection of MAP/complex opt-in.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pinot-spi/src/main/java/org/apache/pinot/spi/config/table/NullValueVectorConfig.java | New index config type carrying the backfill opt-in flag. |
| pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java | Adds the isNonNull column-metadata key constant. |
| pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/StandardIndexes.java | Tightens the null-vector index type signature to NullValueVectorConfig. |
| pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java | Persists/loads and exposes the new nonNull column-metadata flag. |
| pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/ColumnMetadata.java | Adds default isNonNull() API to represent “null-handled but no nulls” columns. |
| pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueVectorHandler.java | New reload-time handler that scans forward indexes to backfill null bitmaps from default-null sentinels. |
| pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java | Updates deserialization/validation and wires the new handler into the index type. |
| pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/nullvalue/NullValueVectorCreator.java | Tracks whether any nulls were seen to avoid writing empty bitmap files and expose isNonNull(). |
| pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java | Writes the isNonNull metadata flag at segment creation when null handling is enabled but no nulls exist. |
| pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java | Validates that scalar backfill opt-in passes and MAP backfill opt-in is rejected. |
| pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueVectorHandlerTest.java | New tests for backfill correctness across types, selectivity, creation-path flagging, and idempotency. |
| pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexTypeTest.java | Tests config resolution semantics and MAP backfill validation behavior. |
Comment on lines
+128
to
+129
| // TODO: Revisit MAP/complex backfill if complex-type null handling matures and a safe (non-occurring) sentinel | ||
| // default null value becomes available. |
Comment on lines
+101
to
104
| public static IndexType<NullValueVectorConfig, NullValueVectorReader, ?> nullValueVector() { | ||
| return (IndexType<NullValueVectorConfig, NullValueVectorReader, ?>) | ||
| IndexService.getInstance().get(NULL_VALUE_VECTOR_ID); | ||
| } |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19072 +/- ##
============================================
- Coverage 65.43% 65.42% -0.01%
Complexity 1421 1421
============================================
Files 3425 3427 +2
Lines 216173 216360 +187
Branches 34239 34287 +48
============================================
+ Hits 141449 141551 +102
- Misses 63340 63415 +75
- Partials 11384 11394 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in, per-column reload-time backfill that reconstructs a column's null value vector for segments that were created before null handling was enabled, by treating every stored value equal to the column's default null value as null. This lets operators enable null handling on a table and have
IS NULL/ null-aware queries work on already-ingested data without re-ingesting.The reconstruction mirrors how ingestion records nulls:
Opt-in and safety
The backfill is lossy — a genuine value that happens to equal the default null value is also marked null — so it is opt-in per column, via the null value vector index config:
NullValueVectorConfigcarries thebackfillflag; itsenabledstate is still derived from null handling (column-basedisNullableor the table-levelnullHandlingEnabled), and the two are merged inNullValueIndexType's deserializer.MIN_VALUE), not for metric/boolean columns whose default is a common value like0.MAP/complex types are rejected at table-config validation time (NullValueIndexType.validate), since the default (empty map) is an ordinary value andOPEN_STRUCT-backed maps have no single scannable parent forward index. ATODOis left to revisit if complex-type null handling matures.nonNullcolumn-metadata flagA new
nonNullcolumn-metadata property records the "null handling enabled, but no null values" case. A bitmap file is written only when a column actually has nulls (unchanged from today); the no-null case is recorded via this flag instead. This keeps behavior symmetric between segment creation and backfill — no empty bitmap files — and makes the backfill idempotent (a flagged column is not re-scanned on subsequent reloads) while distinguishing a null-handled-but-empty column from one that was never null-handled.Backward / mixed-version compatibility
falseand is additive; older segments are unaffected.indexes.nullto the table config), a node still running the pre-change code will fail to resolve that table's index configs (the old null-vector deserializer treats the config as an exclusive alternative and every column already gets a derived config). Enablebackfillonly after the whole cluster is upgraded; a rollback with the config still set will fail on that table.Tests
NullValueVectorHandlerTest— backfill across SV/MV and dict/raw scalar types, selectivity (an un-opted column holding the sentinel is left untouched), the creation-timenonNullflag write, and idempotency on reload.NullValueIndexTypeTest— config resolution (enabledderived from null handling) andvalidaterejectingMAPopt-in.TableConfigUtilsTest— end-to-end validation: scalar opt-in passes,MAPopt-in is rejected.