fix(cubestore): keep the projection schema when rewriting a rolling window - #11810
Open
waralexrom wants to merge 2 commits into
Open
fix(cubestore): keep the projection schema when rewriting a rolling window#11810waralexrom wants to merge 2 commits into
waralexrom wants to merge 2 commits into
Conversation
One of the two aggregates is only read through an expression (`sum(m) / sum(n)`), so nothing in the projection names it. Aggregates are paired with the names the projection gives them positionally, so a short alias list truncates the rolling node's schema while the enclosing subquery keeps the wider schema it was built with; resolving a column against the narrower one panics with an index out of bounds. The multi-aggregate rewrite declines for this shape, so the query runs as a plain aggregate over the range join. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## lts/v1.6 #11810 +/- ##
=========================================
Coverage 58.89% 58.89%
=========================================
Files 225 225
Lines 17950 17950
Branches 3664 3664
=========================================
Hits 10571 10571
Misses 6854 6854
Partials 525 525
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #11410 to the 1.6 line.
Problem
A rolling-window query panics in the router instead of returning rows:
It reproduces on a cube whose measures are two rolling-window measures plus a
type: numbermeasure defined as their ratio, served from a rollup. Reported in #11647.Cause
RollingOptimizerRulebuiltrolling_aggs_aliasfrom whichever aliased columns happened toappear in the projection. A ratio measure (
sum(m) / sum(n)) means one of the two aggregates isnever read under its own name, so the alias list came out one short.
RollingWindowAggregate::schema_fromzips aggregates with aliases, so the short list silentlytruncated the node's schema while the enclosing subquery kept the schema it was built with.
optimize_projectionsthen asked the narrower schema (3 fields) for field index 3 and panicked.What this does
Cherry-pick of
1bf2dd848e, applied with no adaptation —rolling_optimizer.rsandrolling.rsare byte-identical to the upstream commit, and both lines pin the same DataFusion fork
(
cube-46.0.1@ea204976). It:order, and declines the rewrite when an aggregate has no name to output;
DISTINCT/FILTER/ORDER BY/NULLtreatment;schema is wider;
Plus a regression test for the reported shape (a projection reading one aggregate as
snand theother only inside
sum(m) / sum(n)).Note on behaviour
The fix does not make multi-aggregate rolling windows work — it declines the rewrite, so such
a query is served by a plain range join. That is deliberate and matches the upstream line: the
rolling executor advances one shared group counter across its aggregates, so every aggregate past
the first reads slots the others left empty and answers null. Declining trades a panic (and, where
the aliases happened to line up, silent nulls) for a real answer.
Two consequences of that trade are worth naming explicitly, because a review of this diff found
them and they are inherited from upstream rather than introduced here:
extractor treats
Lt/LtEqandGt/GtEqidentically and maps a bare column bound to0, sofor the join filter Cube emits the rewrite computes a two-point trailing window while the SQL
literally means a single point. Measured on identical data: the rewritten path returns
10, 10, 3, 3, 5where the range join returnsNULL, 3, NULL, 5, NULL.alias on the aggregate flips which path serves the query, and therefore the numbers. Measured
on the same shape: aliasing the aggregate
numrewrites (10, 10, 3, 3, 5), aliasing itwinso it collides with a grouped-but-unprojected partition key declines (
NULL, 3, NULL, 5, NULL).Neither is a regression from this backport -- both reproduce on the upstream line, where this code
is byte-identical. They argue for fixing the shared group counter (or refusing these shapes
outright) upstream, not for diverging the 1.6 line here.
How it was verified
by hand (the join reduces to
b.d == date_from + 1) and cross-checked on a wider shape withoverlapping buckets and several rows per bucket.
declines cleanly and returns correct values.
cargo test -p cubestore: 304 passed.cubestore-sql-tests: in-process 186, multi-process 186,migration 132, rolling subset 16/16 in all three configurations.
these commits:
create_table_with_url/table_partition_split_thresholddownload a CSV from anexternal host that currently answers 403, and the full
clustersuite is unstable under parallelload with a different set of failures on each run.
Risk
Confined to
RollingOptimizerRule. Every existing rolling-window test passes unchanged; the threethat fail without the fix are the two from the upstream commit plus the new one. Queries that were
already served by the rolling path keep their plan — only shapes the rule can no longer describe
fall back to the range join.
Release condition: publishing from a non-default line must not move the
latestandv1Docker tags for cube and cubestore. Confirm
detect_branchinpublish.ymlresolves this branchcorrectly before cutting a release from it.