fix(tesseract): stop a rolling window's plan fanning out across measures - #11824
fix(tesseract): stop a rolling window's plan fanning out across measures#11824waralexrom wants to merge 6 commits into
Conversation
Adds ignored planner tests that pin the plan shape reported in #11770: several rolling_window measures queried with a high-cardinality dimension scan the fact table once per measure and join time_series to each base CTE on a date range only, with no equality predicate on the dimension that is in the GROUP BY of both sides. On the reporter's cardinality (4.7K entities over 33 day-anchors) that materialises ~10M intermediate rows for a 5K-row result; Postgres plans the range-only join as a nested loop with a join filter. A third test covers the secondary observation: the base scan's date bound is emitted as a scalar sub-select over time_series, where the legacy planner emitted a literal, so engines cannot use it to eliminate partitions. The tests assert the wanted shape and fail today, so they are #[ignore]d. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeQG6eSzkVJzkcf49U2Uqu
A regular rolling window restricts its base scan to the span the window can reach, and read both ends of that span back off the time series with a scalar sub-select. The bounds are opaque to an engine that way: no partition can be eliminated by them, so every base scan of the query reads the whole table, and a query with several rolling measures pays that once per measure. The series is derived from the time dimension's granularity and date range, so its span is known while planning. Compute it there and render both bounds as literal parameters, falling back to the sub-select only where the span is not derivable — a range that is itself a query, a granularity whose periods come off a calendar cube, and the placeholder ranges of pre-aggregation SQL. The span covers one bucket past the range end: a series materialized while planning snaps its points to bucket boundaries, while one generated in SQL steps from the range start, and its last bucket can end an interval further out. The rolling join applies the exact frame on top either way, so a wider base scan changes no result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A rolling window's base CTE aggregates one measure over the rows the window can reach, and the rows it reads are decided by the window's frame and the query's filters — not by the measure. Every window still got its own CTE, so a query over several rolling measures scanned the fact table once per measure even where two windows were byte-identical apart from the column they summed. Measures whose base scans are equivalent now ride on one CTE: the leaf carries them alongside its own member, and each window reads its own column off the shared scan. Equivalence is the state comparison the multi-stage planner already deduplicates CTEs by, narrowed further to measures reading the same set of cubes — a measure reading one the scan does not would widen its join tree, which is a different scan. Windows differing in frame, filters, grain, or fact keep their own scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 7m 9s —— View job Review: solid change, one likely regression for custom granularities — 1 medium, 3 low, posted inlineFull review
What I checkedI traced both halves of the change end to end rather than reading the diff in isolation: Literal bounds — the over-approximation is sound. I verified the tail claim against the actual dialect templates rather than taking it on faith. Postgres ( Shared base scan — the equivalence check is tight enough. The thing I most wanted to disprove was two windows with different frames sharing a scan. They can't: Interior mutability is safe here. Dedup is also preserved: the new loop runs before the CTE is allocated, and Findings
On #1, the short version: Notes, not findings
|
Deriving the span a rolling window's base scan is bounded by parses the time dimension's date range. A pre-aggregation's range holds partition placeholders rather than dates, so the derivation raised a parse error where it should simply have no bounds to offer and leave the filter to read them off the series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| Ok(Some(( | ||
| self.format_and_allocate_from_date(from)?, | ||
| self.format_and_allocate_to_date(to)?, | ||
| ))) |
There was a problem hiding this comment.
Question, not a blocker: the two branches this now chooses between are not timezone-equivalent.
format_and_allocate_from_date/to_date run the value through apply_db_time_zone, and use_db_time_zone is !filters_ctx.use_local_tz — true in the common case. The fallback date_range_from_time_series reads min("date_from")/max("date_to") straight off the time_series CTE, whose bounds TimeSeries::to_sql emits as bare quote_string(from_date) with no tz wrapping.
So under a non-UTC CUBEJS_DB_TIME_ZONE the literal path shifts the base-scan bounds and the sub-select path does not. The literal path is arguably the correct one (it matches how every other date-range filter renders), but that also means this PR silently changes the base-scan window for those setups, and the Postgres integration runs quoted in the description would not catch it if they run in UTC. Is there a rolling-window test with a db timezone set?
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #11824 +/- ##
===========================================
+ Coverage 60.16% 80.22% +20.05%
===========================================
Files 239 500 +261
Lines 19203 105851 +86648
Branches 3886 3886
===========================================
+ Hits 11554 84920 +73366
- Misses 7099 20381 +13282
Partials 550 550
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…use it Three defects in how the shared rolling-window base scan and its literal bounds were derived, all found in review. Deriving the bounds of a custom granularity's series walked its interval from the granularity's origin through a helper that gives up after 50,000 steps — a guard against an interval that never converges, which a fine interval and a default origin a few years off the range trips on a range that converges perfectly well. That turned a working query into a plan error. It now aligns to the origin the way the series' own range does, without a step cap. Sharing a base scan is now confined to a scan a rolling-window stage consumes. Without a granularity there is no series to walk and no stage on top: the base CTE is the requested measure's own result, registered under that measure, which a CTE shared between measures cannot answer for. Two such measures were registering the same CTE twice, joining it twice under the second measure's name and duplicating a branch of the key union. The partition-range guard on the rendered bounds was unreachable — such a range is declined while planning, before any bounds exist — and its comment promised a protection it was not giving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 5m 15s —— View job Re-review of the two new commits: prior blocker fixed, 2 new low findings, 2 earlier threads still openFull review
What the new commits change, and whether it holds
I re-derived the upper bound against that: the SQL series steps from the aligned start, so its last point is the greatest grid point ≤ The The Filter value indices line up. Findings
Re-affirming 2 earlier threads that the new commits did not touch (left unresolved, not reposted):
Notes, not findings
· branch |
| granularity.align_date_to_origin(QueryDateTime::from_date_str(tz, bound)?) | ||
| }; | ||
| let first = align(&date_range[0])?; | ||
| let past_last = align(&date_range[1])? |
There was a problem hiding this comment.
Two small things about the new custom path.
The second alignment walks from the origin again. align_to_origin steps the interval one at a time from self.origin (no cap now, which is the right call — see the resolved thread). align(&date_range[1]) therefore repeats every step align(&date_range[0]) just took, plus the range's own length. For a 1 minute custom granularity with the default origin (start of the current year) and a range in November, that's ~450K steps done twice. The range end is on the same grid as first, so it can be reached from first instead of from the origin:
let first = align(&date_range[0])?;
let range_end = QueryDateTime::from_date_str(tz, &date_range[1])?;
let mut past_last = first.clone();
while past_last <= range_end {
past_last = past_last.add_interval(interval)?;
}
past_last = past_last.add_interval(interval)?.add_duration(Duration::seconds(-1))?;That's range-length/interval steps instead of origin-distance/interval, and it stays on exactly the same grid because add_interval from a grid point continues the same sequence the origin walk produces (including the month-clamping drift).
precision is honoured on one branch only. rolling_series_bounds computes let precision = 3 and passes it to covering_bounds_predefined, while this function hardcodes .000/.999. Today they agree, so this is only a trap for whoever changes the constant. Either take timestamp_precision here and pad with it, or drop the local and let both branches state 3 directly.
Also worth noting for coverage: covering_bounds_custom and its covering_bounds_custom_start_where_the_walked_series_does unit test were deleted, and the replacement is a private planner method with no direct test. The custom path is now pinned only indirectly, through the executed results in rolling_window::custom_granularities. A unit test asserting custom_series_bounds still starts where generate_custom starts (the assertion the deleted test made) would be cheap to keep.
| // Whether a rolling-window stage consumes this CTE. A caller taking the | ||
| // CTE as the requested member's own result registers it under that | ||
| // member, which one shared between measures cannot answer for. | ||
| feeds_window_stage: bool, |
There was a problem hiding this comment.
The two adjacent bool parameters make the call sites unreadable: add_rolling_window_base(base_member, base_state, false, false, ...) at :925 and (base_member, base_rolling_state, ungrouped, true, ...) at :1030. Nothing at either site says which false is ungrouped and which is feeds_window_stage, and swapping them compiles — the failure mode being either lost sharing or an ungrouped CTE silently grouped.
Since the distinction is exactly "does a rolling stage consume this", a two-variant enum carries it at the call site and lets the doc comment go away:
enum RollingBaseConsumer {
/// A rolling-window stage aggregates this CTE, so it may be shared.
WindowStage,
/// The caller takes the CTE as the requested member's own result and
/// registers it under that member, which a shared CTE cannot answer for.
MeasureResult,
}Call sites then read RollingBaseConsumer::MeasureResult / ::WindowStage.
A pre-aggregation only answers for a query whose every measure it carries. Sharing a base scan between rolling measures therefore made matching all-or-nothing across the merged set: a model storing one rollup per rolling measure — the `partitionedRolling` shape — silently fell back to the fact table for all of them as soon as two were queried together, which costs far more than the scan the merge saves. Sharing now yields wherever some pre-aggregation of the query's cubes rolls up one of the two measures but not the other. The measure lists are read off the declarations, so a rollup that would not have matched this query anyway also holds the merge back; that direction costs a shared scan the query could have had, the other costs the pre-aggregation. Also tightens the scanned span and closes two smaller gaps found in the same review: the span now reaches one interval past the range end rather than a whole further bucket, which is exact for a range whose ends are already bucket boundaries; a granularity declaring a zero interval is rejected instead of hanging the alignment it can never converge; and the reproduction module no longer describes its two now-passing tests as known-failing scaffolding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 5m 55s —— View job Re-review of
|
| # | Severity | Where | Issue |
|---|---|---|---|
| 1 | Low | time_series.rs:96 |
- Duration::seconds(1) + nines padding discards the range end's sub-second fraction, so a SQL-generated series whose last point carries one can end past the bound — silently short results, same shape in custom_series_bounds |
| 2 | Low | time_series.rs:78 |
The new interval param is fully derivable from granularity; nothing enforces they agree, and a mismatched pair narrows the span rather than erroring |
Finding #1 is the only one with a correctness edge, and it is narrow: it needs a dateRange whose bounds carry a sub-second fraction, on a driver that generates the series in SQL. But it is a new narrowing — the sub-select it replaces read max("date_to") exactly — and it fails as wrong numbers rather than an error, so it's worth closing while the code is being touched. The fix also folds in the precision-honoured-on-one-branch-only half of the earlier custom_series_bounds thread.
Earlier threads
Resolved: the fanout_repro module doc (PRRT_kwDOCOH19M6gooon) — this commit rewrites the paragraph to say which observation is fixed and why the join-shape test stays ignored, which is accurate.
Left unresolved, not reposted:
custom_series_bounds— the double walk is fixed, but two of the three points stand:precisionis still hardcoded as.000/.999while the predefined branch takes a parameter, and the deletedcovering_bounds_custom_start_where_the_walked_series_doesunit test still has no replacement, so the custom lower bound is pinned only indirectly throughrolling_window::custom_granularitiesresults.filter_sql_context.rs:202— theapply_db_time_zonedivergence between the literal path and thetime_seriessub-select fallback is untouched. Still a question rather than a claim.multi_stage_query_planner.rs:1215— the two adjacent unnamedboolparams are untouched.
Notes, not findings
- The
integration_rolling_window_fanout_preagg.yamlfixture is well chosen: two rollups differing only in which single rolling measure they carry is the minimal shape that distinguishes "gate present" from "gate absent", andtest_sharing_yields_to_a_rollup_per_rolling_measureasserting onused_pre_aggregationsrather than on SQL means it fails loudly rather than snapshot-drifting. - Splitting
covering_bounds_predefined_snap_outwardsinto a start-only assertion plus a bucket-aligned-end assertion is a better decomposition than the original — the old test conflated two independent properties into one equality. - Reviewed statically. There is no cargo target directory in this environment, so I did not build or run
cargo test -p cubesqlplanner; all reasoning above is from reading the code and the dialect templates.
| /// a range of any width costs the same. | ||
| pub fn covering_bounds_predefined( | ||
| granularity: &str, | ||
| interval: &SqlInterval, |
There was a problem hiding this comment.
The new interval parameter is fully derivable from granularity, and nothing enforces that the two agree.
Granularity::try_new_predefined sets granularity_interval = format!("1 {}", granularity).parse(), so for a predefined granularity the interval is always 1 {granularity} — the caller has no freedom here, it just re-derives what the function already knows. What it does gain is an invariant: pass ("week", "1 day") and first snaps to the Monday while past_end only reaches a day past the range, so the span ends up narrower than the series and the base scan silently drops the tail. That's a wrong-results failure with no error, guarded only by the call site happening to read both off the same Granularity.
Deriving it inside keeps the invariant unstateable:
pub fn covering_bounds_predefined(
granularity: &str,
date_range: &[String; 2],
timestamp_precision: u32,
) -> Result<(String, String), CubeError> {
check_precision(timestamp_precision)?;
if !check_predefined_granularity(granularity) { /* ... */ }
let interval: SqlInterval = format!("1 {granularity}").parse()?;That also removes the need for the is_zero guard at :88 and covering_bounds_predefined_reject_a_zero_interval — 1 X is never zero, so both are testing a state the type system would no longer permit.
Related: the unit tests pass "3 months" for quarter and "1 week" for week, while production passes "1 quarter" and "1 week" off granularity_interval(). Same arithmetic today (total_months = year*12 + quarter*3 + month), but it means no test exercises the pairing production actually uses.
| let range_start = QueryDateTimeHelper::parse_native_date_time(&date_range[0])?; | ||
| let range_end = QueryDateTimeHelper::parse_native_date_time(&date_range[1])?; | ||
| let first = predefined_bucket(granularity, range_start, timestamp_precision)?; | ||
| let past_end = add_interval_to_dt(range_end, interval)? - Duration::seconds(1); |
There was a problem hiding this comment.
- Duration::seconds(1) plus nines truncates the range end to a whole second, so a sub-second range end can put the bound inside the last bucket.
format_with_padding (:411) discards dt's sub-second field and appends nines, so writing range_end = S + f (S = floor_sec(range_end), 0 ≤ f < 1):
bound = floor_sec(range_end + interval - 1s) + .999 = S + interval - 0.001
The f is dropped, not carried. For the materialized series that's harmless — its buckets start on second-aligned boundaries B ≤ S, so B + interval - 0.001 ≤ bound, and the tests pin that. For a series generated in SQL it isn't: the generator steps from the raw range_start, so its points inherit range_start's sub-second fraction. If the last point L lands with L > S (possible whenever L ≤ range_end and both carry the same fraction), its bucket ends at L + interval - 0.001 > bound, and the base scan drops the rows in that gap — up to just under a second at the very end of the range.
Concretely, dateRange: ["2026-08-01T00:00:00.900", "2026-09-02T00:00:00.900"] at day granularity: generate_series steps to 2026-09-02T00:00:00.900, whose bucket runs to 2026-09-03T00:00:00.899, while the bound is 2026-09-02T23:59:59.999. Fact rows in [2026-09-03T00:00:00.000, .899] are excluded from the base scan, and the rolling join can't put back rows that were never scanned — so a sub-second-boundary date range gets silently short results where the old sub-select (max("date_to"), exact) did not.
custom_series_bounds (multi_stage_query_planner.rs:1365-1368) has the identical shape and the identical gap, including for the materialized custom series when the granularity carries a sub-second origin.
Subtracting one unit at the target precision and keeping the fraction removes the truncation entirely:
let past_end = add_interval_to_dt(range_end, interval)? - Duration::milliseconds(1);
Ok((first.start_str, QueryDateTimeHelper::format_to_date(&past_end, timestamp_precision)?))Both bounds then land exactly on the series' edges for every input, and covering_bounds_predefined_stop_at_a_bucket_aligned_end keeps its current expectation.
Fixes two of the three problems behind #11770.
Problem
A query with several
rolling_windowmeasures over a high-cardinalitydimension produces a plan whose cost grows as (entities × window × anchors).
The report is three calculated measures over five rolling sums (two distinct
windows), grouped by a 4.7K-value dimension at day granularity over 33 days —
12 CTEs plus the root, ~10M intermediate rows for a result of at most 5K, and
on BigQuery a CPU guardrail hit at 133K CPU-seconds over 170MB scanned.
Two separable causes, both addressed here:
1. The base scan's date bounds were opaque. Each rolling measure's base CTE
restricts the fact table to the span its window can reach, and read both ends of
that span back off the
time_seriesCTE:No engine can eliminate partitions by a scalar sub-select, so every base scan
read the whole table. The legacy planner emitted literals here
(
BaseQuery.dateFromStartToEndConditionSqltakes the bounds off the series ithas already computed), so this is a Tesseract regression.
2. One base scan per rolling measure. A rolling window's base CTE aggregates
one measure over the rows the window can reach, and which rows those are is
decided by the frame and the query's filters — not by the measure. Every window
still got its own CTE, so the five rolling sums produced five scans of the fact
table where only two distinct (window, filter) pairs exist.
What changed
Literal bounds. The series is derived from the time dimension's granularity
and date range, so its span is known while planning.
QueryTimeSeriesgainedcovering_bounds_predefined/covering_bounds_custom, which derive the spanper bucket rather than by walking the series (so a wide range costs nothing),
and the regular rolling-window filter renders them as literal parameters.
The span runs one interval past the range end on purpose: a series materialized
while planning snaps its points to bucket boundaries, so its last bucket ends at
most an interval past the range end, while one generated in SQL
(
generate_series,GENERATE_DATE_ARRAY, …) steps from the range startinstead, and its last point sits at most an interval before it. The span has to
cover both, and is exact wherever the range's ends are already bucket
boundaries — the common case. A wider span changes no result anyway: the rolling
join applies the exact frame on top.
The sub-select remains where the span is not derivable at plan time: a date
range that is itself a query (
time_series_get_range), a granularity whoseperiods come off a calendar cube, and the placeholder ranges of
pre-aggregation SQL.
Shared base scan. Measures whose base scans are equivalent now ride on one
leaf CTE, and each window reads its own column off it. Equivalence is the state
comparison the multi-stage planner already deduplicates CTEs by, narrowed three
ways:
not would widen its join tree, and that is a different scan;
granularity has no stage on top and its base CTE is registered as the
measure's own result;
query whose every measure it carries, so a model storing one rollup per
rolling measure — the
partitionedRollingshape — would fall back to the facttable for all of them the moment two were merged. That costs far more than the
scan the merge saves, so sharing yields to it.
Windows differing in frame, filters, grain, or fact keep their own scan.
The reported query goes from 5 base scans to 2, and from 13 CTEs to 11.
Testing
cargo test -p cubesqlplanner— 1382 pass. The three reproduction tests fromthe issue are in
rolling_window::fanout_repro; two of them are un-ignored bythis PR (
test_base_scan_date_bound_is_literal,test_base_table_is_not_rescanned_per_rolling_measure), and each wasconfirmed to fail with its fix reverted.
predefined granularity, and against an unaligned SQL-generated series.
different facts keep their own scans, and so do two without a granularity.
CUBEJS_TESSERACT_SQL_PLANNER=true: 112 passing indataschema-compiler,pre-aggregations,pre-aggregations-multi-stage,pre-aggregations-calculated-measures,pre-aggregations-time,postgres-cumulative-measures,multi-stage-grain-rolling-window,rolling-window-offset-no-granularity./code-review highrounds raised eight findings; all eight are fixed, thelast three commits carry them, and each fix was confirmed to fail with itself
reverted. The pre-aggregation regression has a dedicated fixture and test
(
test_sharing_yields_to_a_rollup_per_rolling_measure).Risks
strictly needs. Negligible at day granularity; at month granularity over a
12-month range it is ~8% more rows, traded for partition elimination that was
entirely absent.
rollup that would not have served this query anyway still holds the merge back.
Such a query keeps the plan it has today; it does not gain the shared scan.
time_seriesto its base CTE on a date range only, with no equality on thegroup-by dimension, so the engine still cannot hash-join and its row estimate
stays badly off. That is a plan-shape change for every rolling query and is
deliberately left for a separate PR;
rolling_window::fanout_repro::test_rolling_join_restricts_by_dimensionstays ignored to mark it.
🤖 Generated with Claude Code