Skip to content

Document period materialized views #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 57 additions & 21 deletions documentation/guides/mat-views.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,10 @@ As data grows in size, the performance of certain queries can degrade.
Materialized views store the result of a `SAMPLE BY` or time-based `GROUP BY`
query on disk, and keep it automatically up to date.

The refresh of a materialized view is `INCREMENTAL` and very efficient, and
using materialized views can offer 100x or higher query speedups. If you require
the lowest latency queries, for example, for charts and dashboards, use
materialized views!
The refresh of a materialized view is incremental and very efficient, and using
materialized views can offer 100x or higher query speedups. If you require the
lowest latency queries, for example, for charts and dashboards, use materialized
views!

For a better understanding of what materialized views are for, read the
[introduction to materialized views](/docs/concept/mat-views/) documentation.
Expand Down Expand Up @@ -106,7 +106,7 @@ If you are unfamiliar with the OHLC concept, please see our

```questdb-sql title="trades_OHLC_15m DDL"
CREATE MATERIALIZED VIEW 'trades_OHLC_15m'
WITH BASE 'trades' REFRESH INCREMENTAL AS
WITH BASE 'trades' REFRESH IMMEDIATE AS
SELECT
timestamp, symbol,
first(price) AS open,
Expand All @@ -124,9 +124,9 @@ In this example:
2. The base table is `trades`
- This is the data source, and will trigger incremental refresh when new data
is written.
3. The refresh strategy is `INCREMENTAL`
- The data is automatically refreshed and incrementally written; efficient,
fast, low maintenance.
3. The refresh strategy is `IMMEDIATE`
- The data is automatically refreshed and incrementally written after a base
table transaction occurs; efficient, fast, low maintenance.
4. The `SAMPLE BY` query contains two key column (`timestamp`, `symbol`) and
five aggregates (`first`, `max`, `min`, `last`, `price`) calculated in `15m`
time buckets.
Expand Down Expand Up @@ -172,13 +172,58 @@ will not trigger any sort of refresh.

#### Refresh strategies

Currently, only `INCREMENTAL` refresh is supported. This strategy incrementally
updates the view when new data is inserted into the base table. This means that
only new data is written to the view, so there is minimal write overhead.
The `IMMEDIATE` refresh strategy incrementally updates the view when new data is
inserted into the base table. This means that only new data is written to the
view, so there is minimal write overhead.

Upon creation, or when the view is invalidated, a full refresh will occur, which
rebuilds the view from scratch.

Other than `IMMEDIATE` refresh, QuestDB supports `MANUAL` and timer
(`EVERY <interval>`) strategies for materialized views. Manual strategy means
that to refresh the view, you need to run the
[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly. In case of
timer-based refresh the view is refreshed periodically, at the specified
interval.

## Period materialized views

In certain use cases, like storing trading day information, the data becomes
available at fixed time intervals. In this case, `PERIOD` variant of
materialized views can be used:

```questdb-sql title="Period materialized view"
CREATE MATERIALIZED VIEW trades_daily_prices
REFRESH PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS
SELECT
timestamp,
symbol,
avg(price) AS avg_price
FROM trades
SAMPLE BY 1d;
```

Refer to the following
[documentation page](/docs/reference/sql/create-mat-view/#period-materialized-views)
to learn more on period materialized views.

## Initial refresh

As soon as a materialized view is created an asynchronous refresh is started. In
situations when this is not desirable, `DEFERRED` keyword can be specified along
with the refresh strategy:

```questdb-sql title="Deferred manual refresh"
CREATE MATERIALIZED VIEW trades_daily_prices
REFRESH MANUAL DEFERRED AS
...
```

The `DEFERRED` keyword can be specified for any refresh strategy. Refer to the
following
[documentation page](/docs/reference/sql/create-mat-view/#initial-refresh) to
learn more on the keyword.

#### SAMPLE BY

Materialized views are populated using `SAMPLE BY` or time-based `GROUP BY`
Expand Down Expand Up @@ -357,7 +402,7 @@ useful.
## Limitations

- Not all `SAMPLE BY` syntax is supported, for example, `FILL`.
- `INCREMENTAL` refresh is only triggered by inserts into the `base` table, not
- `IMMEDIATE` refresh is only triggered by inserts into the `base` table, not
join tables.

## LATEST ON materialized views
Expand Down Expand Up @@ -555,15 +600,6 @@ partitioning capabilities.

### Refresh mechanism

:::note

Currently, QuestDB only supports **incremental refresh** for materialized views.

Future releases will include additional refresh types, such as time-interval and
manual refreshes.

:::

Unlike regular views, which recompute their results at query time, materialized
views in QuestDB are incrementally refreshed as new data is added to the base
table. This approach ensures that only the **relevant time slices** of the view
Expand Down
4 changes: 2 additions & 2 deletions documentation/reference/function/meta.md
Original file line number Diff line number Diff line change
Expand Up @@ -518,8 +518,8 @@ materialized_views();

| view_name | refresh_type | base_table_name | last_refresh_start_timestamp | last_refresh_finish_timestamp | view_sql | view_table_dir_name | invalidation_reason | view_status | refresh_base_table_txn | base_table_txn | refresh_limit_value | refresh_limit_unit | timer_start | timer_interval_value | timer_interval_unit |
|------------------|--------------|-----------------|------------------------------|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|-------------|------------------------|----------------|---------------------|--------------------|-------------|----------------------|---------------------|
| trades_OHLC_15m | incremental | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null |
| trades_latest_1d | incremental | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null |
| trades_OHLC_15m | immediate | trades | 2025-05-30T16:40:37.562421Z | 2025-05-30T16:40:37.568800Z | SELECT timestamp, symbol, first(price) AS open, max(price) as high, min(price) as low, last(price) AS close, sum(amount) AS volume FROM trades SAMPLE BY 15m | trades_OHLC_15m~27 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null |
| trades_latest_1d | immediate | trades | 2025-05-30T16:40:37.554274Z | 2025-05-30T16:40:37.562049Z | SELECT timestamp, symbol, side, last(price) AS price, last(amount) AS amount, last(timestamp) as latest FROM trades SAMPLE BY 1d | trades_latest_1d~28 | null | valid | 55141609 | 55141609 | 0 | null | null | 0 | null |


## version/pg_catalog.version
Expand Down
13 changes: 8 additions & 5 deletions documentation/reference/sql/alter-mat-view-set-refresh-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,17 @@ Changes a materialized view's refresh to run on a schedule.

## Description

Sometimes, the view may not need to be updated eagerly. For example, perhaps the data is only queried every five minutes.
Sometimes, the view may not need to be updated eagerly. For example, perhaps the
data is only queried every five minutes.

In this circumstance, you can defer updating the view in small pieces, and instead let it be updated in a larger
incremental write every five minutes.
In this circumstance, you can defer updating the view in small pieces, and
instead let it be updated in a larger incremental write every five minutes.

The schedule is defined using a start time and then a timing unit, with a minimum of `1m`.
The schedule is defined using a start time and then a timing unit, with a
minimum of `1m`.

Each triggered refresh is itself incremental, and will only consider data since the last refresh.
Each triggered refresh is itself incremental, and will only consider data since
the last refresh.

The unit follows the same format as [SAMPLE BY](/docs/reference/sql/sample-by/).

Expand Down
171 changes: 154 additions & 17 deletions documentation/reference/sql/create-mat-view.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,38 +91,142 @@ with the designated timestamp as the grouping key.

:::

## Alternate refresh modes
## Alternative refresh strategies

By default, QuestDB will incrementally refresh the view each time new data is written to the base table.
With the default `IMMEDIATE` refresh strategy, QuestDB will incrementally
refresh the view each time new data is written to the base table. If your data
is written rapidly in small transactions, this will trigger additional small
writes to the view.

If your data is written rapidly in small transactions, this will trigger additional small writes to the view.
Instead, you can use timer-based refresh, which trigger an incremental refresh
after certain time intervals:

Instead, you can specify a refresh schedule, which trigger and incremental refresh after certain time intervals:
```questdb-sql
CREATE MATERIALIZED VIEW price_1h
REFRESH EVERY 1h START '2025-05-30T00:00:00.000000Z' TIME ZONE 'Europe/Berlin'
AS ...
```

In this example, the view will start refreshing from the specified timestamp in
Berlin time zone on an hourly schedule. The refresh itself will still be
incremental, but will no longer be triggered on every new insert. You can omit
the `START <timestamp>` and `TIME ZONE <timezone>` clauses in order to just
start refreshing from `now`.

:::tip

The minimum timed interval is one minute (`1m`). If you need to refresh faster
than this, please use the default incremental refresh.

:::

In case you want to be in full control of when the incremental refresh happens,
you can use `MANUAL` refresh:

```questdb-sql
CREATE MATERIALIZED VIEW price_1h REFRESH START '2025-05-30T00:00:00.000000Z' EVERY 1h AS ...
CREATE MATERIALIZED VIEW price_1h
REFRESH MANUAL
AS ...
```

In this example, the view will start refreshing from the specified timestamp on an hourly schedule.
Manual strategy means that to refresh the view, you need to run the
[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly.

The refresh itself will still be incremental, but will no longer be triggered on every new insert.
For all these strategies, the refresh itself stays incremental, i.e. the
materialized view is only updated for base table time intervals that received
modifications since the previous refresh.

You can omit the `START <timestamp>` clause in order to just start refreshing from `now`.
## Period materialized views

In certain use cases, like storing trading day information, the data becomes
available at fixed time intervals. In this case, `PERIOD` variant of
materialized views can be used:

:::tip
```questdb-sql title="Period materialized view"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS
SELECT
timestamp,
symbol,
avg(price) AS avg_price
FROM trades
SAMPLE BY 1h;
```

The minimum timed interval is one minute (`1m`). If you need to refresh faster than this, please use
the default incremental refresh.
The `PERIOD` clause above defines an in-flight time interval (period) in the
`trades_daily_prices` materialized view that will not receive data until it
finishes. In this example, the interval is one day (`LENGTH 1d`) in London time
zone. The `DELAY 2h` clause here means that the data for the trading day may
have 2 hour lag until it's fully written. So, in our example the current
in-flight period in the view is considered complete and gets refreshed
automatically each day at 2AM, London time. Since the default `IMMEDIATE`
refresh strategy is used, all writes to older, complete periods in the base
table lead to an immediate and asynchronous refresh in the view once the
transaction is committed.

Period materialized views can be used with any supported refresh strategy, not
only with the `IMMEDIATE` one. For instance, they can be configured for
timer-based refresh:

```questdb-sql title="Period materialized view with timer refresh"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH EVERY 10m PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS
...
```

:::
Here, the `PERIOD` refresh still takes place once a period completes, but
refreshes for older rows take place each 10 minutes.

Finally, period materialized views can be configure for manual refresh:

```questdb-sql title="Period materialized view with timer refresh"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH MANUAL PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS
...
```

The only way to refresh data on such a materialized view is to run
[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly. When run,
`REFRESH` statement will refresh incrementally all recently completed periods,
as well as all time intervals touched by the recent write transactions.

## Initial refresh

As soon as a materialized view is created an asynchronous refresh is started. In
situations when this is not desirable, `DEFERRED` keyword can be specified along
with the refresh strategy:

```questdb-sql title="Deferred manual refresh"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH MANUAL DEFERRED AS
...
```

In the above example, the view has manual refresh strategy and it does not
refresh after creation. It will only refresh when you run the
[`REFRESH` SQL](/docs/reference/sql/refresh-mat-view/) explicitly.

The `DEFERRED` keyword can be also specified for `IMMEDIATE` and timer-based
refresh strategies. Here is an example:

```questdb-sql title="Deferred timer refresh"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH EVERY 1h DEFERRED START '2026-01-01T00:00:00' AS
...
```

In such cases, the view will be refreshed only when the corresponding event
occurs:

- After the next base table transaction in case of `IMMEDIATE` refresh strategy.
- At the next trigger time in case of timer-based refresh strategy.

## Base table

Incrementally refreshed views require that the base table is specified, so that
the server refreshes the materialized view each time the base table is updated.
When creating a materialized view that queries multiple tables, you must specify
one of them as the base table.
Materialized views require that the base table is specified, so that the last
base table transaction number can be saved and later on checked by the
incremental refresh. When creating a materialized view that queries multiple
tables, you must specify one of them as the base table.

```questdb-sql title="Hourly materialized view with LT JOIN"
CREATE MATERIALIZED VIEW trades_ext_hourly_prices
Expand Down Expand Up @@ -195,6 +299,39 @@ CREATE MATERIALIZED VIEW trades_hourly_prices AS (
) PARTITION BY DAY TTL 7 DAYS;
```

```questdb-sql title="Creating a materialized view with one day period"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH PERIOD (LENGTH 1d TIME ZONE 'Europe/London' DELAY 2h) AS
SELECT
timestamp,
symbol,
avg(price) AS avg_price
FROM trades
SAMPLE BY 1h;
```

```questdb-sql title="Creating a materialized view with timer refresh each 10 minutes"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH EVERY 10m START '2025-06-18T00:00:00.000000000' AS
SELECT
timestamp,
symbol,
avg(price) AS avg_price
FROM trades
SAMPLE BY 1h;
```

```questdb-sql title="Creating a materialized view with manual refresh"
CREATE MATERIALIZED VIEW trades_hourly_prices
REFRESH MANUAL AS
SELECT
timestamp,
symbol,
avg(price) AS avg_price
FROM trades
SAMPLE BY 1h;
```

## IF NOT EXISTS

An optional `IF NOT EXISTS` clause may be added directly after the
Expand All @@ -220,7 +357,7 @@ Materialized view names follow the

When a user creates a new materialized view, they are automatically assigned all
materialized view level permissions with the `GRANT` option for that view. This
behaviour can can be overridden using `OWNED BY`.
behavior can can be overridden using `OWNED BY`.

If the `OWNED BY` clause is used, the permissions instead go to the user, group,
or service account named in that clause.
Expand Down
7 changes: 5 additions & 2 deletions static/images/docs/diagrams/.railroad
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,10 @@ enableDedup
createMatViewDef
::= 'CREATE' 'MATERIALIZED' 'VIEW' ('IF' 'NOT' 'EXISTS')? viewName
('WITH BASE' baseTableName)?
('REFRESH' ( 'INCREMENTAL' | ( ('START' timestamp)? 'EVERY' interval) ))?
(
'REFRESH' ((('IMMEDIATE' | 'MANUAL') ('DEFERRED')?) | ('EVERY' interval ('DEFERRED')? ('START' timestamp)? ('TIME' 'ZONE' timezone)?) )?
('PERIOD' '(' 'LENGTH' length ('TIME' 'ZONE' timezone)? ('DELAY' delay)? ')')
)?
'AS'
('(')?
(query)
Expand Down Expand Up @@ -394,7 +397,7 @@ alterMatViewSetRefreshLimit
::= 'ALTER' 'MATERIALIZED' 'VIEW' viewName 'SET' 'REFRESH' 'LIMIT' n ('HOURS' | 'DAYS' | 'WEEKS' | 'MONTHS' | 'YEARS')

alterMatViewSetRefreshStart
::= 'ALTER' 'MATERIALIZED' 'VIEW' viewName 'SET' 'REFRESH' ( ('START' timestamp)? 'EVERY' interval )
::= 'ALTER' 'MATERIALIZED' 'VIEW' viewName 'SET' 'REFRESH' (('START' timestamp)? 'EVERY' interval)

refreshMatView
::= 'REFRESH' 'MATERIALIZED' 'VIEW' viewName ('FULL' | 'INCREMENTAL' | ('INTERVAL' 'FROM' fromTimestamp 'TO' toTimestamp))
Expand Down
Loading