Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions GRAMMAR.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,59 @@ and [arguments](#arguments).
- **`string`**` := "'" ("\\" . / !"'" .)* "'"`
- Examples: `'hello'`, `'table name'`, `'C:\path\to\file'`, `'line1\nline2'`, `'quote\'s here'`
- **`null`**` := "null"`
- Examples: `null:i64?`, `null:string?`, `null:date?`
- A type annotation is required for `null`
- **`typed_literal`**` := string ":" type`
- String literals with type annotations for non-primitive types
- Examples: `'2023-01-01':date`, `'2023-12-25T14:30:45.123':timestamp`, `'2023-01-01T12:00:00.123456789':precisiontimestamp<9>`, `'14:30:45.123456':precisiontime<6>`, `'5d 3s':interval_day<0>`

All basic literal types (`integer`, `float`, `boolean`, and `string`) are supported, plus `date`, `time`, `timestamp`, `precisiontime`, `precisiontimestamp`, `precisiontimestamptz`, `interval_day`, and typed null literals. Other Substrait literal types (e.g., `interval_year`, `decimal`, `uuid`) are not yet implemented. The deprecated `timestamp_tz` literal is also not yet implemented; use `precisiontimestamptz<6>` instead.

#### `interval_day` Typed Literals

`interval_day` string literals represent Substrait `IntervalDayToSecond` values.
The string holds up to three duration terms:

```text
interval_day_literal := (duration_days (" " duration_seconds)? (" " duration_subseconds)?)
/ (duration_seconds (" " duration_subseconds)?)
/ duration_subseconds
duration_days := "-"? digit+ "d"
duration_seconds := "-"? digit+ "s"
duration_subseconds := "-"? digit+ subsecond_unit
subsecond_unit := "ms" / "us" / "ns" / "ps"
```

Each term is optional, but at least one is required, and terms appear in
descending order: days, then seconds, then sub-seconds. Terms are separated by
exactly one space, with no leading or trailing whitespace. Each term carries its
own optional sign, matching the separate Substrait fields for days, seconds, and
sub-seconds.

Sub-second precision comes from the type ascription, not the string, so an
`interval_day` literal always names its precision: `interval_day<precision>`.
A literal's precision must be one of 0 (seconds), 3 (milliseconds), 6
(microseconds), 9 (nanoseconds), or 12 (picoseconds) - the precisions that have
a unit to write a value in. Unlike `precisiontimestamp` and `precisiontime`
literals, `interval_day` accepts 12: sub-seconds are stored as a plain integer
count rather than going through `chrono`.

A sub-second term's unit must agree with the ascribed precision - `ms` is
precision 3, `us` is 6, `ns` is 9, and `ps` is 12 - so there is only one place a
value's precision can come from.

Examples:

- `'5d':interval_day<0>`
- `'4d 5s':interval_day<6>`
- `'123456789ns':interval_day<9>`
- `'5d 3s 100ms':interval_day<3>`
- `'-5d 3s':interval_day<0>`
- `'5d':interval_day?<6>` (nullable)

Only non-zero components are written on output, since precision travels in the
type suffix: an interval of 5 days at nanosecond precision is `'5d':interval_day<9>`,
not `'5d 0ns':interval_day<9>`. An all-zero interval is written `'0s'`.

## Types

Expand Down Expand Up @@ -272,6 +325,9 @@ From [official Substrait grammar](https://raw.githubusercontent.com/substrait-io
- `timestamp`, `timestamp_tz`, `date`, `time`
- `interval_year`, `uuid`

`interval_day` is not in this list: it is parameterized by sub-second precision,
so it is written as a compound type (see below).

#### Nullability

- `?` - nullable
Expand Down Expand Up @@ -338,6 +394,12 @@ precisiontimestamptz<3>
precisiontimestamptz?<3>
```

`interval_day` takes an integer sub-second precision from 0 to 12, e.g.
`interval_day<9>`, `interval_day?<0>`. The parameter is required, as it is for
every other parameterized type. Writing an `interval_day` *literal* additionally
requires a precision that has a unit to write values in; see
[`interval_day` Typed Literals](#interval_day-typed-literals).

#### Examples

// TODO: This example uses `map` type, which is not yet implemented in the parser.
Expand Down
1 change: 1 addition & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ pub use extensions::{AnyConvertible, Explainable, ExtensionRegistry};
pub mod extensions;
pub mod grammar;
mod parser;
mod precision;
mod textify;

#[cfg(test)]
Expand Down
36 changes: 34 additions & 2 deletions src/parser/expression_grammar.pest
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,14 @@ precision_timestamp_tz_type = { ^"precisiontimestamptz" ~ nullability ~ "<" ~ in
precision_timestamp_type = { ^"precisiontimestamp" ~ nullability ~ "<" ~ integer ~ ">" }
precision_time_type = { ^"precisiontime" ~ nullability ~ "<" ~ integer ~ ">" }

// interval_day takes an integer sub-second precision parameter, like the
// precision timestamp types above.
// Example: interval_day<6>, interval_day?<9>
interval_day_type = { ^"interval_day" ~ nullability ~ "<" ~ integer ~ ">" }

// A compound type expression, composed of a name, optional parameters, and optional nullability.
// Example: LIST?<fp64?>, MAP<i64, string>
// TODO: interval_day
compound_type = { list_type | map_type | struct_type | precision_timestamp_tz_type | precision_timestamp_type | precision_time_type }
compound_type = { list_type | map_type | struct_type | precision_timestamp_tz_type | precision_timestamp_type | precision_time_type | interval_day_type }

// A user-defined type expression. The u! prefix is optional and ignored for lookup purposes;
// both "json" and "u!json" resolve to the same extension.
Expand Down Expand Up @@ -412,3 +416,31 @@ extension_column = { named_column | reference | expression }
// - + Ext:BlobStoreRead['path/to/file']
addendum = { "+" ~ sp ~ addendum_type ~ ":" ~ name ~ "[" ~ extension_args ~ "]" }
addendum_type = { "Enh" | "Opt" | "Ext" }


/////////////// -- interval_day durations Specifications -- ///////////////

// The contents of an `interval_day` literal string, e.g. "5d", "4d 5s",
// "5d 3s 100ns". A day term, a seconds term, and a sub-second term, in that
// order; each is optional, but at least one is required, so each unit can
// appear at most once.
//
// This rule is deliberately not referenced from `literal`: the string is
// unescaped first, then re-parsed with this rule as the start rule. Terms are
// separated by exactly one space, with no leading or trailing whitespace -
// literals don't need the looser whitespace the surrounding grammar allows.
interval_day_duration = {
SOI ~ (
duration_days ~ (whitespace ~ duration_seconds)? ~ (whitespace ~ duration_subseconds)?
| duration_seconds ~ (whitespace ~ duration_subseconds)?
| duration_subseconds
) ~ EOI
}

duration_days = { integer ~ "d" }
duration_seconds = { integer ~ "s" }
duration_subseconds = { integer ~ subsecond_unit }

// The sub-second units that correspond to a writable `interval_day` precision:
// ms = 3, us = 6, ns = 9, ps = 12.
subsecond_unit = { "ms" | "us" | "ns" | "ps" }
Loading