Skip to content

Add posting index and covering index documentation#404

Open
nwoolmer wants to merge 14 commits intomainfrom
nw_index
Open

Add posting index and covering index documentation#404
nwoolmer wants to merge 14 commits intomainfrom
nw_index

Conversation

@nwoolmer
Copy link
Copy Markdown
Contributor

@nwoolmer nwoolmer commented Mar 31, 2026 •
edited
Loading

Summary

  • Add new documentation page for posting index and covering index (posting-index.md)
  • Update existing index, CREATE TABLE, ALTER TABLE ADD INDEX, SHOW, schema design, and SQL optimizer hints docs with posting index syntax and examples
  • Add posting index entry to sidebar navigation

Updates since initial generation

  • All column types now supported in INCLUDE — UUID, LONG256, BINARY, DECIMAL128/256, and arrays all work
  • Encoding options — document POSTING DELTA and POSTING EF syntax variants
  • Designated timestamp auto-inclusion — the designated timestamp is automatically included in covering index when INCLUDE is present
  • Out-of-line syntax — document INDEX(col TYPE POSTING) alongside inline syntax
  • SHOW COLUMNS — add indexType and indexInclude columns to show.md and table_columns() in meta.md
  • SHOW CREATE TABLE — add example rendering posting index with INCLUDE
  • CAPACITY restriction — note across all pages that CAPACITY is bitmap-only
  • INCLUDE restrictions — inline syntax and ALTER TABLE only; cannot include the indexed column itself
  • Storage details — break down compression strategy per column type category (ALP, FoR, raw copy, var-width sidecar)

Test plan

  • Verify yarn build succeeds with no broken links
  • Review posting index page renders correctly
  • Check cross-links between index pages work
  • Verify SHOW COLUMNS output matches actual server output
  • Confirm SHOW CREATE TABLE example matches actual DDL rendering

🤖 Generated with Claude Code

@nwoolmer
Add posting index and covering index documentation
cecfffb 
New page: concepts/deep-dive/posting-index.md
- When to use, creating with INDEX TYPE POSTING and INCLUDE
- Covering index: how it works, supported types, choosing columns
- Verifying with EXPLAIN, comparison with bitmap index
- All accelerated query patterns with examples
- SQL optimizer hints (no_covering, no_index)
- Trade-offs: storage, write performance, memory
- Architecture: file types, generations, sealing, FSST compression
- Limitations

Updated pages:
- indexes.md: added index type comparison table
- create-table.md: added posting index and INCLUDE syntax
- alter-table-alter-column-add-index.md: added posting + INCLUDE examples
- sql-optimizer-hints.md: added no_covering and no_index hints
- schema-design-essentials.md: added indexing decision guide
- sidebars.js: added posting-index to Deep Dive navigation
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 31, 2026 •
edited
Loading

🚀 Build success!

Latest successful preview: https://preview-404--questdb-documentation.netlify.app/docs/

Commit SHA: 81f178a

📦 Build generates a preview & updates link on each commit.

nwoolmer and others added 13 commits April 10, 2026 14:29
@nwoolmer @claude
Update posting/covering index docs to match current codebase
1527456 
- All column types now supported in INCLUDE (UUID, LONG256, BINARY,
  DECIMAL128/256, arrays were added since initial docs)
- Document encoding options: POSTING DELTA and POSTING EF syntax
- Document designated timestamp auto-inclusion in covering index
- Add out-of-line INDEX(col TYPE POSTING) syntax examples
- Add SHOW COLUMNS indexType and indexInclude columns to show.md,
  meta.md (table_columns), and posting-index.md
- Add SHOW CREATE TABLE example with posting index
- Note CAPACITY restriction (bitmap only) across all relevant pages
- Note INCLUDE restrictions (inline syntax only, cannot include
  indexed column itself)
- Update storage/compression details per column type category

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@nwoolmer @claude
Add posting index docs to explain, symbol, mat-view, config pages
72b98ca 
- explain.md: add CoveringIndex and PostingIndex plan node descriptions
- symbol.md: add posting index example alongside bitmap in indexing section
- alter-mat-view-alter-column-add-index.md: add TYPE POSTING syntax
  (INCLUDE not supported on materialized views)
- _cairo.config.json: add cairo.posting.index.auto.include.timestamp and
  cairo.posting.index.row.id.encoding config keys; clarify bitmap-only
  scope of cairo.index.value.block.size and cairo.spin.lock.timeout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@nwoolmer @claude
Clarify encoding trade-offs: delta vs adaptive modes
630384d 
Delta encoding compresses best for regular, evenly-distributed data and
is faster for large scans. The adaptive (default) mode additionally
trial-encodes a flat layout that compresses better for irregular
distributions and is faster for point queries.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@nwoolmer @claude
Add EF encoding as distinct option, clarify all three modes
e187aac 
- POSTING DELTA: regular data, better compression for even distributions,
  faster for large sequential scans
- POSTING EF: Elias-Fano encoding, better compression for irregular
  distributions, faster for point queries
- POSTING (default): adaptive, trial-encodes both per stride, picks smaller

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@nwoolmer
Merge remote-tracking branch 'origin/main' into nw_index
5ef24ea 
# Conflicts:
#	documentation/query/functions/meta.md
#	documentation/query/sql/alter-table-alter-column-add-index.md
@nwoolmer @claude
Correct posting/covering index facts verified against live instance
96b8930 
posting-index.md:
- Auto-include of designated timestamp applies to any posting index, not
  only when an INCLUDE clause is present (verified in source and EXPLAIN).
  Document SHOW CREATE TABLE round-trip with the expanded list.
- Note that bare INDEX INCLUDE (...) auto-promotes to POSTING.
- Replace "max 125 generations" with the actual seal threshold of 16
  (cairo.posting.seal.gen.threshold).
- Distinguish the two seal-time sub-layouts (Delta sub-layout and Flat
  sub-layout, both internal to delta+FoR) from the SQL DELTA / EF
  encoding variants — they were previously conflated.
- Note the native AVX2 fast path for 8/16/32-bit widths.
- Bitmap storage size: ~15 B/value (PR benchmark figure) instead of
  the older 8-16 B/value range.
- Write-perf comparison baselined against bitmap (~9% slower for the
  index path itself) instead of vs. no-index.
- FSST symbol table is ~2.3 KB and L1-resident; drop the unverified
  ~70 KB per-reader figure.
- Generalise the SAMPLE BY limitation: covering needs a filter on the
  indexed symbol, otherwise unfiltered LATEST ON / SAMPLE BY / GROUP BY
  fall back to a regular page-frame scan.
- Refresh EXPLAIN snippets to match real output: IN-list filter
  rendering, LATEST ON without SelectedRecord wrapper, DISTINCT as
  PostingIndex op: distinct, Async Filter layered on top of
  CoveringIndex for AND filters on covered columns.
- Tighten architecture: .pv encoding depends on variant (delta+FoR or
  EF); .pcN sidecars carry txn-segment suffixes on disk and the
  auto-included timestamp gets its own sidecar.

_cairo.config.json:
- cairo.posting.index.row.id.encoding: default is `adaptive`, valid
  values are `adaptive`, `delta`, `ef` (not the previous
  `posting`/`posting_delta`).
- cairo.posting.index.auto.include.timestamp: clarify that it applies
  to any posting index, including bare INDEX TYPE POSTING.
- Add cairo.posting.seal.gen.threshold (default 16).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nwoolmer @claude
Refine posting/covering index facts after deeper source review
eda31c8 
posting-index.md:
- Encoding-options: Elias-Fano is per-key Elias-Fano coding (low/high
  bit split), not "stride-wide flat layout with FoR". Rewrite the EF
  description to match the actual algorithm and recharacterise the
  three SQL variants as choices that pick the per-key encoding the
  writer uses, with the explicit DELTA/EF variants positioned for
  benchmarking rather than tied to vague data-distribution claims.
- INCLUDE type table: BOOLEAN/BYTE/etc. use Frame-of-Reference
  bitpacking, not raw copies. Split FLOAT/DOUBLE into their own rows
  (both ALP) and TIMESTAMP into its own row (linear-prediction + FoR).
  BINARY/arrays are length-prefixed raw bytes, not "offset-based
  sidecar".
- Trade-offs storage section: same correction — small fixed-width
  types use FoR bitpacking; only UUID / LONG256 / DECIMAL128/256 are
  raw copies.
- SHOW COLUMNS example: column order now matches live output
  (indexType / indexInclude come last, after upsertKey), and adds the
  symbolTableSize column. The indexInclude value shows
  exchange,price,timestamp to reflect auto-include of the timestamp.

meta.md:
- table_columns(): description list adds symbolTableSize and reorders
  indexType / indexInclude to the end (where they actually appear).
- Example table column order matches live output and includes
  symbolTableSize.

show.md:
- SHOW COLUMNS example: column order corrected (indexType /
  indexInclude at end, symbolCached / symbolCapacity / symbolTableSize
  before designated). Mention POSTING DELTA / POSTING EF as possible
  indexType values.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nwoolmer @claude
Drop speculative data-distribution claims from encoding example block
e18de56 
The table above the example block was already corrected to drop the
unverified "irregular data, point queries" / "regular data, large
scans" claims about EF and DELTA. Update the example block's inline
comments to match — both explicit variants are positioned as
benchmarking-only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nwoolmer @claude
Clarify auto-include of timestamp on ALTER ADD INDEX
a3f9cfb 
alter-table-alter-column-add-index.md:
- State explicitly that bare ALTER ... ADD INDEX TYPE POSTING (no
  INCLUDE clause) already covers timestamp + symbol queries because the
  designated timestamp is auto-included.
- Add the EF variant alongside DELTA in the encoding-variant example.

alter-mat-view-alter-column-add-index.md:
- Replace the "INCLUDE not supported, use posting without INCLUDE"
  note with a more accurate explanation: the parser rejects an
  explicit INCLUDE clause on materialized views, but the view's
  designated timestamp is still auto-added, so the bare form produces
  a covering index over timestamp. Verified live via table_columns().

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nwoolmer @claude
Distinguish filtered vs unfiltered LATEST ON in bitmap/posting compar…
3b0df23 
…ison

Live verification: bitmap uses LatestByAllIndexed for unfiltered
LATEST ON (index-accelerated), while posting falls back to
LatestByDeferredListValuesFiltered for the unfiltered case. The
previous "LATEST ON | Yes | Yes" row hid this difference. Split into
two rows so readers see that bitmap retains the edge for unfiltered
LATEST ON, while posting wins on filtered LATEST ON via the covering
path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nwoolmer @claude
Tighten .pci description and COUNT example comment
83f84f5 
posting-index.md:
- .pci was described as "per-column header" but it's a single index-
  level header listing all covered columns by writer index (PCI1
  magic, count, writerIndex array). Reword accordingly.
- COUNT example comment said "uses index" but the actual plan is
  Count over CoveringIndex with no column data read. Make the comment
  describe what the plan node actually says.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nwoolmer @claude
Reflect auto-included timestamp in SHOW CREATE TABLE posting example
12abb60 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@javier
Merge main and apply posting index config properties
81f178a 
Resolve conflict from deleted _cairo.config.json by applying PR 404
changes directly to the new cairo-engine.md epigraph format:
- Add cairo.posting.index.auto.include.timestamp
- Add cairo.posting.index.row.id.encoding
- Add cairo.posting.seal.gen.threshold
- Update cairo.index.value.block.size description (bitmap only)
- Update cairo.spin.lock.timeout description (bitmap and posting)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

DO NOT MERGE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nwoolmer @javier