Experimental: changelog table for crdb#3211
Draft
ecordell wants to merge 16 commits into
Draft
Conversation
appendRelationshipChangelogBatch built one multi-row INSERT with 14 columns per relationship. Postgres/pgx cap bound parameters at 65535, so a BulkLoad of more than ~4600 relationships would overflow the limit and fail the whole transaction, defeating the point of the feature. Split the insert into chunks of changelogInsertChunkSize (default 4000) executed within the same transaction, keeping the per-transaction ordinal counter continuous across chunks so (change_ts, ordinal) uniqueness is preserved. Adds a regression test that lowers the chunk size and bulk-loads across multiple chunks, asserting every row is observed over the changelog watch.
AddChangedDefinition's *core.CaveatDefinition branch looked up a prior entry under nsPrefix+name to decrement its byte size, but caveats are stored under caveatPrefix+name. The mismatch meant re-changing the same caveat within one buffering window never found the prior entry, over-counting currentByteSize on every repeat change. Shared by all datastores' watch path (Postgres, MySQL, and the CRDB changelog path); not data loss, but a real accounting bug. Adds a regression test mirroring the existing namespace-replacement test, which fails against the prior code.
…watch watchViaChangelog always buffers and dedups changes per poll interval via common.NewChanges and does not honor opts.EmissionStrategy, even though the CRDB datastore advertises WatchEmitsImmediately support. A caller requesting EmitImmediatelyStrategy silently gets buffered, checkpoint-grouped delivery instead, and Creates are reported as Touch. Document this as an accepted limitation of the experimental path rather than re-architecting it; immediate emission is moot given the follower-read latency floor anyway.
…ion_metadata table
ecordell
commented
Jul 2, 2026
| delete(record.caveatsDeleted, t.Name) | ||
|
|
||
| if existing, ok := record.definitionsChanged[nsPrefix+t.Name]; ok { | ||
| if existing, ok := record.definitionsChanged[caveatPrefix+t.Name]; ok { |
Contributor
Author
There was a problem hiding this comment.
this looks like a latent bug to me
github-actions
Bot
added
area/cli
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: for reference, hasn't been validated
Description
CockroachDB changefeeds stop emitting
resolvedtimestamps during bulk loads, which stalls SpiceDB's CRDBWatch(it relies on changefeedresolvedcheckpoints to know a revision is complete). This adds an experimental, opt-in, CRDB-only Watch path that doesn't depend on the changefeed for correctness, so Watch keeps advancing under bulk load.Behind
--datastore-experimental-crdb-changelog-watch(default off — when off, behavior is unchanged):relationship_changelogtable (hash-sharded PK on the HLC commit timestamp, row-level TTL) at init.Watchpolls the table at present time instead of consuming a changefeed. The table is append-only, so a present read sees every committed row; completeness comes from the cluster's max clock offset (checkpoint/cursor advances only toclusterNow − maxOffset), not changefeedresolved, so no write is skipped. A lightweight changefeed on the table is used only as a low-latency nudge, with the poll interval as the correctness backstop.Notes:
maxOffset(--datastore-experimental-crdb-changelog-watch-max-offset, default 250ms). Must be ≥ the cluster's--max-offset(CRDB default 500ms) or Watch can miss commits.Testing
Extensive unit and integration tests, but the impact to write throughput has not been measured, that could make this a dealbreaker.
TODO: run with
--datastore-engine cockroachdb --datastore-experimental-crdb-changelog-watchand exercise Watch during a bulk load.