fix(knowledge): run the projection source/ACL backfill in the background instead of the deploy by waleedlatif1 · Pull Request #8058 · simstudioai/sim

waleedlatif1 · 2026-09-20T17:06:22Z

Summary

Script migration 0021_embedding_search_connector backfilled the ranking projections' connector_id and acl columns synchronously, 500 rows per page under a 60s statement timeout. On embedding_search every filled row is re-inserted into each HNSW index, so a page's cost is index maintenance rather than the plan: one 500-row page ran ~68s cold (~130 ms per row), the first page hit the timeout (57014), and the migration and the deploy failed with it. No page size or timeout makes a synchronous backfill of an HNSW-indexed projection fit inside a deploy job.
Search no longer depends on the backfill: projectionCandidateAccessCondition decides a row whose acl IS NULL on its document — acl IS NULL AND EXISTS (SELECT 1 FROM document WHERE document.id = <row>.document_id AND <knowledgeCandidateAccessConditionForConnectors>) — the per-candidate join every row paid before the columns existed, so bounded candidate pools (vector walk, exact slice, Tin window) hold only rows hydration will keep. A filled row is still decided on its mirrored columns. knowledgeCandidateAccessConditionForConnectors at hydration is unchanged and remains the security gate. Covers both the vector walk and the Tin keyword window, which share the function.
The registered migration is now 0022_projection_source_acl_backfill, which supersedes 0021 (the shape 0016 uses over 0015): it installs the triggers and builds the indexes only (CREATE INDEX CONCURRENTLY IF NOT EXISTS, on one reserved connection so the session-scoped lock timeout covers every build; plus a partial index on (id) WHERE acl IS NULL per projection so the unfilled branch stays an index probe beside the ACL GIN and the backfill's keyset pages read from it). Both are idempotent, so the re-run on the next deploy completes quickly, and a database that already recorded 0021 still gains the unfilled indexes. The 0021 file keeps the shared helpers and its direct db:push entry point, which still backfills inline.
backfillProjectionSourceAcl stays exported and is now resumable: keyset pages of 100 unfilled rows, each its own transaction with a per-page statement timeout (exported as PROJECTION_SOURCE_ACL_PAGE_TIMEOUT_MS), one statement per page with scalar binds only, a pause between pages (sleep), a time budget that returns the cursor, progress logging, and an ANALYZE on completion.
New projection-source-acl-backfill Trigger.dev task (apps/sim/background/), modeled on table-backfill: one run at a time, fills for up to an hour, then triggers its continuation from the cursor until both projections are filled. Safe to start repeatedly — a run only fills rows still unset.
Started after the deploy with bun apps/sim/scripts/backfill-projection-source-acl.ts, which enqueues the task the way the table backfill is enqueued (tasks.trigger + region when a Trigger worker is configured, runDetached of the same runner in-process otherwise). Documented in the migration's TSDoc as the operator path.

Type of Change

Bug fix

Testing

predicate.test.ts: renders the on-row predicate with the document branch for unfilled rows; predicate.postgres.test.ts (real database): an unfilled chunk of a document the caller cannot read is refused on the row, unfilled chunks of readable documents are admitted; queries.test.ts: the Tin window carries the branch. Each fails with the predicate change reverted.
New 0021_embedding_search_connector.postgres.test.ts: up re-runs cleanly and builds the unfilled indexes; the backfill fills only unset rows in pages, skips a chunk whose document changed, stops at its budget with a cursor and resumes after it; a rerun writes nothing. The index and budget tests fail with those changes reverted.
New projection-source-acl-backfill.test.ts: projection order, cursor resume, budget return, connection close on failure, Trigger vs detached dispatch.
packages/db registry and push tests, bun run type-check in apps/sim and packages/db, bun run lint, bun run check:audits (47 audits), docs-manifest:check, block-registry check: all pass.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

…und instead of the deploy Script migration 0021 filled the ranking projections' connector_id and acl columns synchronously, 500 rows per page under a 60s statement timeout. On embedding_search every filled row is re-inserted into each HNSW index, so a page's cost is index maintenance rather than the plan: one page ran past the timeout and the migration, and the deploy, failed. The migration now installs the triggers and builds the indexes only, both idempotent, and the backfill runs from a Trigger.dev task in keyset pages of unfilled rows, paced with a pause between pages and chained across bounded runs. Search does not depend on it: an unfilled row passes the on-row candidate predicate and is decided at hydration under the full document predicate, exactly as every candidate was before the columns existed. Claude-Session: https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

vercel · 2026-09-20T17:06:29Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 20, 2026 5:59pm UTC

greptile-apps · 2026-09-20T17:08:58Z

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding findings affecting correctness or repository requirements.

Summary

This PR moves the projection source/ACL data backfill out of the deployment migration while preserving access filtering for rows that have not yet been filled.

Registers an idempotent replacement migration that installs triggers and concurrent indexes without synchronously rewriting HNSW-indexed rows.
Adds a resumable, paginated background backfill with per-page transactions, pacing, time budgets, cursors, and Trigger.dev continuations.
Falls back to document-level candidate authorization while mirrored projection ACL data is absent.
Adds unit and PostgreSQL coverage for access behavior, migration reruns, paging, resumption, and dispatch.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Deploy runs migration 0022] --> B[Install projection triggers]
  B --> C[Build ACL, source, and unfilled-row indexes]
  C --> D[Post-deploy backfill is started]
  D --> E{Trigger.dev configured?}
  E -->|Yes| F[Run bounded background task]
  E -->|No| G[Run operator script inline]
  F --> H[Fill one projection in keyset pages]
  G --> H
  H --> I{Budget exhausted?}
  I -->|Yes| J[Return cursor and enqueue continuation]
  J --> H
  I -->|No| K[Analyze completed projection]
  K --> L{Another projection?}
  L -->|Yes| H
  L -->|No| M[Backfill complete]
  C --> N[Search continues during backfill]
  N --> O{Projection ACL filled?}
  O -->|Yes| P[Filter using mirrored ACL and source]
  O -->|No| Q[Filter through the source document]

_{Reviews (6) · Last reviewed commit: "fix(knowledge): check the projection bac..."}

cubic-dev-ai

All reported issues were addressed across 10 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

… start the backfill from the outbox An unfilled row no longer passes the on-row candidate predicate outright: it is decided on its document under the resolved candidate predicate, the join per candidate every row paid before the columns existed, so the bounded candidate pools and the exact slice hold only rows that hydration will keep. A partial index on the unfilled rows keeps that branch, and the backfill's keyset pages, an index probe. The migration also leaves one outbox event whose handler starts the backfill task, so it runs after every deploy without an operator. Claude-Session: https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

waleedlatif1 · 2026-09-20T17:20:34Z

@greptile

waleedlatif1 · 2026-09-20T17:20:38Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T17:20:43Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 12 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…out a worker fills the projections Without a Trigger.dev worker the outbox handler no longer detaches the backfill and completes the event; it runs one bounded slice per outbox run and yields with continueOutboxHandler until both projections are filled, so a restart loses at most one slice and the event is never marked done ahead of the work. The index builds run on one reserved connection, so the session-scoped lock timeout covers every build and its reset. Claude-Session: https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

waleedlatif1 · 2026-09-20T17:29:37Z

@greptile

waleedlatif1 · 2026-09-20T17:29:41Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T17:29:47Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 13 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…ill is started The outbox event and its slice-and-defer handler are gone; nothing else in the repo starts background work that way. The fill is started from the app side as the table backfill is: tasks.trigger on the Trigger.dev worker when one is configured, runDetached in-process otherwise, with the manual script as the operator path after a deploy. The registered migration is now 0022_projection_source_acl_backfill, which supersedes 0021 so a database that already recorded the synchronous shape still gains the unfilled indexes. The page statement timeout is exported so a caller bounding a run can leave it as headroom. Claude-Session: https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

waleedlatif1 · 2026-09-20T17:42:20Z

@greptile

waleedlatif1 · 2026-09-20T17:42:23Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T17:42:29Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 15 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…positive integer The page size is interpolated into the page statement and a page of nothing would report the projection filled, so a payload that asks for either is refused before the first page rather than quietly reshaped. Claude-Session: https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

waleedlatif1 · 2026-09-20T17:50:52Z

@greptile

waleedlatif1 · 2026-09-20T17:50:56Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T17:51:02Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 16 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…between pages A pause that crossed the budget still let the next iteration open a page; the deadline is now checked after the pause, so a bounded run stops before its next page rather than after it. Claude-Session: https://claude.ai/code/session_01XU6c7pKRpa5CMoMHKDdqxX

waleedlatif1 · 2026-09-20T17:57:26Z

@greptile

waleedlatif1 · 2026-09-20T17:57:31Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T17:57:37Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 16 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

vercel Bot deployed to Preview September 20, 2026 17:07 View deployment

greptile-apps Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread packages/db/script-migrations/0021_embedding_search_connector.ts Outdated

Comment thread apps/sim/lib/knowledge/access/predicate.ts Outdated