fix(knowledge): apply the source and date filters inside a resolved scope's ranking by waleedlatif1 · Pull Request #8059 · simstudioai/sim

waleedlatif1 · 2026-09-20T17:09:59Z

Summary

The Search page's source and updated filters were not applied inside the ranking on the resolved-scope path — only at hydration — so a filtered search ranked candidates the filter then refused. On a production-shaped copy, updated: past week returned 0 rows although thousands of documents qualified, a small-source filter returned 4, and every filtered search's keyword leg ran to its deadline (this predates improvement(knowledge): resolve connector and member state once per search #8040)
A source filter now confines the access plan itself: every on-row predicate, the reach, and the sources the legs walk or rank are that kind of source alone; upload keeps only source-less documents
A date filter enumerates the documents it admits off a new (knowledge_base_id, source_modified_at) partial index and ranks them exactly while the planner estimates the window within the probe's limit; a wider window is walked with the date tested through the document, on the row and on the ranked keyword row
A date-bounded set is ranked exactly even when a member source has its own index, since a walk cannot see the date; the reach is keyed and counted by the plan's sources
A keyword leg whose deadline passes before its ranking is resolved is short rather than failed

Measurements

Same copy, same reader (reaches ~63% of the index), complete results; wall time over a ~80 ms round trip:

filter	share of the index it admits	before	after
source (large)	~95%	20 rows, 2.9–5.1 s	20 rows, 2.1–2.7 s
source (small)	~0.2%	4–20 rows, ~8 s (keyword timed out)	20 rows, 3.0–3.6 s
updated: past month	~12%	20 rows, ~8 s (keyword timed out)	20 rows, 2.3–4.5 s
updated: past week	~0.4%	0 rows, ~8 s (keyword timed out)	20 rows, 2.8–4.0 s

Server side the new paths are hundreds of milliseconds: past week probes off the date index (~300 ms) then ranks the bounded set exactly (~200 ms); past month is one on-row walk at 340–450 ms. Without the index the date probe was a 320–640 ms sequential scan.

Type of Change

Bug fix

Testing

Unit tests for the plan restriction, the uploads flag in both predicates, the reach keyed by sources, the date probe (filter-first, estimate-gated), the on-row date test in walks and the Tin page, exact ranking of a date-bounded set, and the keyword-leg deadline; each verified to fail with its fix reverted
Real-Postgres predicate equivalence test extended with source- and upload-restricted plans
bun run check:migrations origin/staging passes; lint, block registry, check:audits, docs manifest pass

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-09-20T17:10:01Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 20, 2026 6:23pm UTC

greptile-apps · 2026-09-20T17:13:55Z

Confidence Score: 5/5

The PR appears safe to merge, with the previously reported source-filter ranking issue fixed and no new actionable defects identified.

Summary

This PR applies source and modification-date filters during resolved-scope candidate planning and ranking rather than only during result hydration.

Restricts access plans, candidate predicates, reach calculations, and ranking paths to the selected source type.
Enumerates sufficiently small filtered document sets for exact vector and keyword ranking.
Falls back to filter-aware walks when the bounded planning estimate exceeds its deadline.
Adds a concurrent partial index for knowledge-base and source-modification lookups.
Adds coverage for source confinement, uploads, date filtering, exact ranking, reach caching, and planning timeouts.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Resolved search scope] --> B[Resolve connector access plan]
  B --> C{Source filter?}
  C -- Yes --> D[Restrict plan to matching connector type or uploads]
  C -- No --> E[Keep resolved plan]
  D --> F{Source or date filter?}
  E --> F
  F -- No --> G[Resolve normal reach]
  F -- Yes --> H[Estimate filtered document count]
  H --> I{Small enough to enumerate?}
  I -- Yes --> J[Probe permitted documents]
  J --> K[Rank bounded set exactly]
  I -- No or estimate timeout --> L[Use filter-aware ranking walk]
  G --> M[Vector and keyword retrieval]
  K --> M
  L --> M

_{Reviews (5) · Last reviewed commit: "fix(knowledge): estimate a filter's size..."}

cubic-dev-ai

All reported issues were addressed across 10 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-20T17:18:11Z

@greptile

waleedlatif1 · 2026-09-20T17:18:12Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T17:18:18Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

waleedlatif1 · 2026-09-20T17:22:13Z

@greptile

waleedlatif1 · 2026-09-20T17:22:14Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T17:22:19Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 10 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…cope's ranking - a source filter confines the access plan itself, so every on-row predicate, the reach, and the sources the legs walk or rank are that kind of source alone; `upload` keeps only source-less documents - a date filter enumerates the documents it admits off a new `(knowledge_base_id, source_modified_at)` index and ranks them exactly while the planner estimates the window within the probe's limit; a wider window is walked with the date tested through the document, on the row and on the ranked keyword row - a date-bounded set is ranked exactly even when a member source has its own index, since a walk cannot see the date - the reach is keyed and counted by the plan's sources - a keyword leg whose deadline passes before its ranking is resolved is short rather than failed

…nside it

waleedlatif1 · 2026-09-20T18:14:01Z

@greptile

waleedlatif1 · 2026-09-20T18:14:02Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T18:14:09Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 10 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…s own

waleedlatif1 · 2026-09-20T18:23:13Z

@greptile

waleedlatif1 · 2026-09-20T18:23:15Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-20T18:23:21Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 10 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

vercel Bot deployed to Preview September 20, 2026 17:11 View deployment

greptile-apps Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/queries.ts

cubic-dev-ai Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/queries.ts Outdated

Comment thread apps/sim/lib/knowledge/search/queries.ts

Comment thread packages/db/schema.ts

vercel Bot temporarily deployed to Preview September 20, 2026 17:18 Inactive

vercel Bot deployed to Preview September 20, 2026 17:23 View deployment

cubic-dev-ai Bot reviewed Sep 20, 2026

View reviewed changes

waleedlatif1 mentioned this pull request Sep 20, 2026

improvement(knowledge): page search results, take a custom date window, and rerank a person's search #8060

Merged

6 tasks

waleedlatif1 added 3 commits September 20, 2026 11:13

fix(knowledge): enumerate a small filtered source so both legs rank i…

04f17dc

…nside it

chore(db): mark the document date lookup index concurrent in the schema

1b59f04

waleedlatif1 force-pushed the improvement/search-filters-on-row branch from 318f2eb to 1b59f04 Compare September 20, 2026 18:13

vercel Bot deployed to Preview September 20, 2026 18:15 View deployment

cubic-dev-ai Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/queries.ts Outdated

fix(knowledge): estimate a filter's size under a short deadline of it…

8ee7f85

…s own

vercel Bot temporarily deployed to Preview September 20, 2026 18:23 Inactive

cubic-dev-ai Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/queries.ts

waleedlatif1 merged commit 7093409 into staging Sep 20, 2026
34 checks passed

waleedlatif1 deleted the improvement/search-filters-on-row branch September 20, 2026 18:31

waleedlatif1 mentioned this pull request Sep 21, 2026

v0.8.48: prewarm, tin pg extensions, knowledge search speedups, ui consolidation, keyword search speedups #8096

Merged

Conversation

waleedlatif1 commented Sep 20, 2026

Summary

Measurements

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

waleedlatif1 commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 20, 2026 •

edited

Loading

greptile-apps Bot commented Sep 20, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading