fix: evicted observations leave ghost entries in search indexes#1005
fix: evicted observations leave ghost entries in search indexes#1005fix2015 wants to merge 1 commit into
Conversation
when evict deleted observations from KV, it never removed them from the BM25 or vector indexes. this meant search would return IDs for evicted entries, but the second-pass KV lookup would get null, silently dropping results and under-filling search pages. added index removal calls to match how forget and auto-forget already handle it.
|
@fix2015 is attempting to deploy a commit to the rohitg00's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe eviction function in evict.ts now synchronizes the search and vector indexes when deleting entities. Deletions of low-importance observations, project-capped observations, expired memories, and old non-latest memories additionally remove index entries, and pending index changes are flushed at the end when not in dry-run mode. ChangesEviction index synchronization
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant EvictFunction
participant KVStore
participant SearchIndex
participant VectorIndex
EvictFunction->>KVStore: delete evicted entity
KVStore-->>EvictFunction: deletion confirmed
EvictFunction->>SearchIndex: remove(id)
EvictFunction->>VectorIndex: vectorIndexRemove(id)
Note over EvictFunction: repeats for each eviction path
alt not dryRun
EvictFunction->>SearchIndex: flushIndexSave()
end
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Tools execution failed with the following error: Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Found a bug in the eviction logic — when observations are evicted from KV, they're never removed from the BM25 or vector indexes. This means searches return IDs for evicted entries, but the KV lookup in the second pass gets null, silently dropping results and under-filling search pages.
The forget and auto-forget code paths already handle this correctly (they call both `getSearchIndex().remove()` and `vectorIndexRemove()` alongside the KV delete). Applied the same pattern to eviction.
Summary by CodeRabbit