Repository navigation
search: intersect multi-word hit lists from the shortest, by hash set - #736
Open
basnijholt wants to merge 1 commit into
Open
basnijholt wants to merge 1 commit into
basnijholt wants to merge 1 commit into
Conversation
search_pdu_ids read each query word's hit list, newest first, and called set::intersection, which for every candidate of the first list copied every other list and scanned it. A sentence of common words, with hit lists of up to ~200,000 events, made a room search take tens of seconds while any single word took under a second. intersect_hits checks the shortest list against a hash set of each other list and keeps its newest-first order, so next_batch paging and the cross-room merge are unchanged. A word repeated in the query is read from the index once. search_pdu_ids now returns the Vec its only caller collected. Results and count are unchanged. Assisted-by: MindRoom DevAgent:anthropic/claude-opus-5.5 <https://github.com/mindroom-ai/mindroom>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A room search for several common words takes tens of seconds, while any single word returns in about a second, even one with 200,000 hits.
search_pdu_idsreads each query word's hit list, newest first, and passes them toset::intersection. For every candidate in the first list, that function copies every other list and scans it. For a ten-word sentence of common words in one busy room (per-word hits from 1,267 to 206,178), that is about 3.1 billion copied ids per page.The hit lists are now intersected starting from the shortest, checking it against a hash set of each other list. That keeps the shortest list's newest-first order, so
next_batchpaging and the cross-room merge insearch.rsare unchanged. A word repeated in the query is read from the index once.search_pdu_idsreturns theVecits only caller collected. Results andcountare unchanged.After this change
set::intersectionhas no production caller left; I left it in place.Measured on our homeserver: that sentence took 47 s restricted to one room before, 1.7 s after. On synthetic lists of the same sizes, the intersection alone took 34.7 s (release build) before and 72 ms after. Most of the remaining time is reading the posting lists from RocksDB.
Found through MindRoom. Fork change: mindroom-ai/mindroom-tuwunel#177.
Tests:
src/service/rooms/search/tests.rs: intersection order, duplicate lists, one list and no lists; distinct query words in first-seen order.src/main/tests/search_multi_word.rs: exact results newest first,count, and paging at limits 1, 2 and 10 for multi-word (including a later, shorter list), repeated-word, one-word, missing-word and wordless searches. It also passes without the fix, since the old code returned the same results, only slowly.Validation:
cargo fmt --all -- --checkpasses.cargo clippy --offline --locked -p tuwunel_service -p tuwunel --all-targets --all-features -- -D warningspasses.search_multi_wordand the existingsearch_paginationpass.Checklist
cargo fmtand satisfies clippy and rustc lints; any allowed lint is justified by an obvious reason or a comment.src/core/config/mod.rsdoc comments and the regeneratedtuwunel-example.tomlis committed. (No config changes.)docs/. (No user-facing docs change.)