Skip to content

search: intersect multi-word hit lists from the shortest, by hash set - #736

Open
basnijholt wants to merge 1 commit into
matrix-construct:mainfrom
mindroom-ai:upstream-search-hash-intersection
Open

basnijholt wants to merge 1 commit into
matrix-construct:mainfrom
mindroom-ai:upstream-search-hash-intersection

Conversation

@basnijholt

@basnijholt basnijholt commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

A room search for several common words takes tens of seconds, while any single word returns in about a second, even one with 200,000 hits.

search_pdu_ids reads each query word's hit list, newest first, and passes them to set::intersection. For every candidate in the first list, that function copies every other list and scans it. For a ten-word sentence of common words in one busy room (per-word hits from 1,267 to 206,178), that is about 3.1 billion copied ids per page.

The hit lists are now intersected starting from the shortest, checking it against a hash set of each other list. That keeps the shortest list's newest-first order, so next_batch paging and the cross-room merge in search.rs are unchanged. A word repeated in the query is read from the index once. search_pdu_ids returns the Vec its only caller collected. Results and count are unchanged.

After this change set::intersection has no production caller left; I left it in place.

Measured on our homeserver: that sentence took 47 s restricted to one room before, 1.7 s after. On synthetic lists of the same sizes, the intersection alone took 34.7 s (release build) before and 72 ms after. Most of the remaining time is reading the posting lists from RocksDB.

Found through MindRoom. Fork change: mindroom-ai/mindroom-tuwunel#177.

Tests:

  • Unit tests in src/service/rooms/search/tests.rs: intersection order, duplicate lists, one list and no lists; distinct query words in first-seen order.
  • src/main/tests/search_multi_word.rs: exact results newest first, count, and paging at limits 1, 2 and 10 for multi-word (including a later, shorter list), repeated-word, one-word, missing-word and wordless searches. It also passes without the fix, since the old code returned the same results, only slowly.

Validation:

  • Nightly rustfmt (2026-11-19): cargo fmt --all -- --check passes.
  • cargo clippy --offline --locked -p tuwunel_service -p tuwunel --all-targets --all-features -- -D warnings passes.
  • The search unit tests, search_multi_word and the existing search_pagination pass.
  • Complement was not run.

Checklist

  • Code is formatted with nightly cargo fmt and satisfies clippy and rustc lints; any allowed lint is justified by an obvious reason or a comment.
  • Complement compliance changes (new passes or new failures), if any, are noted in the description above.
  • Config option changes were made in src/core/config/mod.rs doc comments and the regenerated tuwunel-example.toml is committed. (No config changes.)
  • User-facing changes are reflected in docs/. (No user-facing docs change.)
  • I agree that my changes may be licensed under the Apache-2.0 licence and my conduct is in line with the Contributor's Covenant and Tuwunel's Code of Conduct.

search_pdu_ids read each query word's hit list, newest first, and called
set::intersection, which for every candidate of the first list copied
every other list and scanned it. A sentence of common words, with hit
lists of up to ~200,000 events, made a room search take tens of seconds
while any single word took under a second.

intersect_hits checks the shortest list against a hash set of each other
list and keeps its newest-first order, so next_batch paging and the
cross-room merge are unchanged. A word repeated in the query is read from
the index once. search_pdu_ids now returns the Vec its only caller
collected. Results and count are unchanged.

Assisted-by: MindRoom DevAgent:anthropic/claude-opus-5.5 <https://github.com/mindroom-ai/mindroom>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant