SOLR-18348: Move building suggester to async process outside of commit lifecycle. - #4742
Draft
epugh wants to merge 4 commits into
Draft
SOLR-18348: Move building suggester to async process outside of commit lifecycle.#4742epugh wants to merge 4 commits into
epugh wants to merge 4 commits into
Conversation
A suggester configured with buildOnCommit=true rebuilds synchronously inside the newSearcherListener callback, which SolrCore's single-threaded searcherExecutor runs. DirectUpdateHandler2.commit() blocks the committing thread on that executor's future with no timeout, so a slow (e.g. I/O-bound) rebuild blocks the commit - and the client waiting on it - indefinitely instead of just slowing down. In production this surfaced as client-side request timeouts requiring manual intervention, not just degraded latency. Add an opt-in buildOnCommitAsync suggester parameter (default false, so existing buildOnCommit behavior and its "suggestions are fresh immediately after commit" guarantee are unchanged unless a suggester asks for the new behavior). When enabled, the rebuild runs on a dedicated executor instead of inline, with a coalescing guard so a burst of commits can't queue up an unbounded backlog of stale builds, and reader ref-counting so the searcher being built from can't be closed out from under the async build. Also: - Track the index version each suggester was last built from and expose it (plus the current index version) in the suggest response, so a caller can tell whether a response reflects the current index or a stale one still catching up. - Fix SolrSuggester.getSuggestions() to catch IllegalStateException from Lookup implementations (e.g. AnalyzingInfixSuggester) that throw rather than return no results when queried before any build has ever completed - newly reachable now that a query can land in that window with buildOnCommitAsync=true. - Expose buildOnCommit status (in progress / last duration) via metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ted suggest finishStage() only ever propagated the "suggest" subsection from each shard's response, silently dropping the per-suggester builtFromIndexVersion/currentIndexVersion info added for the single-core case. In a real SolrCloud deployment that meant a distributed suggest query gave no signal at all about whether any shard answered from a suggester still catching up (e.g. buildOnCommitAsync's rebuild still running). Raw index versions aren't comparable across separate cores, so rather than merge numbers, add a single "stale" boolean per suggester (based on builtFromIndexVersion < currentIndexVersion) at process() time, and have finishStage() OR that flag across every shard into one top-level suggesterStale field: true if any shard answered from a stale suggester. Verified with a lightweight test that drives finishStage() directly against fabricated shard responses (mirroring the existing MockShardRequest pattern) rather than standing up a full multi-shard cluster. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
Two scenarios, since the response shape differs — full detail on a single replica, just the yes/no flag on a distributed query. Direct query to one replica ( {
"responseHeader": { "status": 0, "QTime": 3 },
"suggest": {
"slowSuggester": {
"elec": {
"numFound": 2,
"suggestions": [
{ "term": "electronics", "weight": 100, "payload": "" },
{ "term": "electric fan", "weight": 80, "payload": "" }
]
}
}
},
"suggesterIndexVersions": {
"slowSuggester": {
"builtFromIndexVersion": 41,
"currentIndexVersion": 43,
"stale": true
}
}
}Normal distributed SolrCloud query (fans out to all shards, gets merged) — just the aggregate flag, no per-shard numbers: {
"responseHeader": { "status": 0, "QTime": 12 },
"suggest": {
"slowSuggester": {
"elec": {
"numFound": 2,
"suggestions": [
{ "term": "electronics", "weight": 100, "payload": "" },
{ "term": "electric fan", "weight": 80, "payload": "" }
]
}
}
},
"suggesterStale": true
}
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/SOLR-18348
Description
See the original JIRA for the problem. Basically, commits take longer when you have the suggester get built.
Solution
Messing around so far...
Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.