semantic_query is English-only: non-Latin queries score negative against every node

[Metatau]

Summary

semantic_query is English-only in practice. Queries in other languages do not merely rank worse — they score negative against every node, so the result set is indistinguishable from noise. This is a design question rather than a bug report, so I am opening it for discussion rather than sending a patch.

Measurement

Same project, same index, mode=full, v0.9.0:

query	top hits	score
["validate", "arguments", "schema"]	SpravError.__init__, list_tools, …	+0.018 … +0.003
["проверка", "аргументы", "схема"] (same words, Russian)	builtins.print, builtins.list.pop, builtins.str.lower	−0.041 … −0.062

The Russian run puts Python builtins at the top with negative similarity — the query vector has no meaningful projection onto the token space at all.

Cause

vendored/nomic/ holds static token embeddings distilled from nomic-embed-code (768d, int8, code_vectors_blob.S). The vocabulary in code_tokens.txt is code-and-English; a token that is not in it contributes nothing, and a query made entirely of such tokens contributes nothing at all.

What I am not proposing

Swapping nomic-embed-code for a general multilingual model looks like a bad trade from where I sit, and I do not think an outside PR should make that call:

• the current model is code-specialised, which is the primary use case — a multilingual text model would likely regress English code search, the thing most users actually run;
• scripts/extract_nomic_vectors.py documents 2–3 h on GPU / 6–10 h on CPU per extraction, so this is not something a contributor can validate casually;
• the binary already ships at ~273 MB; vocabulary growth for full multilingual coverage is not free.

Options that seem cheaper

1. Do nothing in the model, fix BM25 instead. Indexing docstrings and accepting non-ASCII query bytes (see the companion issue) restores exact-term search in any language and covers most of what users need. This is what I would do first.
2. Fall back explicitly. When a semantic_query produces no in-vocabulary tokens, say so — return the BM25 result with a note instead of returning near-zero-scored builtins that look like real answers. Silent garbage is worse than an honest "no semantic signal for this query".
3. Optional side-car vocabulary. Keep nomic-embed-code as the default and allow an additional token table to be loaded at runtime for teams that need it, rather than replacing the shipped one.

Happy to prototype (2) if it sounds reasonable — it is contained and does not touch the model.

Environment

• v0.9.0 (release build, UI variant), macOS 15.5 arm64
• mode=full index, 137-node project; reproduced on larger ones as well