Skip to content

chore: delete src/vectorsets/, a PyO3 module that has not compiled since Feb 2026 - #297

Merged
fcostaoliveira merged 3 commits into
masterfrom
chore/remove-dead-vectorsets-module
Aug 9, 2026
Merged

chore: delete src/vectorsets/, a PyO3 module that has not compiled since Feb 2026#297
fcostaoliveira merged 3 commits into
masterfrom
chore/remove-dead-vectorsets-module

Conversation

@fcostaoliveira

@fcostaoliveira fcostaoliveira commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Deletes src/vectorsets/{mod,configure,search,upload}.rs — 1,421 lines that no target in this crate has compiled since 2026-02-27 — plus the 5-line block in memory-bank/standards/coding-standards.md that enumerated exactly those four files. 5 files, 1,426 deleted lines, zero added lines. No source file outside the deleted directory is touched. The whole review surface is "is it really dead", so the evidence is below, with the commands and their verbatim output. Everything was run in dedicated worktrees with their own CARGO_TARGET_DIR.

Base moved: #295 merged, so origin/master is now 407a196. 7e6ba51 merges master in (merge, not rebase). Every gate and guard number below has been re-derived against 407a196 and the merged tree; the static evidence in §§1-5 is quoted at 7ed81eb, where the deleted files still exist and can be inspected. Merge correctness is verified in Merge with 407a196 below rather than inferred from the absence of conflicts.

Closes #291.


1. Nothing declares it

src/lib.rs declares eight modules, and vectorsets is not one of them:

$ grep -n "^pub mod\|^mod " src/lib.rs
10:pub mod config;
11:pub mod metrics;
12:pub mod parsers;
13:pub mod query_filter;
14:pub mod readers;
15:pub mod redis_client;
16:pub mod start_gate;
17:pub mod synthetic;

There is exactly one mod vectorsets; in the tree:

$ grep -rn "mod vectorsets" . --include=*.rs
src/bin/vector_db_benchmark/engine/mod.rs:26:mod vectorsets;

Rust resolves mod foo; against the declaring module's own directory. That declaration is in engine/mod.rs, a mod.rs, so its directory is src/bin/vector_db_benchmark/engine/ — the candidates are engine/vectorsets.rs and engine/vectorsets/mod.rs. engine/vectorsets.rs exists and is the live engine (pub use vectorsets::VectorSetsEngine; at engine/mod.rs:48). src/vectorsets/ is in a different directory entirely and is not a candidate for that declaration.

src/vectorsets/mod.rs does exist, so the directory would be picked up by a mod vectorsets; written in src/lib.rs — which is why item 3 below matters. Nothing else could reach it: there is no #[path] attribute anywhere, and no include!/include_str!/include_bytes! outside v0/:

$ grep -rn '#\[path' --include=*.rs .
(none)
$ grep -rn 'include!\|include_str!\|include_bytes!' --include=*.rs . | grep -v '^./v0/'
(none)

2. Nothing references its symbols

The module defines exactly three public items across the three non-mod.rs files:

$ grep -n '^\s*pub \(fn\|struct\|enum\|trait\|const\|static\|type\|mod\|use\)' src/vectorsets/*.rs
src/vectorsets/mod.rs:1:pub mod configure;
src/vectorsets/mod.rs:2:pub mod search;
src/vectorsets/mod.rs:3:pub mod upload;
src/vectorsets/configure.rs:7:pub struct RustVsetConfigurator {
src/vectorsets/search.rs:30:pub struct RustVsetSearcher {
src/vectorsets/upload.rs:16:pub struct RustVsetUploader {

Searched repo-wide across all file types, including tests/, v0/, .github/ and the manifests:

$ grep -rn "RustVsetConfigurator\|RustVsetSearcher\|RustVsetUploader" . --exclude-dir=.git
src/vectorsets/search.rs:30:pub struct RustVsetSearcher {
src/vectorsets/search.rs:39:impl RustVsetSearcher {
src/vectorsets/upload.rs:16:pub struct RustVsetUploader {
src/vectorsets/upload.rs:39:impl RustVsetUploader {
src/vectorsets/upload.rs:707:impl RustVsetUploader {
src/vectorsets/configure.rs:7:pub struct RustVsetConfigurator {
src/vectorsets/configure.rs:16:impl RustVsetConfigurator {

Definitions and their own impl blocks only. Zero use sites.

3. It genuinely does not compile

Demonstrated rather than asserted. I appended pub mod vectorsets; to src/lib.rs (verified with git diff that the edit landed), ran cargo check --lib, then restored src/lib.rs from a copy taken beforehand — git status --porcelain was empty afterwards.

$ cargo check --lib 2>&1 | head -8
    Checking vector_db_benchmark v0.1.0 (...)
error[E0433]: cannot find module or crate `pyo3` in this scope
 --> src/vectorsets/configure.rs:1:5
  |
1 | use pyo3::prelude::*;
  |     ^^^^ use of unresolved module or unlinked crate `pyo3`
  |
  = help: if you wanted to use a crate named `pyo3`, use `cargo add pyo3` to add it to your `Cargo.toml`

$ cargo check --lib 2>&1 | tail -1
error: could not compile `vector_db_benchmark` (lib) due to 190 previous errors

$ cargo check --lib 2>&1 | grep -oE '^error\[[A-Z0-9]+\]' | sort | uniq -c | sort -rn
    112 error[E0425]
     53 error[E0433]
      1 error[E0432]

190 errors. And this is not a missing-feature-flag situation — pyo3 is absent from both the manifest and the lockfile:

$ grep -n "pyo3" Cargo.toml Cargo.lock
(no matches)

So there is no configuration of this crate under which these files compile.

4. Why it survived: it is an orphaned PyO3 extension

$ git log --follow --oneline -- src/vectorsets/search.rs
1282eda fix(metrics): our mean_precisions held a different quantity than upstream's identically-named key (#249)
530ec70 feat: add recall@K, precision@K, MRR, and NDCG@K retrieval quality metrics
ebd4a5c First version with Redisearch and VectorSets working in rust
2376af4 Search QPS was being wrongfully calculated on search-rs and vectorset-rs
0725e3c Add Rust (PyO3) native extensions for VectorSets and RediSearch engines
  • 0725e3c (2026-02-23) added these files at rust/src/vectorsets/, as a PyO3 extension consumed from Python. That crate root did declare them, so they compiled at the time:

    $ git show 0725e3c:rust/src/lib.rs
    mod config;
    mod redis_client;
    mod redisearch;
    mod vectorsets;
    
    use pyo3::prelude::*;
    ...
    #[pymodule]
    fn vector_db_benchmark_rs(m: &Bound<'_, PyModule>) -> PyResult<()> {
        m.add_class::<RustVsetConfigurator>()?;
        ...
    
  • ebd4a5c (2026-02-27), "First version with Redisearch and VectorSets working in rust", is the supersession. In one commit it moved rust/src/vectorsets/{mod,configure,search}.rs to src/vectorsets/ and added the real engine src/bin/vector_db_benchmark/engine/vectorsets.rs (532 lines):

    $ git show ebd4a5c --stat -M | grep -i vectorsets
     engine/clients/vectorsets_rs/__init__.py           |    84 -
     rust/src/vectorsets/upload.rs                      |   504 -
     src/bin/vector_db_benchmark/engine/vectorsets.rs   |   532 +
     {rust/src => src}/vectorsets/configure.rs          |    23 +-
     {rust/src => src}/vectorsets/mod.rs                |     0
     {rust/src => src}/vectorsets/search.rs             |    43 +-
     src/vectorsets/upload.rs                           |   916 ++
     ...
    

    The new src/lib.rs written by that commit declared only config, readers, redis_client — the mod vectorsets; was not carried over. That is the moment it went dead. Its Python caller (engine/clients/vectorsets_rs/__init__.py) was deleted in the same commit.

Was anything left unported? No. The dead module is a PyO3 shim: three #[pyclass] structs whose methods take/return PyObject and enter Python::with_gil. Its VectorSets protocol work — VADD/VSIM, FP32/Q8/BIN quantization, pipelined batch upload, VSIM ... WITHSCORES parsing — is all present in engine/vectorsets.rs, which has since grown well past it (per-config keys, filter expressions, geo, the start gate). The one thing the dead copy has that the live one does not is a stale behaviour, not a missing one — see below. I did not find anything worth porting, so no follow-up issue is filed for lost functionality.

5. Nothing in packaging or CI globs it

  • Cargo.toml has no include/exclude keys, and its four [[bin]] targets are src/bin/bench_jsonl.rs, src/bin/bench_npy.rs, src/bin/vector_db_benchmark/main.rs, src/bin/generate_dataset.rs. There is no [[bench]]. Cargo's target auto-discovery looks at src/bin/, benches/, examples/, tests/src/vectorsets/ is none of those. (Auto-discovery does add a fifth, undeclared bin, src/bin/bench_hdf5.rs, which has no [[bin]] stanza. That is why the cargo test --lib --bins sum in the Gates table has six terms rather than five — lib + 5 bins. It is not a typo.)

  • No build.rs exists.

  • The nested fuzz/ crate cannot reach it either. fuzz/Cargo.toml declares its own [workspace] and depends on the root crate by path ([dependencies.vector_db_benchmark] path = ".."), so it can only see what src/lib.rs exports — which never included vectorsets. And it does not ask for it:

    $ for f in $(git ls-tree -r --name-only 7ed81eb -- fuzz/); do git show 7ed81eb:$f | grep -in vectorset | sed "s|^|$f:|"; done
    (no output)
    

    Zero occurrences of the substring vectorset anywhere under fuzz/ — targets, corpora, dictionaries, README, lockfile.

  • Workflows: the only src references in .github/ are path filters (- 'src/**' in ci.yml, docker-build-pr.yml, validate-datasets.yml). Those decide whether a workflow runs; they do not compile or package individual files, and this PR touches src/** so CI triggers normally.

  • Dockerfile is the one literal hit for the string src/vectorsets in the whole repo:

    $ grep -rn "src/vectorsets" . --exclude-dir=.git
    Dockerfile:26:RUN mkdir -p src/bin/vector_db_benchmark/engine src/readers src/redisearch src/vectorsets \
    

    That grep is literal, and one surviving reference escapes it by writing vectorsets/ without the src/ prefix: memory-bank/standards/coding-standards.md:58-62, whose source-tree diagram enumerated exactly the four deleted files. Unlike the Dockerfile line, that one is documentation that would describe a tree which no longer exists, so this PR deletes it (commit 52ce283) rather than deferring it — see Scope. The Dockerfile line is different, and is not a reference to the repo tree at all. At that point in the build only Cargo.toml and Cargo.lock have been copied (line 23), so no src/ exists in the image at all; line 26 fabricates an empty stub tree so the dependency-only build can be cached. Line 39 then runs rm -rf src, and line 43 COPY src ./src brings in the real tree. mkdir -p on a directory that no longer exists upstream is a no-op either way. I deliberately left the Dockerfile untouched — stated as a known-stale reference in Scope.

6. The repo's own source-scanning guards

tests/overhead_invariants.rs::all_sources() (line 214) walks every .rs under src/, so deleting files does change its input. Two tests consume it: inv4_no_fixed_count_start_barrier (line 460) and stripper_preserves_line_count_of_every_source_file (line 681).

Measured inputs before → after:

input before after
.rs files under src/ (the test's checked) 62 58
\-continuations under src/ (the test's continuations) 307 307
lines matching Barrier under src/vectorsets/ 0

Both columns are measured on the merged tree (407a1967e6ba51). #295 added no files under src/, so the file count is unchanged from the old base; it did add \-continuations, which is why that row reads 307 rather than the 281 this body quoted before the merge. stripper_preserves_line_count_of_every_source_file asserts checked > 20 and continuations > 100; 58 and 307 clear both with room to spare. inv4 had nothing to find in the deleted files (grep -rn "Barrier" src/vectorsets/ → no matches), so its verdict is unchanged.

On the inventories: the only one that exists at 7ed81eb is EXCUSED in gated_engine_files() (line 249), and it is keyed on module names parsed out of engine/mod.rs, not on paths under src/ — the deleted directory never appeared in it.

$ grep -rn "EXCUSED\|KNOWN_UNRECORDED\|EXEMPT_CALLS\|NON_LITERAL_ENV_VAR_SITES" --include=*.rs . | grep -v '^./v0/'
tests/overhead_invariants.rs:249:    const EXCUSED: &[(&str, &str)] = &[
tests/overhead_invariants.rs:277:        if EXCUSED.iter().any(|(e, _)| *e == name) {

KNOWN_UNRECORDED, EXEMPT_CALLS and NON_LITERAL_ENV_VAR_SITES do not exist on master — nor does src/bin/vector_db_benchmark/effective_config.rs. They arrive with open PR #264.

⚠️ Merge-order coordination with #264 — the one thing about this PR that is not delta-zero.
The grep below is scoped to master, so it structurally cannot see #264's inventories. #264 has a fourth one that my enumeration above omits:

// effective_config.rs:1122 @ 93bb97b
const UNCOMPILED: &[&str] = &["src/redisearch/", "src/vectorsets/"];

and 93bb97b added uncompiled_directories_are_present_and_genuinely_uncompiled (line 1615), which asserts root.join(dir).is_dir() for every row. The order is now settled — #264 is not merging in this cycle (a fuzz campaign found leak classes at its head), so #297 lands first and #264 owes the deletion of its "src/vectorsets/" row before it can go green. Without it:

UNCOMPILED lists `src/vectorsets/`, which no longer exists — delete the row
(and any EXEMPT_CALLS reason that cites it) … Tracked in #275 / #296 / #297.

It is a one-line deletion, and the assertion is doing exactly its job — it is why that row cannot rot silently. I am not making the edit here: effective_config.rs exists on neither 407a196 nor this branch, so there is nothing to edit. Recording it in the body rather than only in a PR comment, so that "Delta: 0 on every gate" is not read as "nothing to coordinate".

Restricted to master, where the inventories listed above are the only ones that exist, no path under src/vectorsets/ is named in any of them:

$ grep -rn "src/vectorsets\|vectorsets/search\|vectorsets/upload\|vectorsets/configure" --include=*.rs . | grep -v '^./src/vectorsets/'
(no hits)

Why delete rather than leave

It reads as the implementation while disagreeing with the shipping engine on its most load-bearing detail. It still writes the pre-#236 hardcoded key:

$ grep -n '\.arg("idx")' src/vectorsets/*.rs
src/vectorsets/search.rs:100:            .arg("idx")
src/vectorsets/search.rs:348:        .arg("idx")
src/vectorsets/upload.rs:109:                .arg("idx")
src/vectorsets/upload.rs:899:            .arg("idx")

whereas engine/vectorsets.rs:141 derives a per-config key:

key: derive_index_name("VECTORSETS_INDEX_NAME", "idx", &engine_config.name),

One correction to the issue's framing. #291 says two open PRs are editing src/vectorsets/search.rs. As of today that is not what the diffs show: #286 merged as 7ed81eb and its file list contains engine/vectorsets.rs, not src/vectorsets/search.rs; #264's current file list is 22 files, none under src/vectorsets/. The underlying cost is nonetheless real and is visible in history rather than in open PRs — two repo-wide refactors did edit this never-compiled file: 530ec70 (recall/precision/MRR/NDCG) and 1282eda (#249, 2026-08-08 — 162 days, i.e. 5 months and 12 days, after it went dead).

Related issues — and the overlap with #275

#275 ("dead code: uncompiled src/redisearch + src/vectorsets, recorder-bypassing RedisConfig::from_env, and WEAVIATE_USE_GRAPHQL=false selects GraphQL") already tracked both directories before #291 was filed, and #264's EXEMPT_CALLS prose cites #275 by number as the tracker. Flagging the overlap rather than letting it sit silent:

Gates

Run in a worktree at 7ed81eb with a private CARGO_TARGET_DIR, touch src/lib.rs src/bin/vector_db_benchmark/main.rs before each test invocation to defeat stale fingerprints, and the running N tests line asserted for every run. A pure deletion of never-compiled files should move nothing, and nothing moved.

gate master 407a196 merged head 7e6ba51 delta
cargo fmt --check exit 0 exit 0
cargo clippy --all-targets -- -D warnings exit 0 exit 0
cargo test --lib --bins 919 = 129 + 0 + 0 + 0 + 2 + 788 919 = 129 + 0 + 0 + 0 + 2 + 788 0
cargo test --test overhead_invariants 9 passed 9 passed 0
cargo test --test integration_redis -- --test-threads=1 41 passed 41 passed 0

Verbatim, on the merged head 7e6ba51:

running 129 tests
test result: ok. 129 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.83s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 2 tests
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s
running 788 tests
test result: ok. 788 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.36s

running 9 tests
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.38s

running 41 tests
test result: ok. 41 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 16.15s

and on master 407a196, same worktree, same CARGO_TARGET_DIR, git switch --detach between them:

running 129 tests
test result: ok. 129 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.02s
…
running 788 tests
test result: ok. 788 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.42s

running 41 tests
test result: ok. 41 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 16.22s

Delta: 0 on every gate. Both columns are my own runs, not quoted baselines — including integration_redis, which on the old base I had only run after the deletion. It used a private redis:8.8.0 container (vdb291c-redis, host port 6492, REDIS_TEST_PORT=6492, flushall between the two runs, removed afterwards) — never the shared tests-redis-1 on 6399. touch src/lib.rs src/bin/vector_db_benchmark/main.rs before every invocation, running N asserted on all of them, find src -name '*.rs' | wc -l = 62 on master and 58 on the merged head.

One number I was handed that did not survive checking, flagged because the total hides it. The 919 master baseline is right, but the split I was given — "126 lib + 2 generate_dataset + 791 main" — is not what the run prints. It is 129 lib + 2 + 788 main: #295 added 3 library tests (readers/sparse_reader.rs, synthetic.rs) and 11 to the main binary, not 0 and 14. Same total, different provenance, so a future delta traced to the wrong target would come out wrong.

integration_redis note, stated plainly rather than glossed: I ran it only after the deletion, against a private redis:8.8.0 container (vdb291-redis, host port 6491, REDIS_TEST_PORT=6491, torn down afterwards) — not the shared tests-redis-1 on 6399. The 39 is compared against the count recorded for master rather than a before-run I performed myself. Since the deleted files are not compiled into any target, an integration run cannot link them, so a before/after pair could not have differed.

Merge with 407a196

7e6ba51 is a merge commit with parents 52ce283 + 407a196. Git reported no conflict — which on its own proves nothing, so the merge is checked from both sides instead.

The merged tree differs from master by exactly this PR, and by nothing else:

$ git diff --stat 407a196 HEAD
 memory-bank/standards/coding-standards.md |   5 -
 src/vectorsets/configure.rs               |  59 --
 src/vectorsets/mod.rs                     |   3 -
 src/vectorsets/search.rs                  | 443 ---------------
 src/vectorsets/upload.rs                  | 916 ------------------------------
 5 files changed, 1426 deletions(-)

$ git diff 407a196 HEAD | grep -E '^\+' | grep -v '^\+\+\+'
(no output — zero added lines)

And it differs from my pre-merge tip by exactly #295, byte for byte — nothing of theirs was dropped in the merge:

$ diff <(git diff --name-only 52ce283 HEAD) <(git diff --name-only 7ed81eb 407a196)
IDENTICAL file lists            # 13 files

$ # per-file: diff-of-diffs between (52ce283..merge) and (7ed81eb..master)
(no DIFFERS lines — every hunk landed identical)

src/vectorsets/ is gone from the merged tree (ls -d src/vectorsetsNo such file or directory) and engine/vectorsets.rs is present and untouched at 99,629 bytes. A grep for conflict markers across src/, tests/ and memory-bank/ returns nothing. #295 touched 13 files — README.md, datasets/datasets.json, generate_dataset.rs, cli.rs, config.rs, dataset.rs, engine/{qdrant,redis}.rs, experiment.rs, readers/{mod,sparse_reader}.rs, synthetic.rs, tests/integration_redis.rs — and added or deleted no files; the intersection with my 5 is empty.

Scope — what I deliberately did not do

  • Did not touch src/bin/vector_db_benchmark/engine/vectorsets.rs, the live engine, at all.
  • Knowingly left a stale reference in the Dockerfile. After this PR, Dockerfile:26 names src/vectorsets, a directory that no longer exists in the repo (and src/redisearch will join it once src/redisearch/ is the second dead PyO3 module — same state as src/vectorsets/ (#291) #296 lands). I am stating that plainly rather than only saying "did not touch it". It is cosmetic, not functional: as shown in §5 the line fabricates a stub tree inside an image that has no src/ at that point, and rm -rf src 13 lines later deletes it before the real COPY src ./src. mkdir -p neither reads nor requires the repo tree, so the build behaves identically with or without the name. Editing it would be a behaviourless change to a caching layer and is out of scope here; the one-line tidy belongs in the src/redisearch/ is the second dead PyO3 module — same state as src/vectorsets/ (#291) #296 PR, where it can drop both names at once.
  • Did not delete src/redisearch/, which is in the same state — 5 files, no mod redisearch; anywhere in the tree, the same orphaned PyO3 origin in 0725e3c, pyo3 on 57 lines across its four non-mod.rs files, and zero references to RustRedisConfigurator/RustRedisSearcher/RustRedisUploader outside itself. The live RediSearch engine is engine/redis.rs (mod redis;, engine/mod.rs:22). Filed as src/redisearch/ is the second dead PyO3 module — same state as src/vectorsets/ (#291) #296 so this PR stays one reviewable deletion. Those are the only two such directories: the other entries under src/ are bin/ and readers/, and readers is declared (src/lib.rs:14).
  • Did delete one documentation reference, and only that one. memory-bank/standards/coding-standards.md:58-62 listed the four deleted files by name in its source-tree diagram; commit 52ce283 removes those 5 lines. A module's own tree-map entry is part of the module rather than adjacent tidying — but the limits are worth stating. src/redisearch/'s block (lines 52-57) stays: that directory still exists on disk, so the diagram is still accurate about it, and it goes with src/redisearch/ is the second dead PyO3 module — same state as src/vectorsets/ (#291) #296. And the diagram remains stale in ways that predate this PR and are not fixed here — src/ also holds metrics.rs, parsers.rs, query_filter.rs, start_gate.rs, synthetic.rs; readers/ also holds sparse_reader.rs; bin/ also holds generate_dataset.rs; and under engine/ it names 3 entries where the directory holds 22 .rs files. This PR does not claim to make that diagram correct, only to stop it naming files this PR removes. Nothing reads memory-bank/ (git grep -n "memory-bank" 7ed81eb -- src/ tests/ .github/ → no output), so no gate is affected.
  • Did not rename or tidy anything else. Every line in the diff is a deletion; there are zero added lines.
  • Did not verify the deleted code's behaviour in any way. It has never run in this crate; there is nothing to regress.

🤖 Generated with Claude Code

…nce Feb 2026

`src/vectorsets/{mod,configure,search,upload}.rs` is unreachable from every
target in this crate. `src/lib.rs` declares only `config`, `metrics`,
`parsers`, `query_filter`, `readers`, `redis_client`, `start_gate`,
`synthetic`. The single `mod vectorsets;` in the tree
(`src/bin/vector_db_benchmark/engine/mod.rs:26`) sits in `engine/mod.rs`, so
Rust resolves it against `src/bin/vector_db_benchmark/engine/` and it binds to
the live `engine/vectorsets.rs`. There is no `#[path]` attribute anywhere in
the repo, and no `include!`/`include_str!` outside `v0/`, so nothing can
re-route a declaration at it.

It also cannot compile. Temporarily appending `pub mod vectorsets;` to
`src/lib.rs` and running `cargo check --lib` produces 190 errors, the first
being `error[E0433]: cannot find module or crate 'pyo3'` at
`src/vectorsets/configure.rs:1`. `pyo3` is not in `Cargo.toml` and not in
`Cargo.lock`. These files were the PyO3 native extension added in 0725e3c
(2026-02-23) under `rust/src/`, whose crate root `rust/src/lib.rs` did declare
`mod vectorsets;`. ebd4a5c (2026-02-27) moved `rust/src/vectorsets/*` to
`src/vectorsets/*` and in the same commit added the real
`src/bin/vector_db_benchmark/engine/vectorsets.rs` (532 lines); the new
`src/lib.rs` did not carry the declaration over. Nothing was lost by that move
that is not already implemented in the live engine.

Nothing references its three public types (`RustVsetConfigurator`,
`RustVsetSearcher`, `RustVsetUploader`) anywhere in the repo, including
`tests/`, `v0/` and the workflows.

The cost of leaving it is that it reads like the implementation while
disagreeing with it: it still writes the pre-#236 hardcoded key
(`.arg("idx")` at search.rs:100,348 and upload.rs:109,899), whereas the live
engine derives `idx:<config>` via `derive_index_name` (vectorsets.rs:141).
Two repo-wide refactors have already spent edits on it (530ec70 and 1282eda /
PR #249).

Deletion only. No other file is touched.

Closes #291.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fcostaoliveira added a commit that referenced this pull request Aug 9, 2026
`UNCOMPILED` was the one inventory in this module with a skip filter and no
assertion. `KNOWN_UNRECORDED` and `EXEMPT_CALLS` are both bidirectional — the
latter was upgraded precisely "so a removed one cannot leave a stale excuse" —
and the same reasoning applies here with more force: a stale skip row silently
WIDENS what the scanner ignores, which is the direction that hides violations
rather than surfacing them. `src/vectorsets/` is being deleted (#297) and
`src/redisearch/` next (#296), so the row would have rotted unnoticed.

Two legs, each proven to bite:

* the directory must still EXIST. Simulating #297 by moving `src/vectorsets/`
  away fails with "UNCOMPILED lists `src/vectorsets/`, which no longer exists —
  delete the row … Tracked in #275 / #296 / #297".
* no crate root may DECLARE it, or the code is compiled, can read the
  environment at runtime, and is being skipped anyway. Adding a row for
  `src/readers/` (which lib.rs does declare) fails with "IS declared as a module
  and therefore compiled".

Leg 2 could not be demonstrated by declaring `vectorsets` itself: adding
`pub mod vectorsets;` to lib.rs does not COMPILE, which is independent evidence
for #297's premise that the directory is dead.

The four `src/config.rs` exemption reasons justified themselves by naming
`src/redisearch/` and `src/vectorsets/` as `RedisConfig::from_env`'s only
callers — half-true once #297 lands, false once #296 does. They now state the
durable property instead ("no caller compiled into any binary") and
`the_exempted_dead_constructor_has_no_live_caller` asserts it; planting a real
call in `engine/redis.rs` fails it. After both deletions the constructor has
zero callers and the right move is deleting it, which the reason now says.

Citations cross-linked: #275 covers both directories and the constructor, and is
superseded piecemeal by #291/#296/#297.

Refs #212
@fcostaoliveira

Copy link
Copy Markdown
Collaborator Author

Heads-up from #264 (provenance recording, #212), which is in flight: after this merges, #264's guard will fail until one line is deleted. Flagging now so it is not discovered mid-merge.

effective_config.rs keeps a skip-list of directories the source-scanning guards ignore because nothing compiles them:

const UNCOMPILED: &[&str] = &["src/redisearch/", "src/vectorsets/"];

That list is now asserted in both directions (uncompiled_directories_are_present_and_genuinely_uncompiled), so deleting src/vectorsets/ fails with:

UNCOMPILED lists `src/vectorsets/`, which no longer exists — delete the row
(and any EXEMPT_CALLS reason that cites it) so the skip filter cannot silently
widen what the scanner ignores. Tracked in #275 / #296 / #297.

The fix is deleting "src/vectorsets/" from that array, nothing more. Whichever of us merges second does it.

This is deliberate, and the same contract config::KNOWN_UNREAD already has here: a stale skip row is worse than a stale exemption, because it widens what the scanner ignores rather than narrowing it.

Incidentally, #264's verification supports this PR's premise from another angle: adding pub mod vectorsets; to lib.rs does not compile, so the directory is not merely unreferenced, it is unbuildable. Independent of the docker build check.

One bookkeeping note: #275 already covers both src/redisearch/ and src/vectorsets/ and the recorder-bypassing RedisConfig::from_env, so this PR, #291 and #296 supersede it piecemeal. Worth closing #275 against them, or narrowing it to just the constructor, so #264's inline citations do not end up pointing at a half-dead issue.

…s tree map

`memory-bank/standards/coding-standards.md` lines 58-62 enumerated exactly the
four files the previous commit deleted, so after that commit the diagram
described a directory that no longer exists. Removed the block; nothing else in
the file is touched.

Scoped deliberately:

- `src/redisearch/` keeps its block (lines 52-57). That directory still exists
  on disk, so the diagram is still accurate about it. It goes with #296.
- The diagram remains stale in ways that predate this PR, are unaffected by it,
  and are NOT fixed here: `src/` also holds `metrics.rs`, `parsers.rs`,
  `query_filter.rs`, `start_gate.rs` and `synthetic.rs`; `readers/` also holds
  `sparse_reader.rs`; `bin/` also holds `generate_dataset.rs`; and under
  `engine/` the diagram names 3 entries (`mod.rs`, `redis.rs`, `vectorsets.rs`)
  where the directory holds 22 `.rs` files — `engine/mod.rs` declares 21
  modules plus its own `mod.rs`. This commit does not claim to make the diagram
  correct, only to stop it naming files this PR removes.

No test or workflow reads `memory-bank/`, so this changes no gate:

    $ git grep -n "memory-bank" 7ed81eb -- src/ tests/ .github/
    (no output)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

Docker Build Validation

Platforms tested:

  • linux/amd64 (built and tested)
  • linux/arm64 (build validated)

Tests performed:

  • --help command
  • --describe datasets command
  • Integration test (random-100 dataset + Redis)
  • Multi-platform build validation (amd64 + arm64)

@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

Docker Build Validation

Platforms tested:

  • linux/amd64 (built and tested)
  • linux/arm64 (build validated)

Tests performed:

  • --help command
  • --describe datasets command
  • Integration test (random-100 dataset + Redis)
  • Multi-platform build validation (amd64 + arm64)

1 similar comment
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

Docker Build Validation

Platforms tested:

  • linux/amd64 (built and tested)
  • linux/arm64 (build validated)

Tests performed:

  • --help command
  • --describe datasets command
  • Integration test (random-100 dataset + Redis)
  • Multi-platform build validation (amd64 + arm64)

@fcostaoliveira
fcostaoliveira merged commit caefe0f into master Aug 9, 2026
18 checks passed
fcostaoliveira added a commit that referenced this pull request Aug 9, 2026
Merged, not rebased: the branch is published and the standing rule here is to
add a commit rather than force-push.
fcostaoliveira added a commit that referenced this pull request Aug 9, 2026
…ps printing the leak

#297 landed, so `UNCOMPILED` lost its `src/vectorsets/` row — the bidirectional
assertion added when #297 was opened failed on the merge and named the string to
delete. Contract working as designed.

SF1 — a third identity path published operator input verbatim. `prose` was
derived by matching the JSON key name against `PROSE_KEYS = ["reason","covers"]`
at ANY depth, and `set_declared` forwards a raw un-allow-listed config subtree
into `note_override`. A config placing a `{"reason": …}` object at one of three
CI-pinned paths therefore published it in the clear — the same value `<dropped>`
on the allow-listed path and cleartext two keys away, which is exactly the
"artifact asserts a redaction it did not perform" class this series exists to
kill. `assert_no_cleartext_secrets` could not see it: `reason` is not
credential-named.

`PROSE_KEYS` and the prose arm are deleted. `reason` is now `&'static str` on
`Override`, held OUTSIDE the scrubbed tree and spliced in by position after the
scrub; `covers` is a hoisted literal spliced the same way. Only a string literal
can reach either, which a key name cannot forge. Regression test plants four
canaries through both routes.

SF2 — `assert_no_cleartext_secrets` printed the credential it caught, under no
`cfg`, so in release too. It fires only when the redactor has a bug, i.e. the
moment the value must not be written anywhere, and benchmark stderr gets pasted
into issues. The offending value is now digested in the message.

Generator and corpus:
  * the `canary_before_at >= 5` floor had slack 9 against an actual 14, so nine
    `@`-rows could be defanged unnoticed — including `userinfo-slash-pw`, the
    row that memorialises round 5's regression, whose defanging survived
    850/850. Now asserted per-row for every `userinfo*` label; verified the
    exact mutation fails.
  * the plain-token alphabet is pinned by `plain_token_alphabet_is_pinned`.
    Adding `@`, `;`, `&`, `#`, `?`, `%`, `,`, `*` or `!` each survived the whole
    suite; only `:` and `=` were covered.
  * `assert!(checked > 5000)` was tautological (incremented once per iteration);
    it now counts cases that actually reached the digest path.
  * the doc states where the generator's coverage STOPS — it drives
    `redact_connection_like` and `invocation.host` only, so no seed of it could
    have produced SF1.

Documented rather than changed: `is_plain_token` protects nothing
punctuation-free (a bare `hunter2`, a UUID, `ghp_…` are published; `is_secret`
on the key is the sole defence for those), and the digest is an UNSALTED hash
and therefore a commitment — a guessed endpoint can be confirmed offline. Both
are the intended trade; neither was written down.

Also: Cargo.toml's comment still advertised the deleted URL parser;
`shared_corpus_key_prefix`'s digest buys no secrecy while `env.REDIS_KEY_PREFIX`
shows the same text; paths outside `$HOME` stay absolute.

Refs #212
fcostaoliveira added a commit that referenced this pull request Aug 9, 2026
SF1 — `<dropped>` and `<redacted:default>` never reached an artifact, and eight
comments plus a commit message said they did. `snapshot` scrubs the whole
assembled document, so both were re-processed: `<dropped>` is not a plain token
and digested to the hash of the literal `<dropped>`, and `<redacted:default>`
under a credential-named key fell through to `<redacted:set>`.

The second is the damaging one and is the prose bug inverted. An ES/OS/pgvector
run with NO password set published

    effective.ELASTIC_PASSWORD = "<redacted:set>"

— the artifact asserting a credential was supplied when the built-in default was
used, which misleads exactly the operator auditing which credentials a run
actually had. `record_effective_raw` existed to prevent this and was defeated,
so `env_or`'s `is_secret(name) && !was_set` branch was dead as far as the
artifact was concerned.

`scrub_value` now passes an already-emitted sentinel through unchanged. Verified
live: with no password set the artifact reads `<redacted:default>`, an unknown
declared key reads `<dropped>`, and `m`/`ef_construction` still read normally.
Two tests pin it — the write path and `env_or`'s set-vs-default branch end to
end — and removing the arm fails them.

SF2 — the splice's positional coupling between `assemble` and `snapshot` was a
convention shared by two functions and asserted nowhere: reversing the iteration
in `assemble` mis-attributes every reason and left all 852 tests green, because
no test had two overrides with distinct reasons. Now three, checked pairwise;
the `.rev()` mutation fails it.

NITs:
  * round 5's exact shape was still defangable on the row that memorialises it.
    "Some canary before the `@`" passed for
    `redis://admin:pw/CANARY@10.0.0.5:6379/0` — the canary sitting in the half
    the buggy code dropped. The rule is now "in the userinfo, before its first
    internal delimiter", and that mutation dies.
  * the docstring claimed "a canary on BOTH sides" while the code three lines
    down required some side; it now describes what is actually asserted.
  * `MINIMUM_SHAPES` was 55 against an actual 56 — one row deletable silently.
  * `mongodb-srv` had userinfo but its label skipped the per-row loop; renamed
    and given a two-part canary.
  * the #297 remediation was half-followed: the row went, the prose did not.
    Three passages still described `src/vectorsets/` as live or its deletion as
    pending.

Refs #212
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant