True incremental indexing: clean-rebuild parity, 6x post-edit indexing speedup, explicit freshness, and graph-derived search ranking

[ahundt]

1. What does this PR do?

Hey, thanks again for the cool project! This PR continues my earlier PR #151. This Aug 2 addresses your requests: the api-consolidation branch is now merged with upstream main. Two true merge commits bring in every upstream commit through f0c65fb2a997 (2026-07-29): d0f00de6bd6d combines upstream 560ad40d19b2 with the previously verified head b42801f3f765 and resolves all 14 textual conflicts, and 41fd25cdd86b integrates the f0c65fb2a997 release-hardening wave. No history was rewritten; both parents are preserved in each merge. At that merge the branch was 0 commits behind upstream main, and every upstream commit through f0c65fb2a997 remains contained here, so CI can run. A third merge, a0f4380b6f18 (2026-08-01), integrates every upstream commit through d6be58ef9d43 (2026-07-31). At the time of writing (2026-08-02 16:39 UTC, d6be58ef9d43) that is still the upstream main head, so the branch has caught main and is 0 commits behind. Every defect claim in section 4 was re-verified at that revision. Beyond the three merges, 37 follow-up commits fix defects found while validating the merged results (listed with their verification in each commit message). The merge also removed the 86 committed soak/memlab run-log files (1.3 MiB) reported separately as issue #1330; upstream has since removed them on main in 66326c2c.

The core claim is unchanged: upstream main's post-edit indexing is labeled incremental but returns a graph that differs from an independent clean rebuild in 3 of 3 repetitions on the measured build, that is, the graph it serves after an edit is incorrect, and upstream's own source comment documents that FILE_CHANGES_WITH and DATA_FLOWS edges are not reproduced incrementally (pipeline_incremental.c:351-360 at d6be58ef9d43). This PR makes incremental source-structure indexing exact, and in its default configuration indexes the same large-codebase edit 3.2× faster than the measured upstream build (569 vs 1,828 ms). After an edit the server names every deferred derived view (rank, routes, semantic edges, architecture), omits stale rank values where returning them would mislead, and with incremental_derived_results_refresh=at_publish produces a graph identical to the clean rebuild in 3 of 3 repetitions while still beating its own full rebuild (8,568 vs 9,768 ms).

The PR adds optional indexing of installed dependency sources with package identity, optional graph-derived search ranking, and a shorter default MCP tool list, and it extends the read-only Cypher subset, Git-snapshot change detection, and semantic analysis. One attribution correction to my previous description: the shared daemon is upstream's subsystem (src/daemon/, merged here on 2026-07-23); this PR retains it and contributes fixes and hardening on it, listed in section 2. Since findings like these are easy to misread from descriptions alone, I would appreciate review against the code, tests, and published run reports rather than summaries.

The full classic API, request schemas, and upstream 3D visualization are retained. Every major capability has a one-command switch and defaults can be trivially revised if preferred.

2. Capabilities and controls

Every setting is declared in one registry with its default, accepted values, and environment override: CBM_CONFIG_REGISTRY in src/cli/cli.c:12997. Two commands control everything:

codebase-memory-mcp config set <switch> <value>   # flip one switch, e.g. config set rank_enabled false
codebase-memory-mcp config preset apply <name>    # apply a validated multi-setting bundle in one atomic transaction

The six preset bundles pair a tool surface (streamlined- or classic-) with automatic-dependency-source-indexing-disabled or -enabled, plus the rank-disabled and minimal-indexing benchmark ablations (codebase-memory-mcp config preset list shows each name with its description; the apply transaction is cbm_config_apply_preset() in cli.c:12925). When an environment override pins one of a preset's keys, apply returns status 1 rather than claiming an ineffective change.

File names below are unique in the tree and live under src/; each switch is the exact argument list for codebase-memory-mcp, with the default in parentheses. Line numbers are at the branch head 7433fee6bed3.

Capability	What it does and why it is better	Key code	Switch (default)
Exact incremental indexing	After an edit, the non-stale incremental graph equals an independent clean rebuild instead of silently diverging; changed files are scoped by per-file Git status	cbm_pipeline_run_incremental() pipeline_incremental.c:2826	config set incremental_reindex always (default always; full_rebuild rebuilds atomically from scratch and fast_mode_indexes_only restricts incremental reindexing to fast-mode indexes)
Explicit post-edit freshness	The default policy quickly refreshes source structure and names each deferred derived view (rank, routes, semantic edges, architecture) as stale with an actionable refresh instruction; at_publish recomputes everything immediately and matches the clean rebuild exactly	cbm_pipeline_run_incremental() pipeline_incremental.c:2826	config set incremental_derived_results_refresh at_publish config set rank_refresh at_publish (default: defer_all_incremental_reindexes for both)
Shared daemon fixes	The upstream shared daemon is retained with its session-managed lifecycle; this PR's fixes on it: honor the auto_index default at session start, suppress release notices for dev builds, initialize the active-image fingerprint before comparison, validate fingerprint-cache epochs and admit cross-volume caches, and initialize capped memory once in the daemon instead of in every CLI frontend	main_run_daemon_ctl() main.c:1728	daemon start / daemon stop / daemon status (default: session-managed, started automatically)
Class-to-method connection	Upstream links a class to its methods only when extraction supplies the parent; this PR adds the MEMBER_OF reverse edge and a cross-language normalization pass that repairs extraction-miss cases, so classes participate in traversals and inherit rank from their members instead of staying disconnected	process_def() pass_definitions.c:279 repair: pass_normalize.c	always on
Graph-derived search ranking	Orders search results by PageRank/LinkRank computed over the code graph; in the measured search task the known correct answer, zz_order_core, moved from ninth to first	cbm_pagerank_compute() pagerank.c:407	config set rank_enabled false (default true)
Dependency source indexing	Indexes installed package source so search and tracing return external APIs with package identity; upstream resolves manifest imports but does not index installed source. Automatic indexing is off by default so the dependency cost stays out of the default first index; index_dependencies indexes named packages on demand, and when installed packages exceed auto_dep_limit (default 20, 0 = unlimited, max 10,000) the most-imported packages are selected, ranked by project import references	cbm_dep_auto_index_effective() depindex.c:984 ranking: depindex.c:583-616	config set auto_index_deps true (default false)
Streamlined tool list	New MCP sessions see the common tools plus _hidden_tools; calling _hidden_tools reveals the full classic tool list live, no restart (upstream advertises all tools and curates only statically via startup --profile flags)	cbm_mcp_tools_list_range() mcp.c:3429	config set tool_mode classic (default streamlined)
Response encoding	Default reply encoding is configurable between TOON (Token-Oriented Object Notation, a compact text format) and JSON, with a per-call format override and an actionable invalid-format error; the merged head keeps upstream's UTF-8 output guard so quoting or truncation cannot split a multibyte character	cbm_mcp_response_format() mcp.c:2685	config set default_response_format json (default toon)
Automatic first-use indexing	Indexes a project when it is first queried, so first-use calls do not fail on an absent index; while startup indexing is still publishing, tool calls receive structured retry metadata instead of an error or an indefinite wait	registry entry auto_index in CBM_CONFIG_REGISTRY	config set auto_index false (default true; upstream defaults to false and skips with a log hint)
Read-only Cypher subset	Fixes aggregate, post-WITH, and optional-match semantics; adds multi-key ordering, output caps, and schema-aware rejection of unsupported syntax; the tool description, parser, and executor share one definition, and the query schema is stable against incidental storage changes	cbm_cypher_execute_impl() cypher.c:9091	always on
Semantic and similarity edges	Retains and extends upstream semantic analysis; each is now independently selectable instead of tied to index modes	cbm_pipeline_pass_semantic_edges() pass_semantic_edges.c:1783	config set semantic_edges_enabled false config set similarity_enabled false (default true for both)
Git-history analysis	Extends upstream snapshot change detection into a shared module and scopes incremental reindexing by changed Git-status paths; the Git scan now runs concurrently with graph population	cbm_git_snapshot_read() git_snapshot.c:319	config set githistory_enabled false (default true)
HTTP route linking	Keeps upstream's Route-node bridging (pass_route_nodes.c, run in both the full and incremental pipelines) and adds a dedicated pass that links each HTTP call site to its handler, fixing missing or misclassified links	cbm_pipeline_pass_httplinks() pass_httplinks.c:1375	config set httplinks_enabled false (default true)
Startup fingerprint cost policy	The exact executable SHA-256 is reused while kernel-bound file identity and strong change metadata are unchanged, removing a full-image hash from every process start; any ambiguity falls back to an exact full hash, and installer/update candidates always use full exact hashes	registry entry build_fingerprint_mode in CBM_CONFIG_REGISTRY	config set build_fingerprint_mode always_rehash (default cached_exact)

3. Measured results

The headline numbers come from a 39-cell benchmark experiment: 13 configurations, each run 3 times, paired and interleaved across configurations to reduce timing drift, on production builds in isolated Linux Docker containers (clang-18, -O2, 16 build jobs, 16 container CPUs, container --init, fresh case-owned repositories and caches, arm64). Every cell applies the same real C source edit to a large codebase, runs incremental indexing, and compares the resulting graph with a separate clean rebuild after the same edit; every configuration also passes 12 retrieval-evidence checks (4 post-edit retrieval probes across the 3 repetitions: the changed file, its source text, its graph node, and its architecture evidence). All 39 planned cells completed with zero missing, corrupt, duplicate, or unplanned cells. Docker Desktop exposed 7.75 GiB despite the requested 16 GiB limit; the largest observed candidate peak was about 1.6 GiB, so no run was memory-capacity limited.

Two scope statements, up front. First, the measured upstream build is aedb979f5570 (upstream main as of 2026-06-28), pinned when the benchmark cohort was frozen; the upstream main now merged into this branch is d6be58ef9d43 (2026-07-31). The graph-correctness defect reproduced on the measured comparator, and at current upstream main d6be58ef9d43 upstream's source still documents non-reproduced incremental edges (pipeline_incremental.c:351-360, the file is unchanged since f0c65fb2a997) with no parity verification mechanism. Second, this matrix ran at PR revision 2c2de761a24e, so its timing ratios apply to the named revisions, edit, and machine; a second full 39-cell matrix has since re-measured the merged branch head 7433fee6bed3 against an upstream build at d6be58ef9d43 (below). What the merged head does run is the full test suite, including the incremental-parity tests, at the final verified head listed in section 7.

All timings are medians (p50) over the 3 repetitions. Repeated query p50 is the median latency of the repeated read-only graph queries the retrieval probes issue within each candidate's persistent MCP session, so it measures a warm service, not process startup. Response payload p50 is the exact default tool-response size (tokens are the recorded deterministic bytes/4 estimate). Peak RSS is the highest peak resident memory observed across the 3 repetitions. "Deferred views named" means the server explicitly names each optional derived view awaiting recomputation after the fast source-structure refresh, and the freshness-scoped comparison still checks every remaining canonical node, edge, property, and file hash against the independent rebuild. A clean rebuild of the same post-edit tree is the ground truth: a post-edit graph that differs from it is incorrect, and queries against it can return wrong answers.

Configuration	Post-edit graph vs independent clean rebuild (3 repetitions)	Post-edit reindex p50	Full index p50	Repeated query p50	Response payload p50	Peak RSS
Upstream main aedb979f5570, default	Incorrect after the edit: differs from the ground-truth rebuild, 3 of 3	1,828 ms	7.055 s	6.2 ms	604 B (~151 tokens)	not recorded
This PR, upstream-equivalent capabilities (rank, route linking, and dependency indexing off to match upstream's feature set)	Correct on every non-stale row, verified against the ground-truth rebuild, 3 of 3; deferred views named	651 ms	10.265 s	19.5 ms	429 B (~108 tokens)	1,469 MiB
This PR, default	Correct on every non-stale row, verified against the ground-truth rebuild, 3 of 3; deferred views named	569 ms	10.029 s	23.2 ms	429 B (~108 tokens)	1,434 MiB
This PR, automatic dependency indexing enabled	Correct on every non-stale row, verified against the ground-truth rebuild, 3 of 3; deferred views named	774 ms	36.148 s	28.8 ms	429 B (~108 tokens)	1,614 MiB
This PR, derived results refreshed at publish	Fully correct: identical to the ground-truth rebuild, 3 of 3	8,568 ms	9.768 s	22.7 ms	429 B (~108 tokens)	1,450 MiB
This PR, minimal-indexing preset	Fully correct: identical to the ground-truth rebuild, 3 of 3	508 ms	5.136 s	16.7 ms	429 B (~108 tokens)	1,459 MiB

Upstream main's post-edit reindex is labeled incremental but returns an incorrect graph, one that differs from the ground-truth rebuild (speed bought with wrong answers), while this PR's default is both 3.2× faster on the edit (569 vs 1,828 ms) and exact on non-stale source structure. The minimal preset outputs a graph identical to the clean rebuild while indexing the edit 3.6× faster (508 vs 1,828 ms) and full-indexing 1.37× faster (5.136 vs 7.055 s) than the measured upstream build. Two metrics go the other way and I state them plainly: upstream's repeated read-only query latency is lower (6.2 vs 16.7 to 23.2 ms; this PR's replies carry first-response freshness and provenance context and are 29% smaller, 429 vs 604 bytes), and upstream's full index is faster than this PR's default configuration (7.055 vs 10.029 s) because the default enables optional capabilities upstream does not have: disabling semantic edges alone brings the full index to 6.659 s, and the rank, similarity, git-history, and route ablations measure 9.269 to 10.140 s. Upstream peak RSS was not recorded in this run, so no memory comparison against upstream is claimed.

Refresh-at-publish recomputes every global derived result (rank, semantic edges, architecture, routes) immediately after the edit; at 8,568 ms that is still faster than its own 9,768 ms full rebuild, which is the only other way to get them all fresh. It is opt-in; the default defers exactly that graph-wide work and names the deferred views until refreshed.

Automatic dependency indexing stays off by default so its cost stays out of the default first index: enabling it indexed this workload's six installed packages in about 27 s of dependency work at first index, which is work upstream cannot do, and after an edit, checking unchanged dependencies cost 13 ms.

Two earlier development checkpoints (c9726c7fd9b7, 0fcb7e109518) ran in the same matrix and quantify what the post-merge work fixed: their native configurations auto-indexed dependencies (about 20.5 to 20.8 s of the 43.8 s full index) and carried a since-removed startup regression that pushed repeated queries to 187 to 206 ms; the current default full-indexes in 10.0 s with 23.2 ms repeated queries, and both checkpoints also failed the canonical-graph gate that the current head passes.

A second 39-cell matrix re-measured the merged result: same container design (13 configurations, 3 repetitions each, paired-interleaved, isolated Linux Docker containers, arm64, 16 container CPUs), run at branch head 7433fee6bed3 (2026-08-02) with the upstream comparator freshly built at d6be58ef9d43, the upstream main head this branch merges. All 39 planned cells completed with zero missing, corrupt, duplicate, or unplanned cells. The mutation differs from the first matrix (a new isolated C source file is added instead of editing an existing one), so timings are not comparable between the two matrices. Upstream's default again served an incorrect post-mutation graph in 3 of 3 repetitions: it differs from the ground-truth rebuild of the same tree, so queries after the edit can return wrong answers, and upstream ships no mechanism that detects this. Every configuration of this PR passed its graph gate.

Configuration	Post-mutation graph vs independent clean rebuild (3 repetitions)	Post-mutation reindex p50	Clean rebuild p50	Repeated query p50	Response payload p50	Peak RSS
Upstream main d6be58ef9d43, default	Incorrect after the edit: differs from the ground-truth rebuild, 3 of 3	3,208 ms	7.508 s	17.3 ms	306 B (~77 tokens)	1,536 MB
This PR, upstream-equivalent capabilities	Correct on every non-stale row, verified against the ground-truth rebuild, 3 of 3; deferred views named	473 ms	8.025 s	17.6 ms	407 B (~102 tokens)	1,532 MB
This PR, default	Correct on every non-stale row, verified against the ground-truth rebuild, 3 of 3; deferred views named	523 ms	8.674 s	20.0 ms	407 B (~102 tokens)	1,566 MB
This PR, derived results refreshed at publish	Fully correct: identical to the ground-truth rebuild, 3 of 3	7,805 ms	8.437 s	22.0 ms	407 B (~102 tokens)	1,522 MB
This PR, minimal-indexing preset	Fully correct: identical to the ground-truth rebuild, 3 of 3	437 ms	4.141 s	16.6 ms	407 B (~102 tokens)	1,515 MB

minimal-indexing is the measured Pareto frontier of this matrix: no passing, fully measured candidate in the same workload is at least as good on overall quality and every cost axis while strictly better on any. All speed comparisons below are how much faster this PR is than current main (d6be58ef9d43) in this matrix; a negative value means slower, and every negative is bought by a named capability gain. Post-mutation indexing: +578% (473 vs 3,208 ms) at upstream-equivalent capabilities and +513% (523 ms) at the default, roughly 6x either way, and correct where upstream's result is incorrect. Clean rebuild: -6% (8.025 vs 7.508 s) at upstream-equivalent capabilities (parity verification runs in the test and benchmark harness, so it adds nothing here; the difference is graph-building work this branch always does, including the MEMBER_OF class-connectivity repair and exact statistics publication, with no single component isolated by an ablation, and at this price the post-mutation graph is verified correct where upstream's is incorrect); -13% (8.674 s) at the default, which adds ranking, similarity, git history, and route linking with the measured benefits in the table below; +81% (4.141 s) with minimal-indexing, fully correct. Repeated query: -2% (17.6 vs 17.3 ms) at upstream-equivalent capabilities, closing the first matrix's 3 to 4x gap, -14% (20.0 ms) at the default, +4% (16.6 ms) with minimal-indexing, at effectively equal peak memory (1,532 vs 1,536 MB). In this workload upstream's default reply is smaller (306 vs 407 bytes); the PR's replies carry the first-response freshness and provenance context described above. Refresh-at-publish again produced the fully identical graph while beating its own clean rebuild (7,805 ms vs 8,437 ms).

The default's remaining costs are itemized capability prices, not overhead, and each is refundable with one switch. Overhead, cost that returns no value, is what this head (7433fee6bed3) removed: the first matrix's repeated-query deficit does not reproduce (17.6 ms with upstream-equivalent capabilities vs upstream's 17.3 ms), and the like-for-like clean rebuild is -6% (0.52 s, 8.025 vs 7.508 s) at equal peak memory. Parity verification runs in the test and benchmark harness, not in production indexing, so that 0.52 s is not a verification tax: it is graph-building work this branch always does, including the MEMBER_OF class-connectivity repair and exact statistics publication, with no single component isolated by an ablation. What the build delivers at that price is the guarantee upstream does not offer at any price: the post-mutation result is verified correct. Every cost above that baseline is priced below against the same build with exactly that capability toggled (all other defaults unchanged); the default configuration's own rebuild spread is [8.437, 9.599 s] over 3 repetitions, so prices under 0.6 s are smaller than that spread and should be read as approximate:

Capability (default state)	Clean-rebuild price; repeated-query price	Measured benefit on the same build (section 3 quality experiment; matrix cells)	Refund switch
Semantic edges (on)	+3.25 s (8.674 vs 5.421 s); +0.8 ms	Every judged SEMANTICALLY_RELATED pair matches the clean rebuild, none missing, none spurious	config set semantic_edges_enabled false
Graph-derived ranking (on)	+0.56 s, below the run spread; +1.8 ms	The known correct answer returned 1st of 9 instead of 9th of 9; MRR 1.000 vs 0.111	config set rank_enabled false
Similarity edges (on)	+0.34 s, below the run spread; +1.0 ms	SIMILAR_TO edges for near-duplicate discovery	config set similarity_enabled false
HTTP route linking (on)	+0.31 s, below the run spread; +1.0 ms	Each HTTP call site linked to its handler	config set httplinks_enabled false
Git-history analysis (on)	+0.18 s, below the run spread; +0.7 ms	FILE_CHANGES_WITH coupling edges and Git-scoped incremental change detection	config set githistory_enabled false
Dependency source indexing (off)	+23.25 s and +5.4 ms only when enabled	An installed dependency's API returned 1st with package identity, vs absent	off by default; config set auto_index_deps true

Summed, the default's entire repeated-query premium over upstream at the merged head (7433fee6bed3) is 2.7 ms (20.0 vs 17.3 ms). The one axis where upstream is faster like-for-like is the clean rebuild, where this PR is -6%; what stands on the other side of that price is the post-mutation result: +578% (473 vs 3,208 ms) with a verified-correct graph, where upstream's faster-looking incremental serves an incorrect graph, one that differs from its own ground-truth rebuild in 3 of 3 repetitions, speed bought with wrong answers and no switch that refunds it.

Result-quality contribution of the two new optional capabilities, isolated on one PR build (13a4679b27f6) and test project by repeating the same index-then-query measurement 5 times per state, alternating the capability between enabled and disabled. MRR (Mean Reciprocal Rank) rewards placing the known correct answer near the top, Hit@1 and Hit@5 report whether it appears first or within the first five, nDCG@5 (normalized Discounted Cumulative Gain) rewards correct ordering within the first five, and task success means the task's named expected answer was returned:

Capability	Enabled	Disabled (same build)	Cost of enabling
Graph-derived ranking. Task: rank the test project's structurally central order workflow (zz_order_core) ahead of eight lexical-only decoys	zz_order_core returned 1st of 9; MRR, Hit@1, Hit@5, nDCG@5, task success all 1.000	zz_order_core 9th of 9; MRR 0.111, the other four metrics 0	+0.2 ms median query, +12 ms test-project index
Dependency source indexing. Task: retrieve an installed dependency's imported API (canonicalDependencyAPI) with its dependency, package, and read-only provenance on the same result	canonicalDependencyAPI returned 1st, with package identity	canonicalDependencyAPI absent	+0.6 ms query, +13 MiB RSS, test-project full index 81 vs 51 ms

Semantic refresh quality was re-verified in the same matrix: with refresh at publish, every judged SEMANTICALLY_RELATED pair matches the clean rebuild, with none missing and none spurious.

4. Upstream main defects this PR corrects

Every claim in this table was re-verified by direct code reading at upstream main d6be58ef9d43 (2026-07-31, the revision merged into this branch) unless a different revision is named; upstream file:line references are at that revision.

Improvement in this PR	Key code	Issue at upstream main d6be58ef9d43	Consequence
The non-stale graph produced after an edit equals an independent clean rebuild, and the fully refreshed graph is identical (measured above); the PR's core correction	cbm_pipeline_run_incremental() pipeline_incremental.c	Upstream's own comment documents that FILE_CHANGES_WITH and DATA_FLOWS edges are not reproduced incrementally (pipeline_incremental.c:351-360); no parity guarantee or verification mechanism exists; measured divergence 3 of 3 on aedb979f5570	Any answer could be wrong after an edit
A query detects that another process replaced the database file and reopens it, with no global lock (ThreadSanitizer clean)	resolve_store() mcp.c	The cached fast path returns the held store with no file-identity check (resolve_store_internal, mcp.c:2081-2083); upstream's daemon single-ownership makes cross-process replacement rarer but the check is still absent	Queries silently read a replaced or deleted database
Cypher aggregates, post-WITH stages, optional matches, and multi-key ORDER BY compute correctly	cbm_cypher_execute_impl() cypher.c	parse_order_by_clause parses exactly one ordering key, so secondary ORDER BY keys are silently dropped (cypher.c:1699-1709); the aggregate, post-WITH, and optional-match failures were verified on the measured comparator aedb979f5570; upstream has since fixed aggregate-column classification and two optional-match defects (PRs #1221, #1177, #1385, contained here through the d6be58ef9d43 merge), while multi-key ORDER BY remains absent upstream	Wrong values, no error
A Cypher query considers every eligible row before output limits apply; LIMIT lowers but cannot bypass the configured row cap, and truncation is reported with the limiting condition instead of being silent	scan_pattern_nodes() cypher.c	Unlabeled scans cap candidates at max_rows times a fixed growth factor before inline property filters and WHERE run (cypher.c:3027-3052, cap applied at cypher.c:3037), so matching rows past the prefix are discarded	Results silently missing, whatever the query needed
The first query against a project indexes it automatically; when indexing is blocked or still publishing, the reply names the required action or returns structured retry metadata	registry entry auto_index in CBM_CONFIG_REGISTRY	Auto-index defaults to false and an unset value skips indexing with only a log hint (mcp.c:11156-11169)	First use fails outright, recovery requires reading server logs
Classes get a MEMBER_OF reverse edge and a cross-language repair pass for extraction-miss cases	process_def() pass_definitions.c:279 repair: pass_normalize.c	Upstream mints DEFINES_METHOD only when extraction supplies the parent; there is no reverse edge and no repair pass (pass_normalize.c does not exist upstream), so extraction-miss classes stay disconnected	Disconnected classes invisible to traversals and rank
index_status reports whether the working tree is dirty and whether the index matches it	add_git_context_json() mcp.c	add_git_context_json (mcp.c:2242) emits branch and commit identity but no working-tree-state field; upstream's watcher tracks a dirty signature internally but does not report it as provenance	Change tracking can trust stale provenance
Git-history coupling edges are selected best-first under a configurable budget, and a partial result is reported with exact written, eligible, and omitted counts; per-file temporal data grows exactly with observations	cbm_pipeline_githistory_compute_with_limits() pass_githistory.c:646	The initial index caps FILE_CHANGES_WITH candidates at a hard-coded 8,192 (MAX_COUPLINGS, pass_githistory.c:346) collected in hash-table iteration order with no sort, no configuration, and no log (pass_githistory.c:215-217, 331), so on a history with more qualifying pairs the emitted coupling edges are an arbitrary subset rather than the strongest couplings; per-file temporal facts stop at the first 16,384 files encountered (MAX_FILE_TEMPORAL, pass_githistory.c:406)	Coupling and hotspot answers silently arbitrary on large histories

Three defect claims from my previous description are retired because re-auditing upstream showed them already fixed there: the half-opened SQLite handle on failed open (closed upstream in e6e9c58e, before the measured comparator), the config-linking pass's large stack arrays (the pass was rewritten upstream and now holds only small static tables), and the dropped route-discovery pass (rebuilt upstream as pass_route_nodes.c; the merged branch keeps upstream's pass and adds its own, section 2). Shutdown lifecycle also converged: upstream's daemon now joins worker threads and plumbs cancellation, and the merged branch carries both sides' shutdown work; this PR's remaining distinct contribution there is the verification discipline (leak gate, cancellation and reaping tests) rather than a defect claim.

One further upstream hard-limit defect is fixed here but deliberately kept out of the table, because neither tree currently wires the affected scanner into a production pass: upstream's env-URL walker holds a fixed 512-byte-per-frame path stack and silently skips directories nested deeper than 256 levels (pass_envscan.c:514, 532 at d6be58ef9d43). This branch's walker uses dynamically grown path storage (walk_path.h), logs unrepresentable bindings instead of dropping them, and pins both behaviors with the regression tests envscan_walks_more_than_256_pending_directories and envscan_accepts_root_path_longer_than_512_bytes.

Every remaining row is regression-tested. Defects found while merging upstream and validating the result were fixed before this update; each is documented in its commit message. Two behaviors are boundaries rather than defects: the Cypher subset does not claim full openCypher conformance, and optional derived results refresh on the documented deferred schedule. Production src/ contains no system() calls, and the heap-leak gate reports 0 leaked bytes at the final verified head.

5. Compatibility, breaking changes, and migration

1. No classic API removal: classic request schemas, handlers, and the full tool list remain, and upstream's CLI access to every read-only MCP operation is retained; config set tool_mode classic restores the classic list on a running installation. Streamlined and classic surfaces answer from the same graph and Cypher contract.
2. New installations change only the defaults in the capability table; each row names the switch that restores upstream-like behavior.
3. The shared daemon is upstream's subsystem and keeps its upstream lifecycle: session-managed start, stop with the last session, daemon stop refusal while committed clients remain, and orderly SIGINT/SIGTERM shutdown (cancel and reap work, join background threads, close transports, release claims and leases, remove the endpoint). This PR's daemon changes are the fixes listed in section 2.
4. Upstream's packaging decisions are carried through the merge unchanged: one binary per platform (the Windows launcher stub upstream removed after antivirus false positives stays removed), and the binary carries no self-update capability or download URL; install scripts own updating.
5. Upgrades migrate installer-owned state, preserve user edits, and keep existing indexes unless --reset-indexes is passed; the system-wide upgrade verified at an earlier head preserved all 462 existing indexes across 17 detected clients. The installer recognizes configuration written by earlier development builds and replaces it canonically instead of rejecting it, keeps user-owned TOML child tables when replacing a managed parent table, and refuses to overwrite genuinely user-edited entries.
6. This local validation ran on macOS, with the benchmark matrix in Linux Docker containers. Windows-specific code paths are covered by automated tests, and because the branch contains every upstream commit through d6be58ef9d43, the repository's CI has run its full matrix on this pull request at 7433fee6bed3: the test shards (Linux gcc, macOS 14 and 15-intel, Windows CLANG64), ThreadSanitizer on three platforms, lint, smoke on Linux, macOS, and Windows, CodeQL, and the security gates all pass; the DCO sign-off check is the only failing check.

6. How to run the tests

make -f Makefile.cbm build/c/codebase-memory-mcp   # production binary
make -f Makefile.cbm test          # full C suite under AddressSanitizer/UndefinedBehaviorSanitizer
make -f Makefile.cbm test-leak     # heap-leak check (Apple leaks on macOS, LeakSanitizer on Linux); report in build/c/leak-report.txt
make -f Makefile.cbm lint-ci       # the CI-pinned cppcheck 2.20 and LLVM 20 format gates
make -f Makefile.cbm test-analyze  # Clang static analyzer (requires clang)
uv run python benchmarks/run_experiments.py        # one-repetition benchmark smoke; --full reruns the 39-cell matrix in Docker

The full suite passed with zero failures at the final verified head (7,823 passed, 2 platform-specific skips), and the CI lint gates (lint-ci) exit 0 there. The suite includes generalized extraction and incremental tests across C, C++, Go, Python, JavaScript, TypeScript, Java, Kotlin, and Rust, mixed-language call resolution, the daemon lifecycle, bootstrap, and IPC suites, and the tests that arrived with the merged upstream commits. The benchmark runner records exact candidate commits, binary SHA-256 values, compiler and optimization flags, CPU and memory allocation, environment overrides, case cleanup, and invalid-run status in an immutable audit manifest.

7. Evidence and revisions

Name	Revision	Role
Upstream main merged	f0c65fb2a997	Upstream main head at the 2026-07-29 merge; every upstream commit through this revision is contained in the branch
Upstream main merged, current	d6be58ef9d43	Upstream main head (2026-07-31), integrated by the full-sync merge below so the branch is again 0 commits behind; every section 4 defect claim re-verified at this revision; also the second benchmark matrix's freshly built upstream comparator
Upstream sync merge	d0f00de6bd6d	True two-parent merge of upstream 560ad40d19b2 and the previous verified head b42801f3f765; all 14 textual conflicts resolved, no history rewrite
Release-hardening merge	41fd25cdd86b	True two-parent merge integrating upstream f0c65fb2a997
Full-sync merge	a0f4380b6f18	True two-parent merge (2026-08-01) integrating every upstream commit through d6be58ef9d43; no history rewrite
Final verified HEAD	24aa8a415f5c	Full C suite, ThreadSanitizer suite, heap-leak gate, and CI lint gates ran at this commit
Branch head	7433fee6bed3	Current head (2026-08-02); the repository CI matrix passes here (section 5) and the second 39-cell matrix ran here
Benchmark production build, first matrix	2c2de761a24e	The first 39-cell matrix and the installed dogfood ran at this commit
Measured upstream comparator, first matrix	aedb979f5570	Upstream main 2026-06-28, pinned when the first benchmark cohort was frozen (section 3)
Development checkpoints	c9726c7fd9b7, 0fcb7e109518	Regression boundaries measured in both matrices
Quality-experiment build	13a4679b27f6	Enabled/disabled ranking and dependency measurements

All experiments ran on production builds. The first 39-cell run set's immutable audit manifest has SHA-256 1becd57c00edbe965073f4d7c9a79cca8812947f207ed51851c856369aa2c002, and the second run set's (branch head 7433fee6bed3, 2026-08-02) has SHA-256 1e8f0d96128bd352107a0fb1633bcb0bcbd696a5ea8cb8226569d62e580a65e0; each experiment root retains the input specification, per-run manifests, container environment records, logs, build identities, and cleanup status, so the numbers are independently reproducible with the command above. The generated reports are published in a gist; its current tree contains the 2026-07-28 run set and the earlier 39-cell run be2097bdbf14, and the focused ranking, dependency, and semantic reports remain in the gist revision history.

Ranking follows PageRank (Brin and Page, 1998) and LinkRank (Kim, Son, and Jeong, 2010); nDCG (Järvelin and Kekäläinen, 2002) defines the ranking metric; RepoGraph (ICLR 2025) motivates repository graphs. The repository experiments, not the citations, are the evidence that this implementation works.

8. Checklist

• Tests pass locally (make -f Makefile.cbm test at the final verified head: 7,823 passed, 2 platform-specific skips)
• Lint passes with the CI-pinned toolchain (make -f Makefile.cbm lint-ci exit 0 at the final verified head: cppcheck 2.20, LLVM 20 clang-format, suppression-policy and source-safety checks; the previous body's 49 open findings and 20 formatting violations are cleared)
• New behavior is covered by a test (reproduce-first for bug fixes)

[DeusData]

Thank you for the substantial follow-up and for working through the earlier blockers. We have recorded the updated benchmark, scope changes, and current DCO state.

This still needs a fresh end-to-end correctness and security review of the current diff before we can make any merge commitment. Please keep contributor-authored commits DCO-compliant. If maintainers ultimately distill part of the work into a separate implementation, we will preserve clear contributor credit through appropriate co-authoring or explicit attribution. The review queue remains full, but this is queued for that fresh review.