True incremental indexing: clean-rebuild parity, 6x post-edit indexing speedup, explicit freshness, and graph-derived search ranking

.clangd

@@ -0,0 +1,36 @@
+# clangd configuration for codebase-memory-mcp
+#
+# Mirrors the include paths and defines from Makefile.cbm CFLAGS_COMMON so
+# clangd can resolve all headers without needing compile_commands.json.
+# Paths are relative to the project root (where this file lives).
+#
+# Works with both clang (macOS/Linux) and gcc — clangd uses these flags
+# directly regardless of which compiler is selected for the build.
+
+CompileFlags:
+  Add:
+    - -std=c11
+    - -D_DEFAULT_SOURCE
+    # Project source headers
+    - -Isrc
+    # Vendored libraries: yyjson, xxhash, sqlite3 wrappers
+    - -Ivendored
+    - -Ivendored/sqlite3
+    - -Ivendored/mimalloc/include
+    # Internal cbm extraction layer and tree-sitter runtime
+    - -Iinternal/cbm
+    - -Iinternal/cbm/vendored/ts_runtime/include
+  # Remove flags clangd cannot handle (sanitizer, link flags)
+  Remove:
+    - -fsanitize=*
+    - -fno-omit-frame-pointer
+    - -lstdc++
+    - -lpthread
+    - -lm
+    - -lz
+
+Diagnostics:
+  # Suppress false-positive "implicit declaration" warnings caused by
+  # clangd analysing files in isolation without the full TU context.
+  Suppress:
+    - pp_file_not_found

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -98,7 +98,7 @@ body:
     attributes:
       label: Logs
       description: |
-        Relevant log output (stderr of the binary; for the UI variant also
+        Relevant log output (stderr of the binary; for UI issues also
         `GET /api/logs`). Will be auto-formatted as code — no backticks needed.
         Please check for private paths/strings before pasting.
       render: text

.github/workflows/_lint.yml

@@ -20,6 +20,18 @@ jobs:
       - name: No-skips policy (tests pass or fail)
         run: bash scripts/check-no-test-skips.sh
 
+      # Runs before the C toolchain setup so a harness regression fails in seconds
+      # rather than after the cppcheck build. Pure stdlib: no binary, no network,
+      # synthetic SQLite fixtures and the checked-in manifests only.
+      - name: Benchmark harness tests
+        run: python3 benchmarks/test_rank_quality.py
+
+      # The allowlist is generated from CBM_CONFIG_REGISTRY and the cbm_config_get_*
+      # call sites, so a config change that does not regenerate it is caught here
+      # instead of silently letting a stale key through benchmark validation.
+      - name: Benchmark config-key allowlist is current
+        run: python3 benchmarks/generate_config_keys.py --check
+
       - name: Install build deps
         run: sudo apt-get update && sudo apt-get install -y zlib1g-dev cmake
 

.github/workflows/_test.yml

@@ -534,7 +534,7 @@ jobs:
         run: repo/scripts/ci/verify-shard-union.sh manifests
 
   # Windows product-surface regression guards. Distinct from test-windows above
-  # (the sanitizer C suite): these drive a real product binary + external UI pack
+  # (the sanitizer C suite): these drive a real product binary with embedded UI
   # over stdio / CLI / HTTP and fail if a Windows bug already fixed on main comes
   # back -- non-ASCII repo paths dropping definitions (#636/#357, fixed by #700),
   # the PreToolUse hook augmenter no-op on drive-letter cwd (#618, fixed by #619),
@@ -571,9 +571,9 @@ jobs:
             ccache-guards-${{ github.ref }}-
             ccache-guards-
 
-      - name: Build product binary with external UI assets and guard seams
+      - name: Build product binary with embedded UI assets and guard seams
         shell: msys2 {0}
-        # --with-ui builds the frontend and its content-addressed asset pack, so
+        # --with-ui builds and embeds the frontend, so
         # the drive-picker guard's HTTP UI is available. Functional gate only
         # (no sanitizers).
         run: scripts/build.sh --with-ui CC=clang CXX=clang++ TEST_SEAMS=1

.github/workflows/dry-run.yml

@@ -147,7 +147,7 @@ jobs:
           path: ${{ runner.temp }}/release-archives
 
       # Same extraction the release verify job performs. VirusTotal receives
-      # only extracted files and unpacked UI assets, never .tar.gz/.zip containers.
+      # only extracted release objects, never .tar.gz/.zip containers.
       - name: Extract archive artifacts
         run: |
           scripts/ci/extract-release-archives.sh "$RUNNER_TEMP/release-archives" binaries \

.gitignore

@@ -10,7 +10,10 @@ bin/
 # Test artifacts
 *.test
 *.out
+*.log
 coverage.txt
+# Clang static analyzer (make -f Makefile.cbm test-analyze) per-file reports
+*.plist
 
 # Test fixture temp dirs (created by C test suite in CWD instead of /tmp/)
 cbm_*/
@@ -26,6 +29,16 @@ cli-*/
 .DS_Store
 Thumbs.db
 
+# Git worktrees (created by Claude Code subagents)
+.worktrees/
+
+# Runtime/session artifacts
+session_project
+project
+project|params.project
+conductor/
+with
+
 # Database files (local cache)
 *.db
 *.db-wal
@@ -40,9 +53,11 @@ Thumbs.db
 # Local project memory (Claude Code auto-memory)
 memory/
 reference/
+.claude/
 
 # Local-only scratch / session notes (never pushed)
 private/
+notes/
 
 # Build artifacts
 build/
@@ -54,11 +69,23 @@ src/ui/embedded_assets.c
 
 # Generated reports
 BENCHMARK_REPORT.md
+benchmark-results/
+*.facts/
 TEST_PLAN.md
+scripts/autotune_results.json
 CHANGELOG.md
 
-# Soak test output
+# Local memory/soak outputs (uploaded as CI artifacts, never committed)
+memlab-*.jsonl
 soak-results/
+soak-results-query-leak/
+# ...and ANY ad-hoc run directory at the repo root. The two exact names above did
+# not match hand-named runs, so 13 soak directories and 4 memlab files (78 files
+# of logs and CSVs) were committed to main by accident in 7808eee. Root-anchored
+# so nothing under scripts/ or tests/ is affected — scripts/soak-legs.sh and
+# scripts/soak-test.sh stay tracked.
+/soak*/
+/memlab-*
 
 # LSP originality-check reference cache (scripts/check-lsp-originality.sh)
 .lsp-refs/
@@ -70,13 +97,15 @@ graph-ui/.npm-cache-local/
 __pycache__/
 *.pyc
 
-# Local soak-leg outputs (uploaded as CI artifacts, never committed)
-soak-results/
-soak-results-query-leak/
-# ...and ANY ad-hoc run directory at the repo root. The two exact names above did
-# not match hand-named runs, so 13 soak directories and 4 memlab files (78 files
-# of logs and CSVs) were committed to main by accident in 7808eee. Root-anchored
-# so nothing under scripts/ or tests/ is affected — scripts/soak-legs.sh and
-# scripts/soak-test.sh stay tracked.
-/soak*/
-/memlab-*
+# Benchmark data derived from a corpus, never authored. Test labels are generated by
+# benchmarks/generate_test_labels.py from whichever checkout was measured, so a copy in
+# the repository is both large and able to drift from the tree actually indexed; the
+# harness derives them at run time instead. Campaign specs and rollups are likewise
+# regenerated from benchmarks/campaign_specs.py and benchmarks/rank_report.py.
+# NOT ignored, deliberately: benchmarks/config-keys-v1.json and
+# benchmarks/config-spellings-v1.json, which are generated but checked in as allowlists
+# and guarded by `generate_config_keys.py --check` in CI.
+benchmarks/labels/
+benchmarks/campaign-specs/
+benchmarks/rollup*.json
+benchmarks/rollup*.md

.mcp.json

@@ -0,0 +1,8 @@
+{
+  "mcpServers": {
+    "codebase-memory-mcp": {
+      "command": "sh",
+      "args": ["-c", "exec $HOME/.local/bin/codebase-memory-mcp"]
+    }
+  }
+}

CLAUDE.md

@@ -0,0 +1,61 @@
+# codebase-memory-mcp — Developer Notes for Claude
+
+Every commit needs a DCO sign-off: commit with `git commit -s`; CI enforces it via `scripts/check-dco.sh` (see CONTRIBUTING.md).
+
+Before changing capabilities or architecture, map the existing design first: look for
+equivalent tools, config, helpers, metadata, algorithms, and conventions. Prefer extending
+the established path over adding a parallel one. New abstractions should close a named gap
+and fit the repo's ownership, allocation, threading, logging, portability, protocol I/O,
+and naming patterns.
+
+Adding a new node/edge JSON property key in `src/pipeline/*.c` or `src/git/*.c`? Add it to
+the declared registry in `src/store/store.c`/`store.h` (`schema_declared_node_property_keys`/
+`schema_declared_edge_property_keys`) or `tests/test_schema_declared_property_keys.c` will fail.
+
+## Build & Test (C server)
+
+All C targets use `Makefile.cbm`:
+
+```bash
+make -f Makefile.cbm test          # build + run full test suite (ASan/UBSan)
+make -f Makefile.cbm test-leak     # heap leak check (see below)
+make -f Makefile.cbm test-analyze  # Clang static analyzer (requires clang, not gcc)
+```
+
+## Memory Leak Testing
+
+**macOS** — uses Apple's `leaks --atExit` on a separate ASan-free binary:
+```bash
+make -f Makefile.cbm test-leak
+# Report saved to build/c/leak-report.txt
+# Target line: "Process NNNNN: 0 leaks for 0 total leaked bytes."
+```
+
+**Linux** — uses LSan via ASan env var on the regular test runner:
+```bash
+make -f Makefile.cbm test-leak
+# Report saved to build/c/leak-report.txt
+# Exit 0 = no leaks.
+```
+
+Why a separate binary on macOS: `leaks` cannot inspect processes that use a custom malloc (ASan replaces it). The `test-runner-nosan` target rebuilds without `-fsanitize` flags specifically for this purpose.
+
+## Memory-Corruption Debugging (macOS)
+
+For non-deterministic corruption (uninit reads, use-after-free, overruns) that ASan/TSan miss — used to investigate the custom-writer B1 bug. Both run the nosan binary (ASan replaces malloc, which defeats these libmalloc knobs); set `CBM_ONLY_SUITE=<suite>` to target a slow suite.
+
+```bash
+make -f Makefile.cbm test-memory    # MallocScribble=1 + MallocPreScribble=1
+                                    # uninit reads -> 0xAA, use-after-free -> 0x55 (deterministic)
+make -f Makefile.cbm test-gmalloc   # Guard Malloc (libgmalloc): guard page per allocation
+                                    # allocation-owning suites; exact overrun/UAF stack
+# Report saved to build/c/mem-report.txt
+```
+
+`test-memory` is the macOS MSan-equivalent for uninit reads (scribble makes them deterministic). `test-gmalloc` is the strictest — it crashes at the exact bad write, pinpointing the line. Its default suite list stays on allocation-owning in-process surfaces because macOS propagates `DYLD_INSERT_LIBRARIES` into external helpers such as `git`; use `CBM_ONLY_SUITE`/`CBM_ONLY_TEST` for a narrower probe. (No valgrind/MSan on macOS; on Linux use `-fsanitize=memory`.)
+
+## Project Structure (C server)
+
+Sources live under `src/`; tests under `tests/`; vendored C libs under `vendored/`.
+Tree-sitter extraction and its vendored language grammars live under `internal/cbm/`.
+See `CONTRIBUTING.md` for the complete source layout and contribution workflow.

CONTRIBUTING.md

@@ -26,19 +26,101 @@ The binary is output to `build/c/codebase-memory-mcp`.
 scripts/test.sh
 ```
 
-This builds with ASan + UBSan and runs the full C test suite. Key test files:
+This is the local maintainer gate used by the project scripts. It cleans the C build, runs the
+full ASan + UBSan C test suite, builds the production binary, and then runs the parent-watchdog
+and security-string regression scripts. The exact test count changes as suites are added; use the
+runner summary as the source of truth. Key test files:
 - `tests/test_pipeline.c` — pipeline integration tests
 - `tests/test_httplink.c` — HTTP route extraction and linking
 - `tests/test_mcp.c` — MCP protocol and tool handler tests
 - `tests/test_store_*.c` — SQLite graph store tests
 
+Useful script options and environment:
+
+```bash
+scripts/test.sh --arch arm64             # macOS: force target architecture
+scripts/test.sh CC=clang CXX=clang++     # override compiler
+CBM_RUN_HANG_TEST=1 scripts/test.sh      # include the slower C++ index-hang guard
+```
+
+## Run C Server Tests
+
+The MCP server core is written in C and has its own test suite under `tests/`:
+
+```bash
+make -f Makefile.cbm test          # full suite with ASan + UBSan
+make -f Makefile.cbm test-tsan     # thread-sensitive suites with ThreadSanitizer
+make -f Makefile.cbm test-leak     # heap leak check (see below)
+make -f Makefile.cbm test-memory   # macOS MallocScribble/PreScribble nosan run
+make -f Makefile.cbm test-gmalloc  # macOS Guard Malloc nosan run
+make -f Makefile.cbm test-analyze  # Clang static analyzer (requires clang, not gcc)
+```
+
+Focused runs are opt-in. Leave these unset for the complete suite:
+
+```bash
+CBM_ONLY_SUITE=pipeline make -f Makefile.cbm test
+CBM_ONLY_SUITE=pipeline CBM_ONLY_TEST=exact build/c/test-runner
+```
+
+By default, tests isolate `CBM_CACHE_DIR` in a temporary directory so local indexes are not
+polluted. Set `CBM_TEST_NO_ISOLATE=1` only when intentionally testing the user's configured cache.
+
+### Build profiles
+
+Use `scripts/build.sh` for a clean local release build. It is the same entry point used by the
+release workflow and produces the default installable binary with `-O2`. For a faster incremental
+release rebuild, use `make -f Makefile.cbm cbm`; `make -f Makefile.cbm install` installs that same
+optimized artifact.
+
+Use `scripts/test.sh` for normal development validation. Its C test binary uses `-g -O1` with
+ASan and UBSan, then it builds the production binary for the parent/worker watchdog checks. Use
+the dedicated `test-tsan`, `test-leak`, `test-memory`, and `test-gmalloc` targets above when
+diagnosing concurrency or allocator lifetime behavior. These diagnostic binaries are not valid
+performance-benchmark inputs.
+
+For performance changes, use the canonical fact tables and comparison rules in
+[`docs/BENCHMARK_EXPERIMENTS.md`](docs/BENCHMARK_EXPERIMENTS.md). Field, step,
+concurrency, and lifecycle meanings come from the generated
+[`docs/BENCHMARK_TERMINOLOGY.md`](docs/BENCHMARK_TERMINOLOGY.md), whose source of
+truth is `benchmarks/terminology.json`.
+
+Additional build flags follow the Makefile conventions:
+
+```bash
+make -f Makefile.cbm test SANITIZE=      # disable ASan/UBSan, mainly for unsupported toolchains
+make -f Makefile.cbm cbm CFLAGS_EXTRA=-DCBM_VERSION=dev
+make -f Makefile.cbm cbm STATIC=1        # static link where supported
+```
+
+**Memory leak detection:**
+
+On **macOS**, `test-leak` builds a sanitizer-free binary (`test-runner-nosan`) and runs Apple's
+`leaks --atExit` on allocation-owning store, pipeline, ranker, and parallel-worker
+suites. ASan replaces malloc, so the standard `test-runner` cannot be inspected by `leaks` — the
+separate nosan build is required. Deliberate crash tests are excluded because Apple's debugger
+stops their child processes before the parent can reap them; this includes the subprocess,
+stack-overflow, MCP crash-quarantine, and HTTP socket-inheritance suites.
+
+On **Linux**, `test-leak` runs the regular `test-runner` with `ASAN_OPTIONS=detect_leaks=1` to
+activate LSan.
+
+In both cases the full report is written to `build/c/leak-report.txt`. A clean run ends with:
+```
+Process NNNNN: 0 leaks for 0 total leaked bytes.
+```
+
 ## Run Linter
 
 ```bash
 scripts/lint.sh
+make -f Makefile.cbm lint-source-safety
 ```
 
-Runs clang-tidy, cppcheck, and clang-format. All must pass before committing (also enforced by pre-commit hook).
+Runs clang-tidy, cppcheck, and clang-format. `lint-source-safety` runs the source guard and its
+self-tests; it blocks new MCP stdout writes, insecure string APIs, raw env/filesystem calls in
+reviewed paths, and other regressions that should use existing CBM helpers instead. All must pass
+before committing (also enforced by pre-commit hook).
 
 ## Run Security Audit
 
@@ -55,7 +137,7 @@ src/
   foundation/       Arena allocator, hash table, string utils, platform compat
   store/            SQLite graph storage (WAL mode, FTS5)
   cypher/           Cypher query → SQL translation
-  mcp/              MCP server (JSON-RPC 2.0 over stdio, 14 tools)
+  mcp/              MCP server and tools (JSON-RPC 2.0 over stdio)
   pipeline/         Multi-pass indexing pipeline
     pass_*.c        Individual pipeline passes (definitions, calls, usages, etc.)
     httplink.c      HTTP route extraction (Go/Express/Laravel/Ktor/Python)