Index store: one shared store replaces the trigram, SQLite call-graph and views indexes (0.64)

.alfonso/release-notes/next/callgraph-edit-memory.md

@@ -0,0 +1 @@
+- After a broad edit, AFT's index store keeps the call-graph files it re-resolved encoded until the changes are published, so a busy worktree's call graph holds about a tenth of the memory it did (on AFT's own repository, typically about 120 MB instead of 1 GB).

.alfonso/release-notes/next/import-old-embeddings.md

@@ -0,0 +1 @@
+- Meaning search's first fill after upgrading reuses the vectors the previous meaning index already computed for unchanged files with the same embedding model, instead of embedding them again.

.alfonso/release-notes/next/index-store-bounded-segments.md

@@ -0,0 +1 @@
+- The index store no longer piles up files while worktrees keep editing, and AFT holds at most 1,024 index files open, so busy daemons stay well below macOS's open-file limit.

.alfonso/release-notes/next/index-store-threads-and-edits.md

@@ -0,0 +1 @@
+- AFT's index store resolves every edited worktree's call graph on one shared set of threads instead of a set per worktree (a daemon with dozens of edited worktrees ran hundreds of resolver threads), and write, edit and apply_patch no longer parse the edited file before returning: search sees the edit at once, and the call graph parses it when it next refreshes.

.alfonso/release-notes/next/index-store-worktree-start.md

@@ -0,0 +1 @@
+- A new worktree of a repository AFT already indexes is ready in seconds instead of re-reading and re-indexing every file, which removes the CPU spikes when agents open worktrees.

.alfonso/release-notes/next/prune-legacy.md

@@ -0,0 +1 @@
+- `npx @cortexkit/aft cache prune-legacy` lists the index folders AFT used before 0.64 and their sizes; add `--yes` to delete them and free the disk space. AFT does not delete them on its own in this release.

.alfonso/release-notes/next/refuse-old-config-names.md

@@ -0,0 +1 @@
+- AFT now refuses configuration names retired in 0.62 or earlier, with `removed_config_key`: `tool_surface`, `hoist_builtin_tools`, top-level `enabled`, `search_index`, `semantic_search`, `callgraph_store`, `github.enabled`, `gh_read`, `gh_shim.enabled`, `sandbox.enabled`, `idle.lsp_ttl_minutes`, `inspect.tier2_soft_deadline_ms`, `inspect.max_drill_down_items`, the flat `experimental_*` keys, the `experimental.bash` block, and `aft_`-prefixed names in `disabled_tools`. Run `npx @cortexkit/aft doctor --fix` to rewrite them; AFT still rewrites your user config automatically when it can.

.github/workflows/release.yml

@@ -177,10 +177,10 @@ jobs:
         # fallback silently masks the version-mismatch bug (we try to publish
         # vN, it already exists, we say success even though we wanted vN+1).
         #
-        # `agent-file-tools` depends on `aft-tokenizer` and `aft-secret-rules`
-        # by path; cargo refuses to publish a crate whose path-only dep has no
+        # `agent-file-tools` depends on `aft-tokenizer`, `aft-secret-rules` and
+        # `aft-store` by path; cargo refuses to publish a crate whose path-only dep has no
         # `version =` or isn't on crates.io. All three crates therefore declare
-        # versions (synced by version-sync.mjs) and we publish the two leaf
+        # versions (synced by version-sync.mjs) and we publish the leaf
         # crates first so `agent-file-tools` can resolve them on crates.io.
         # scripts/lib/test_native_release_workflow.py fails when a new path
         # dependency is added without a publish line here. The `--no-verify` flag on the
@@ -204,6 +204,8 @@ jobs:
             || { ec=$?; cargo publish --package aft-tokenizer --dry-run 2>&1 | grep -q "already exists" && true || exit $ec; }
           cargo publish --package aft-secret-rules \
             || { ec=$?; cargo publish --package aft-secret-rules --dry-run 2>&1 | grep -q "already exists" && true || exit $ec; }
+          cargo publish --package aft-store \
+            || { ec=$?; cargo publish --package aft-store --dry-run 2>&1 | grep -q "already exists" && true || exit $ec; }
           # Wait briefly for the new leaf crate versions to become visible on
           # crates.io's sparse index before publishing `agent-file-tools` —
           # otherwise the verify step in the second publish can race against
@@ -821,7 +823,7 @@ jobs:
       # Cargo's local registry overlay, without needing unpublished siblings on
       # crates.io. Stable Cargo >=1.90 supports this selection mode.
       - name: Rehearse crates.io publishing
-        run: cargo publish --dry-run --locked --allow-dirty -p aft-tokenizer -p aft-secret-rules -p agent-file-tools
+        run: cargo publish --dry-run --locked --allow-dirty -p aft-tokenizer -p aft-secret-rules -p aft-store -p agent-file-tools
 
       # Mirrors the inline 10 MiB check in scripts/release.sh (crates.io package
       # sizes). A publish dry run does not enforce the registry's upload limit.
@@ -839,7 +841,7 @@ jobs:
           # Multi-package publish stages archives in tmp-crate and tmp-registry;
           # single-package packaging leaves them directly in the package folder.
           crates = sorted(crate_dir.rglob("*.crate"))
-          expected = {"aft-tokenizer", "aft-secret-rules", "agent-file-tools"}
+          expected = {"aft-tokenizer", "aft-secret-rules", "aft-store", "agent-file-tools"}
           filenames = {crate.name for crate in crates}
           found = {
               package["name"] for package in metadata["packages"]

ARCHITECTURE.md

@@ -95,38 +95,43 @@ boundaries. Process-wide heavy builds share slots through
 `cold_build_limiter.rs`, and `build_breaker.rs` suspends a build domain that
 keeps crashing.
 
-**Indexes.** Three background indexes, each on by default (`indexes.*` in
-config):
-- *Trigram* (`search_index.rs`): a disk-backed postings file read with
-  positioned reads, serving indexed `grep`/`glob` and the lexical lane of
-  `aft_search`. `grep_executor.rs` falls back to a bounded walk while the index
-  is building or unavailable.
-- *Semantic* (`semantic_index.rs`): dense embeddings from fastembed (local
-  ONNX), an OpenAI-compatible endpoint, Ollama, or Synapse over subc
-  (`synapse_embed.rs`). `aft_search` blends lexical and semantic lanes
-  (`commands/semantic_search/`, `search_b2/`) and degrades to lexical when a
-  query embedding times out.
-- *Callgraph store* (`callgraph_store/`): a persisted SQLite graph of call
-  edges behind `aft_callgraph` and zoom enrichment. Each build writes a new
-  generation file and swaps a `.current` pointer, so readers never see a
-  half-built graph. `callgraph.rs` keeps the lighter per-file call data.
-
-Artifacts live in root-keyed cache directories under the storage root. The
-first checkout of a repository to claim `owner.json` (`artifact_owner.rs`)
-becomes the writer; `root_cache.rs` adds per-domain writer leases, reader
-markers, and publish epochs so a superseded worker cannot publish stale data.
-Another live checkout of the same repository, typically a linked worktree,
-runs **borrow-only**: it opens the owner's search and semantic indexes through
-read-only openers (`readonly_artifacts.rs`) and the callgraph through
-`ReadonlyCallGraphStore`, and never builds or repairs them. With
-`worktree.ram_overlay` (on by default), a borrow-only checkout first compares
-the borrowed search index with its own files, then layers its own edits into
-private in-RAM search deltas. Path-listing answers (glob, `aft_search`) also
-drop any returned path that is missing on disk and report the count, as a guard
-against a stale index. Content-addressed views (`views/`, `blob_store/`,
-`pins/`, `gc/`, `refresh/`; `views.enabled`, off by default) instead assemble
-each checkout's semantic and callgraph artifacts from per-file blobs behind an
-atomic manifest. User-configured standing roots (`index.roots`,
+**Index store.** Text search, the call graph and meaning search live in one
+index store per repository (`indexes.trigram`, `indexes.callgraph`,
+`indexes.semantic`; each on by default). `crates/aft-store` owns the formats:
+immutable segment files of checksummed sections, read with positioned reads
+through one process-wide block cache, with at most 1,024 segment files open at
+once. A segment can hold text units with trigram postings, content records
+(each file's parsed symbols, references and outline, keyed by content,
+language and extractor version), call-graph fragments, and vectors (int8 rows
+scanned, f32 rows for rescoring). Worktrees and clones of one repository share
+a store under `storage/data/<family>`, keyed by the repository's root commits;
+each checkout keeps its own state file naming its paths and their units. A
+manifest is the commit record of each publication, readers lease the
+generation they read, and cleanup never removes a file a live lease or the
+newest manifest names.
+
+`data_layer/` is AFT's side. A checkout binds to its published state and
+rescans against disk before serving. A new worktree starts from the sibling
+checkout whose state differs least and processes only the files that differ;
+farther than a quarter of its paths, it builds cold, still reusing stored
+records. Builds, refreshes and meaning-search fill run in small process-wide
+pools (`data_layer/runtime.rs`); under the daemon the root's executor admits
+each job, so an unbound root starts nothing. A first build publishes text
+before the call graph, so `grep` is ready in about a second. Edits update text
+at once; the call graph refreshes after two quiet minutes, or at once when a
+call-graph reader asks (it waits up to 30 s, then reports building). The
+incremental engine (`data_layer/incremental.rs`) re-resolves only files whose
+recorded inputs changed, and changes are published as small segments once
+enough accumulate. The resolver itself (`callgraph_store/`: extraction,
+binding, dispatch) is shared with the gates that compare incremental results
+with cold builds. `aft_search` blends lexical and meaning lanes
+(`commands/semantic_search/`, `search_b2/`); embeddings come from fastembed
+(local ONNX), an OpenAI-compatible endpoint, Ollama, or Synapse over subc
+(`synapse_embed.rs`, chunking in `semantic_index.rs`). A plane that is still
+building says so: `grep` and `glob` walk the folder until text is ready, and
+call-graph tools answer "building" rather than from an old graph.
+Parent-folder sessions (`views/parent.rs`) answer from each child
+repository's checkout. User-configured standing roots (`index.roots`,
 `standing_roots.rs`, `scoped_key.rs`) are indexed by the subc standing actor
 independent of sessions.
 
@@ -272,11 +277,12 @@ ledger, access grants, standing roots, GitHub cache, breaker rows) lives in
 | Command handlers | `crates/aft/src/commands/` |
 | Executor | `crates/aft/src/executor/`, `cold_build_limiter.rs`, `build_breaker.rs` |
 | Runtime state | `context.rs` (`AppContext`) |
-| Trigram search | `search_index.rs`, `grep_executor.rs` |
-| Semantic search | `semantic_index.rs`, `synapse_embed.rs`, `commands/semantic_search/`, `search_b2/` |
-| Callgraph | `callgraph_store/`, `callgraph.rs`, `calls.rs` |
-| Artifact ownership and borrowing | `artifact_owner.rs`, `root_cache.rs`, `readonly_artifacts.rs`, `legacy_partitions.rs` |
-| Index views | `views/`, `blob_store/`, `pins/`, `gc/`, `refresh/`, `path_status/`, `alias/`, `migration/` |
+| Index store formats | `crates/aft-store/` |
+| Index store in AFT | `data_layer/` (checkout, runtime, incremental, meaning, `tools/` adapters) |
+| Text search | `grep_executor.rs`, `data_layer/tools/grep.rs` |
+| Semantic search | `commands/semantic_search/`, `search_b2/`, `synapse_embed.rs`, `semantic_index.rs`, `data_layer/vectors.rs` |
+| Call graph | `callgraph_store/` (resolver), `data_layer/callgraph.rs`, `callgraph.rs`, `calls.rs` |
+| Parent-folder sessions | `views/parent.rs`, `data_layer/parent.rs` |
 | Standing roots | `standing_roots.rs`, `scoped_key.rs`, `subc/standing.rs`, `db/standing_roots.rs` |
 | Parsing and symbols | `parser.rs`, `symbols.rs`, `symbol_cache_disk.rs`, `language.rs`, `imports/` |
 | Edit, patch, hashline | `edit.rs`, `format.rs`, `patch/`, `hashline/`, `fuzzy_match.rs` |
@@ -355,7 +361,6 @@ The canonical field list is the `Response` doc comment in
 - [docs/gh-shim.md](docs/gh-shim.md): `gh` shim routing and refusals
 - [docs/health-digest.md](docs/health-digest.md): the `health.digest` operation
 - [docs/memory-census.md](docs/memory-census.md): memory attribution
-- [docs/design/content-addressed-index-views.md](docs/design/content-addressed-index-views.md): index views
 - [docs/design/subc-readiness-warmup.md](docs/design/subc-readiness-warmup.md): subc readiness and warmup
 - [docs/design/opencode-v2-plugin-playbook.md](docs/design/opencode-v2-plugin-playbook.md): OpenCode V2 plugin
 - [docs/investigations/watcher-backends-reference.md](docs/investigations/watcher-backends-reference.md): watcher backends

Cargo.toml

@@ -2,6 +2,7 @@
 resolver = "2"
 members = [
     "crates/aft",
+    "crates/aft-store",
     "crates/aft-tokenizer",
     "crates/aft-fake-lsp",
     "crates/secret-rules",

STRUCTURE.md

@@ -8,6 +8,7 @@ it stays current and this document does not list them.
 
 ```text
 crates/aft/            Rust engine: the `aft` binary, command handlers, indexes, integration tests
+crates/aft-store/      Index store formats: segments, family lifecycle, text, content records, call graph, vectors
 crates/aft-tokenizer/  Standalone Claude token-counting library
 crates/secret-rules/   Secret placeholder grammar, key location rules, response scrubbing
 crates/aft-fake-lsp/   Fake language server used by LSP tests
@@ -27,45 +28,39 @@ assets/                Repository images
 ## `crates/aft/src/`
 
 Top-level `.rs` files are shared engine modules (parser, edit, config,
-context, search and semantic indexes, and so on). A folder with the same name
+context, search and semantic search, and so on). A folder with the same name
 as a top-level file holds that module's submodules (`edit/`, `backup/`,
 `run_tool_call/`, `sandbox_spawn/`, `subc_translate/`, ...). The other folders:
 
 ```text
 agent_data/        Session undo and checkpoint data carried through engram's session door
-alias/             Git blob-id aliases and byte-exact manifest path identities
 bash_background/   Background, PTY and remote task registry, watchdog, pattern watches, persistence
 bash_db_hints/     Schema hints when a database CLI fails on an unknown table or column
 bash_permissions/  Permission scan of parsed bash commands
 bash_rewrite/      Rewrites simple bash commands into AFT tool calls
 bin/               Auxiliary binaries (hashline schema artifact generator)
-blob_store/        Content-addressed per-plane blob storage for index views
-callgraph_store/   Persisted SQLite callgraph: build, refresh, queries, dead-code projection
+callgraph_store/   Call-graph resolver: extraction, binding, dispatch, facts, query contract, dead-code projection
 cli/               Native `aft` subcommands (setup, index, warmup, profile, fix-config, backups, cache cleanup, sandbox launch)
 commands/          One handler per protocol command; `tool_call.rs` is the agent tool entry
 compress/          Bash output compressors, TOML filter engine, builtin filters, filter trust
+data_layer/        AFT's side of the index store: checkouts, runtime pools, incremental call graph, meaning fill, tool adapters
 db/                aft.db stores (backups, bash tasks and watches, call ledger, access grants, GitHub cache, ...)
 exec_remote/       Client for the daemon's remote execution capability (`runon`)
 executor/          Per-root actor scheduler, lanes, job classes, cancellation
-gc/                Mark-and-sweep for view blob stores
 gh_shim_relay/     Daemon-side relay for governed `gh` bot writes
 gh_shim_speech/    Launch-bound bot-write authority
 github_read/       `issue://` and `pr://` fetch, normalize, render, fallback cache
 hashline/          Hashline edit engine: scan, snapshot, syntax, apply, transaction, recovery, oracle
 imports/           Per-language import engines for `aft_import`
 inspect/           `aft_inspect` categories, scanners, Oxc liveness engine, tier-2 scheduler
 lsp/               Language-server client, registry, roots, diagnostics, child processes
-migration/         One-way import of legacy semantic snapshots into views
 page_collect/      Page and referenced-file snapshots for sealed HTML page bundles
 patch/             `apply_patch` parser, matcher, applier
-path_status/       Pending or failed path annotations for view generations
-pins/              Generation pins held while a view is assembled or read
-refresh/           Plane-worker refresh coordination for views
 search_b2/         `aft_search` query router, lane plans, token variants, readiness
 secret/            Secret vault seam and its consent adapter
 secret_proxy/      Command-private proxy that releases resolved secrets to one command
 subc/              Subc daemon edge: binds, trust, tool-provider role, scopes, code mode, consent, push, drain, health
-views/             Per-checkout view manifests and generation publication
+views/             Manifest path types and parent-folder sessions
 watcher/           Project file watcher
 watcher_backend/   Native watcher backends (FSEvents, inotify, Windows)
 ```
@@ -121,6 +116,8 @@ tests are `*.test.ts`.
 - **List truncation surface:** `crates/aft/src/list_surfaces/[surface].rs`, registered in `list_surfaces.rs`.
 - **Build breaker domain:** `crates/aft/src/build_breaker.rs`.
 - **Semantic embedding backend:** `crates/aft/src/semantic_index.rs`, `synapse_embed.rs`, and the backend config in `config.rs`.
+- **Index store format (a section, record or lifecycle rule):** `crates/aft-store/src/`. How AFT builds, refreshes or reads it: `crates/aft/src/data_layer/`; a tool's read path: `data_layer/tools/`.
+- **Call-graph resolution rule:** `crates/aft/src/callgraph_store/` (dispatch, bindings, facts); bump `EXTRACTOR_VERSION` in `data_layer/content.rs` when stored records change.
 - **Standing root keys:** `scoped_key.rs` and `db/standing_roots.rs`.
 - **Windows path normalization:** `crates/aft/src/windows_path.rs`.
 - **Bridge export:** `packages/aft-bridge/src/[module].ts`, re-exported from `src/index.ts`.