Skip to content

Index store: one shared store replaces the trigram, SQLite call-graph and views indexes (0.64) - #1405

Open
aft-alfonso[bot] wants to merge 432 commits into
mainfrom
land/index-store-0.64
Open

aft-alfonso[bot] wants to merge 432 commits into
mainfrom
land/index-store-0.64

Conversation

@aft-alfonso

@aft-alfonso aft-alfonso Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

This is the 0.64 index-store cutover. Text search, the call graph and meaning search move to one index store per repository, shared by its worktrees and clones; the old trigram index, SQLite call-graph store and content-addressed views are deleted (about 215,000 lines net).

What changes for users

  • Each repository builds its new indexes once on first use: text in about a second, the call graph in under a minute for large projects. Worktrees of one repository share them, and a new worktree starts from a sibling's indexes and processes only the files that differ.
  • Meaning search reuses old vectors for unchanged files embedded with the same model.
  • Old index folders stay on disk; npx @cortexkit/aft cache prune-legacy lists and removes them.
  • views.enabled, views.callgraph_copies and worktree.ram_overlay show a removal notice; views.idle_checkout_hours becomes idle.checkout_hours. Config names retired in 0.62 or earlier are refused in user config and load with a notice in project config.
  • trace_to and impact label each root (main, test, attribute, public API, no caller found).

Measured on AFT's own repository (7,300 files)
Call-graph build 207 s to 17 s; index disk 1.7 GB to 0.3 GB; typical edit catch-up 4.7 s to 0.1 s; dead code first run 72 s / +4.25 GB to 6-8 s / +0.8 GB; an edited worktree's call graph median 3 MiB between publications; a new worktree at the same commit under a second instead of about 160 CPU-seconds.

Verification

  • Full Rust suite on Linux: all pass except the 12 native-secret-shell tests and one rust-analyzer test the build server can't run (both fail the same way on main).
  • Incremental call graph equals a cold build after each of 430 real edits (edit, reuse, extract and checkout gates).
  • Dead-code findings unchanged across a 91-project corpus; search-quality benchmark rows byte-identical except five declared pattern-lane candidate counts (land-index-store-0.64.json).
  • macOS and Windows compile with warnings as errors.

The release notes are in review; the version bump and tag follow through scripts/release.sh after this merges.


Summary by cubic

This is the 0.64 index-store cutover: text search, the call graph and meaning search move to one index store per repository, shared by its worktrees and clones; the old trigram index, SQLite call-graph store and content-addressed views are deleted (about 215,000 lines net).

What changes for users

  • Each repository builds its new indexes once on first use; worktrees share them, and a new worktree starts from a sibling's indexes and processes only the files that differ.
  • Meaning search reuses stored vectors for unchanged files embedded with the same model, and a repository is filled by one process at a time so no chunk is embedded twice.
  • Editing a config file (Cargo.toml, tsconfig.json) rebinds only the files it changed, not every file that reads any config.
  • A repository whose index-store files a newer build wrote is refused with storage_requires_newer_reader, naming the file and version, instead of being rebuilt over them.
  • Old index folders stay on disk; npx @cortexkit/aft cache prune-legacy lists and removes them.
  • views.enabled, views.callgraph_copies and worktree.ram_overlay show a removal notice; views.idle_checkout_hours becomes idle.checkout_hours, and config names retired in 0.62 or earlier are refused in user config.
  • trace_to and impact label each root: main, test, attribute, public API, or no caller found.

Measured and verified

  • On AFT's own repository (7,300 files): call-graph build 207 s to 17 s, index disk 1.7 GB to 0.3 GB, typical edit catch-up 4.7 s to 0.1 s, new worktree at the same commit under a second instead of about 160 CPU-seconds, and a busy worktree's call graph holds about a tenth of the memory after a broad edit (about 120 MB instead of 1 GB).
  • The store keeps at most 1,024 index files open, so busy daemons stay below macOS's open-file limit.
  • Incremental call graph equals a cold build after each of 430 real edits; dead-code findings unchanged across a 91-project corpus; search-quality benchmark rows byte-identical except five declared pattern-lane candidate counts.
  • Replaying 6,956 recorded callers/call_tree/trace replies through the new store matches main's corpus except 34 reviewed differences (C++ type-matched method calls and macro-template calls now resolve).
  • Full Rust suite passes on Linux except the 12 native-secret-shell tests and one rust-analyzer test the build server can't run (both fail the same way on main); macOS and Windows compile with warnings as errors.

Written for commit da21c65. Summary will update on new commits.

View guided diff Turn on auto-fix

ualtinok and others added 30 commits November 5, 2026 23:14
…route

The test took today's answers through the views route (children publishing
v3 views and a legacy trigram artifact) and compared the data layer's against
them; with the data layer on by default those indexes are never built, so its
first wait never ended. It now binds only the parent folder and asserts the
data layer's answers against what the fixture's files hold: the six COMMON
files of alpha, beta and gamma, BARE in the 59 bare folders within the cap,
their six Rust files for the glob, alpha's caller and callee, and the named
gaps for building, failed, the capped children and README.md. The timing
comparison and the inspect comparison had nothing left to compare and are
gone, as are the helpers only today's half used.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
views_on_back_to_back_configures_leave_one_worker pinned the views semantic
lane's worker count and served model fingerprint, and
views_on_semantic_member_stores_no_call_graph_database pinned which files the
v3 views members under blobs/v3 hold. Both are internals of the views store the
data layer replaces. The other per_checkout_semantic_runtime tests check
answers or embedding cost and are left for the meaning-search port.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…standalone binds at configure

Three defects the cutover test port found:
- A watcher that lost events (RescanRequired) reset only today's indexes, so
  edits in the lost batch never reached the data layer. Each bound checkout,
  the root's and a parent folder's children, now rescans.
- An ignore file that ignores itself is dropped from the watcher's changed
  paths and reported only as an ignore-rule change, which never reached the
  data layer. That change now records the ignore files, which rescans their
  folders.
- A standalone configure bound the data-layer checkout only if it left a
  pending configure step; otherwise the first query did. The standalone loop
  now binds after each successful configure, as the daemon's tail does.
The three reproduction tests are no longer ignored.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Unit tests (data_layer/publication_tests.rs): an engine started from a
publication (from_resolved_timed over stored slices), one over a newer
segment another checkout of the same files wrote, one started from a
publication of the step before, and one started cold are each compared
with a cold resolution after every step: files, rows, sites, site
dependencies, dispatch, recorded reads, fragment bytes and
stale_recordings(). Sequences: the audit's scenarios (symlink chains, a
folder that becomes a symlink, name-matched methods, Go methods and
interfaces, Rust modules, a Python module that becomes a package, tsconfig
paths), plus TypeScript, Rust and Go sequences of a content change, an
added, renamed and removed file, a symlink retarget, a config field change
and a folder added and removed. Every step of those three must change a
stored fragment of a file it did not edit (the Go symlink step is exempt:
the Go resolver follows no symlinked package folder or go.mod).

Checkout gate (data_layer/checkout_gate.rs, examples/checkout_gate.rs):
- two checkouts of one repository in one family store (Git repositories
  with the same root commit). A replays the pack through Checkout; B runs
  some commits behind through the runtime (DataCheckout::open_planes,
  note_paths, fresh_view), opened over a publication of its text index
  alone (publish_text_first), and restarts once;
- A publishes its changes with publish_changes every 8 edits and every
  third commit end, is built again whole once publish_changes reports more
  than 8 segments, is built again under a recording watcher with a file
  changed unrecorded, restarts with that commit's edits made while down,
  and restarts once more over B's newer fragments with only a probe
  symlink moved;
- at every commit end each checkout is compared with a cold build of its
  folder: sampled primitives and text candidates as before, plus the
  engine file by file through Checkout::inspect_engine: paths, parse keys,
  configuration, answers, recorded reads, fragment bytes, and
  stale_recordings(). After a publish A is compared again, and so is a
  second handle bound fresh over the store. check_stored_inputs runs after
  both restarts. A probe folder (a symlink chain that differs between A and
  B, and a .gitignore whose rule moves) is checked to move what it must,
  so those checks cannot pass vacuously. A comparison of zero files fails.

Checkout::inspect_engine is a new gate-only accessor: it runs a closure on
the engine the call graph is served from, or on one started from the
publication as the next refresh would.

gate-all.sh runs the checkout, grep/glob and inspect gates as stages and
unsets the settings that make a gate check less (EDIT_GATE_TIMING_ONLY,
AFT_GATE_ALLOW_DIFFS, AFT_GATE_CASES, CHECKOUT_GATE_ONLY and others). A
stage that compares nothing fails: the checkout stage needs comparisons
with files, the grep stage a report whose replays have cases, and the
parity gate parsed files, rows and queries.

The grep/glob gate's stale-path class keeps `complete`: a data-layer reply
that claims complete: true where today's discloses a stale index is a new,
failing class. edit_gate with EDIT_GATE_TIMING_ONLY=1 says it checked
nothing and exits 3. inspect_gate.py exits 2 with no --root or --corpus
checkout, and fails a run that compared no reply.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
With the data layer each child of a parent folder is named as loading only
until its own checkout publishes, then answered. The test pinned the views
route, where children stayed unavailable, so its count depended on how many
children had published by the time of the query.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
A call written directly in a macro_rules! template (its caller is the
macro, name!) was assembled as an ordinary call reference and edge, so
callers listed the template as a plain caller and the callers command's
macro lookup (ref_sites_named("macro_body_call")) found nothing. The
slices and graph rows now store it as today's database did: reference
kind macro_body_call naming the macro in local_name, and an edge of the
same kind, which caller queries leave out. The callers command then lists
the template and each invocation of the macro, name-matched and marked
via the macro.

Test: callgraph_test::callgraph_rust_macro_template_call_lists_invocations_via_macro
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
The extractor found identifiers inside Rust macro token trees that do not
parse as Rust (FileCallData.macro_mentions_by_symbol), but the parse output
the data layer stores dropped them, so callers of such a name had no
macro_note. They are now references of a new kind, MacroMention (RefKind in
aft-store's content records, BlobRefKind in the parse output), stored in
the file's record, rebuilt into FileCallData, bound as macro_mention
references that never resolve, and kept in slices and graph rows under
that kind, where the callers command's ref_sites_named("macro_mention")
finds them.

The record version suffix goes to +record-2 so every record is extracted
again. The new kind codes are appended (content RefKind code 8, fragment
reference kind code 8), so existing records and fragments still decode.

Test: callgraph_test::callgraph_rust_unparseable_macro_mention_is_noted
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
… defines

A macro_rules! template that defines several functions, stamped into an
impl by invoking the macro, has them call each other through self. calls.rs
extracts those calls by parsing the template, but the template is tokens in
the file's tree, so dispatch extraction found no call node and made no
method-call site, and ordinary resolution cannot bind self without an
owning impl: the callee had no callers. Today's database linked such a call
by name (name_match) to the unique same-named function in the project.

Dispatch extraction now makes a site for self.name(...) written inside a
function the template defines when the same template defines exactly one
other function name; the site's receiver (@template-self:<symbol>) names
that function, and the resolver links it with name_match provenance. The
site is named by the reference's key, so calls sharing a token-tree
ordinal keep their own sites. Receiver type binding skips such receivers.
Unlike today's project-wide name match, only the template's own function
qualifies. Record version suffix +record-3.

Test: callgraph_test::callgraph_rust_methods_defined_in_macro_template_have_callers
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…s store

Dispatch extraction marked every JavaScript and TypeScript computed member
call dynamic. Its site resolved to nothing and, being a method-call site,
also kept ordinary resolution from linking the call, so obj["method"]()
left method without its caller. Today's database had no site for it and
resolved the reference (callee name method, taken from the string literal)
as an ordinary call by that name.

A computed member whose index is a string literal is no longer dynamic and
gets no site, so ordinary resolution links it as before; any other index
stays a dynamic site. calls.rs's extract_computed_member_name, which names
the callee, is shared for the check. Record version suffix +record-4.

Test: callgraph_test::callgraph_computed_member_call_uses_static_property_name
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Callers of a C++ method found none on the data layer: dispatch extraction
had no C++ member functions and no C++ call sites (a C++ field_expression
names its object argument, which the generic receiver lookup does not
read), and ordinary resolution cannot bind widget.value(). Today's
database inferred the receiver's type from its declaration in the
enclosing function and linked the one member function whose scoped name
ends in Type::member (type_match).

Dispatch extraction now records C++ method and function symbols with a
scoped owner as method facts, and a site for receiver.member(...) when the
receiver is a plain name whose declaration names its type, read with the
same scan today's database uses (infer_cpp_receiver_type_from_scope). The
resolver matches that site among same-named C++ methods by scoped-name
suffix, leaving out the caller, through methods_named, which records the
method-name dependency. One match links with type_match provenance; the
memo keys C++ answers by caller too. The fragment's provenance codes gain
type_match (appended). receiver->member(...) keeps ordinary resolution,
which can link it by name, and calls with an undeclared receiver keep no
site. Record version suffix +record-5.

Test: outline_languages_test::h_header_cpp_outline_zoom_and_method_callers
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Comments in the method-call extraction and resolver referred to an
unnamed earlier database; they now say what they do, naming the SQLite
call-graph store where its behaviour is the reference.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
with_the_data_layer_off_every_surface_reports_todays_stores pinned the
AFT_DATA_LAYER=0 route, which goes away at cutover.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…nied flag

write_denied_search_index_reports_ready_not_building and
in_progress_search_index_still_reports_building set SearchIndex.ready and
build_denied directly. The data layer has no such index; its building and
unavailable states on health are covered by data_layer::surfaces tests
(status_health_and_zoom_follow_the_checkout_through_each_state,
a_store_that_cannot_open_reports_why_on_status_and_health).

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Deleted, as they pinned stores the data layer replaces:
- view_health_* (five tests and their fixture): the views runtime's
  path-status health ledger and its rollup projection.
- disabled_callgraph_store_reports_disabled_despite_lingering_build_receiver:
  the legacy SQLite call-graph store's build receiver lingering across a
  disabling configure.

Ported: tier2_first_scan_keeps_root_in_warming_rollup now expects the
data layer's word for a switched-off text index and call graph, "off"
(the shared consumer reply policy's word), where today's indexes said
"disabled".

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
exact_memo_publication_retires_stale_sets_but_keeps_warm_hits installed
legacy SearchIndex publications through search_index_rx. Its port,
exact_memo_is_retired_by_recorded_edits_but_keeps_warm_hits, records the
edits with the data layer and checks the same answers and memo counts.

a_watcher_rescan_reaches_the_data_layer is ignored: it shows that a
watcher rescan never reaches the data layer's checkout.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
The views-routed checkout test is ported to the data layer's checkout:
use inside the idle window keeps it bound, an in-flight call keeps it,
the reaper then releases it, and the next grep binds it again and
answers from the publication. The v3 view-session ledger poll and the
checkout-view retention sweep it also pinned are dropped with the views.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ayer

subc_bridge_health_check_returns_root_status_report: a switched-off text
index and call graph report "off" (was "disabled").

subc_bridge_callgraph_maintenance_is_per_root: the legacy cold-build
spawn counter is gone with the SQLite store; cold callers now wait for
the data layer's build and answer with the built result instead of
callgraph_building.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…a layer

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…tops its quiet timer

After eviction each worked root kept one aft-data-quiet thread, asleep
until the call graph's quiet period (120 s) after its last recorded
change. The fix is in data_layer/runtime.rs on the data-layer branch; the
test is unchanged and comes back once that fix is merged.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
aft_zoom_callgraph_matches_the_shared_fixtures stubbed legacy build
receivers; its scenarios now put the root's data-layer checkout in each
state: indexes off (never bound), not bound yet, and a bind in progress
(building and partially_ready; zoom does not read the text plane).

The frozen aft_zoom.callgraph/unavailable row now holds what the data
layer gives for an enabled but unbound call graph: code
callgraph_unavailable with index {status: off, reason: not_bound}, where
it held index {status: unavailable, reason: runtime_not_observed}.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
future_search_index_is_refused_by_name_and_left_byte_identical planted a
newer legacy trigram cache.bin, which the data layer never reads. Its
port plants a manifest from a newer record format in the data layer's
family store and checks the same rule: the bind refuses it, no
generation, segment or manifest is written or removed, and status reports
both planes unavailable (bind_failed) with an error naming the file and
its version.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
disabled_callgraph_store_reports_disabled_despite_lingering_build_receiver
is ported rather than dropped: the lingering legacy build receiver goes,
but the property it checked under three forced scheduler collisions
stays. A root bound in the data layer with the call graph switched off
reports callgraph_store {status: off}, never building, and is not
degraded.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
A grep before the data layer published fell back to today's search
index and started a build of it that nothing drained, which then blocked
the idle release as a build in flight. The test now binds and waits for
the publication first, and names the blocker if the release stalls.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
The text plane is published before the call graph, and the call graph's
publication changes the view's text generation, so the exact-memo test
saw two verifications for one query when it searched between the two.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
A watcher rescan reaches the data layer's checkouts on the data-layer
branch now, so a_watcher_rescan_reaches_the_data_layer is no longer
ignored.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…the data layer

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
force_off_for_test lost its last caller with the tests of the
AFT_DATA_LAYER=0 route, and the -D warnings Windows check failed on it.
The thread-local is a plain on switch again, as force_on_for_test sets
it. Test-only code.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…mparison

A watcher rescan now reaches the data layer, which compares the
checkout's files with its state on a background run, so the test polls
grep for the file (up to 30 s) the way overflow_control polls through
subc.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
ualtinok and others added 30 commits November 7, 2026 23:02
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>

# Conflicts:
#	crates/aft/tests/callgraph_query_bench.rs
#	crates/aft/tests/callgraph_store_test.rs
The retired SQLite call-graph store's stand-down flag, a test-only syntax
tree accessor and three empty test modules had no remaining users after the
old store's code was deleted. The file-identity tests import rusqlite's
Connection only where their POSIX-only tests use it.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
… the deletion

Search results again carry each symbol hit's caller count and caller files
("↩N a.rs,b.rs"), now read from the index store's current call graph; a
search never waits for a refresh or starts one. They were dropped with the
retired call-graph store.

Tests: three fixtures read the checkout path at run time (the build-path
guard), the configure migration test follows the project-config ruling,
two tests compose call_tree through the adapter as production does, and the
compile-fence pin for the removed exit-index delay hook is gone.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ollow the index store

The call-graph plane's status is observed from the index store, as the text
plane's is; it had been left reading the retired store and so always said
unavailable. The debounced ignore-rule refresh is counted again, the reap
fixture holds two pending sets now, and the exit-phase hook records only the
LSP shutdown, the one phase left. Tests of the retired store's own call-tree
budget, manifest fingerprint and in-process census are gone.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
The never-reaped test checks that the timeout reason names the group's pid,
which comes from each member's Debug output; dropping the field as unread
broke that.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
The index store becomes aft_search's only text and meaning source; ranked rows
must stay byte-identical to the committed reference.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…nges

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
A cold call-graph read could not see the build it asked for. Data-layer
builds and refreshes reach their pool through
RootMaintenanceExecutor::submit_maintenance. That call queued a
Maintenance/MaintenanceCommit job whose only work was to hand the build
to the pool. The scheduler dispatches maintenance only while
general_idle > interactive_reserve. Meanwhile each call-graph read
waits up to 30 s inside DataCheckout::fresh_view on its executor
worker. With the subc bridge test executor (pool 4, so 8 workers counting
the bind and provider reserves, and interactive_reserve 2), the two
callers reads left general_idle = 6 - 2 - 2 = 2. The handoff stayed queued
until both reads gave up and answered callgraph_building.

Production hits the same wait: every read with pending edits asks for
its refresh from inside fresh_view. With default pools it takes one
concurrent read on pool 2, two on pools 3-4, and six busy general
workers on pool 8.

submit_maintenance now admits synchronously under the scheduler lock.
It keeps the refusals the worker path applies to a maintenance job: the
actor is not registered or failed, the root is unbound, the root folder
is missing, or the token is cancelled. It then runs the handoff on the
calling thread with the root+lifecycle token installed, so a root that
unbinds still cancels the pool job at its checkpoints.

Regression test: executor::tests::data_layer_handoff_runs_while_reads_wait_for_it

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 95c0fb14ba921b3eb3f8ba57efcd6cfa43dcde09)
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>

# Conflicts:
#	crates/aft/src/commands/semantic_search/mod.rs
…stores' deletion

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
The busy-executable spawn test and the daemon heap probe run only on Linux,
so on macOS their helpers looked unused. They now carry an allow for other
targets so the pass leaves them alone.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…tted imports

- A member kept alive by an unknown-receiver call in test code is now a test
  seed, not a production root, on both dead-code read paths (the per-file graph
  and the rows snapshot, which carries the site line as an
  UNKNOWN_RECEIVER_LIVE_PROVENANCE call). Kept dead-code graphs move to header 8.
- service.render() resolves exactly to the object literal's method, so the two
  uncalled exported render functions are reported dead; the test now says so.
- Python: import pkg.mod then pkg.mod.f() no longer records an unresolved
  method-call site for pkg.mod that replaced the resolved module binding.
  Content records move to version 11, so stored records are extracted again.
- Exported methods and types stay dead-code roots for now: dropping that rule
  needs a name rescue for unresolved method calls, which still left false
  findings (an impl in another file than its type, methods reached through a
  re-export). Four tests assert the conservative verdict and name the gap.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…y more

A change publication only ever added a segment. Builds were the only
place a segment left the store, and a checkout that keeps editing the
same files reads few segments, so it was seldom built again: every
publication of its changes stayed, one more open file each. A
publication now retires, as a build does, the segments no checkout
(this one's new state included) reads any more, and the checkouts whose
folder is gone.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit a148f88318d6e706ac0d5fd1fcf4381737e5722a)
Every open segment held its own descriptor for as long as anything held
the segment, so a store whose segments pile up ran a daemon towards
macOS's limit of 10,240 open files. Segments now read their file
through a process-wide table of open files capped at
MAX_OPEN_SEGMENT_FILES. Past the cap, the file read least recently (and
not being read) is closed; its segment opens it again on its next read,
with the same Windows sharing flags, and refuses it as damaged unless it
has the length and header bytes the segment was opened with. Block
checksums still check every byte read.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit b37105ecf7f98b9e6a7726a2c3f94a091c9f945b)
…reading too many segments is built again

Readers take each unit from the newest segment holding it. A change
publication used to write again units other checkouts already read, so
their reads moved into it and an idle checkout kept it in the store
once the writer edited the unit away. A build left out every unit any
segment another checkout reads held, so two checkouts reading one change
segment each left it to the other and neither ever folded it.

- A change publication leaves out what another checkout already reads,
  as a build does.
- A build shares a unit only when another checkout reads that very unit,
  and only from the SHARE_FROM (4) segments serving the most of them; it
  writes everything else again, folding small change segments in.
- After a publication, a checkout is built again when it reads more than
  COMPACT_AT (8) segments only it reads, as before, or more than READ_AT
  (16) in all (DeltaReport::wants_build). The runtime, the checkout gate
  and the memory probe all use that one rule.

A test replays worktrees editing in turn while main takes their changes
in and new worktrees are made from main. Over 60 rounds the store stays
at 14 to 21 segments for 5 checkouts, where it grew by a segment or more
every round. Every checkout answers as a cold build does.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 15e4731b9a5786ccd0e5d14fe4174f7b1fdbebfc)
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit e371d0cb2954d7746068c8a0d03ff20ccd39978c)
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit a907787f2793924854174b292cc031a0a3971f06)
Every new worktree built cold: it read and parsed every file and
resolved the whole call graph on all cores, only for its build to find
nearly every unit already stored (15 to 60 s at 6 to 8 cores for a
large repository, the daemon's CPU spikes).

A checkout with nothing published now hashes its files without parsing
them and starts from the published checkout of the family whose files
differ least: that state, naming this checkout's root, is published for
it as it stands, the paths that differ are read and recorded as changes
as a bind records them, the call graph is refreshed for them, and the
changes are published. Everything the call graph records is keyed by a
path inside the checkout (a symlink by its target's bytes, never
followed out of the checkout), so nothing else can differ. A checkout
differing in more than a quarter of its paths, or with no sibling
published, builds cold as before, and a cold build's scan now reads a
content record the store already holds instead of parsing that file.

a_second_worktree_stores_only_what_no_sibling_holds checked a cold
build's sharing with a worktree differing in one of four files, which
now starts from its sibling instead. Its worktree now differs in two of
four files, so the test still checks what a cold build shares.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 34d8bd251ef3cdd6684ced63d97e257d77a613ab)
Builds three exported checkouts of one repository into one store in
turn (nothing to start from, the same commit, and the same commit with
some files taken from another) and prints each build's wall and CPU
time with its report.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 689bde9304d008903945bddf3e4ed2c9a7c14dcb)
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit cb6ef303c63345b9be514c2b98d302305dbb7b21)
…r first build

At every commit end that publishes, the gate copies A's folder, makes
the next commit's edits in the copy, binds and builds it as a new
worktree (which starts from the closest publication of the family) and
compares it with a cold build of its folder, failing the comparison if
it did not start from a sibling.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit fbc28a437df84f95eaf3ba2d4f531e0251f3968f)
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 0154b287d977b9cf1461d09f1a579777cc3afbe6)
The fixture dropped the guard on return, so a fill round the query started
could send its own embedding request to the fake backend: the request count
came out 3 instead of 2, or the lane was paused before the query saw it
(failed about 1 run in 3 alone; 0 of 12 now).

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…l-graph and views stores

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>

# Conflicts:
#	crates/aft/tests/fixtures/tool_provider_catalog.json
#	crates/aft/tests/fixtures/tool_provider_catalog_presets.json
…ows, stale allowlist entries

- inspect_tier2_reuse tests that build an index-store graph now canonicalize
  their project root: the store binds a checkout at its canonical path, and on
  macOS a temp folder sits under /var, a link to /private/var, so the raw-path
  snapshot named no file the call graph held.
- worktree_start_probe reads CPU time with getrusage only on Unix; Windows
  reports wall time.
- The legacy-vocabulary allowlist drops six entries for test lines the
  deletion removed.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…ne-legacy tests use the plain path

The dead-code checkout registry compared a normalized checkout root with the
caller's root as given and keyed the map by it, so a root spelled with the
Windows verbatim prefix (what std::fs::canonicalize returns there) never found
its checkout and dead code reported 'call graph building' forever. Bind and
every lookup now key by the normalized root.

prune-legacy's lib and CLI fixtures canonicalized their temp folder, which on
Windows adds the verbatim prefix while the command reports and keys the plain
spelling; they now normalize it. The inspect_tier2_reuse integration roots use
canonicalize_like_product for the same reason.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…-legacy fixtures

Product inspect roots are canonical and verbatim-stripped; tests that built
their root with bare std::fs::canonicalize got the Windows \\?\ spelling and
compared it with plain paths. They now use canonicalize_for_snapshot or
canonicalize_normalized. The prune-legacy external-cache fixture normalizes its
temp folder, and the CLI test joins expected paths part by part so Windows gets
its own separators.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant