Ghost stack 2/6 — Surface model build-out (Phases 1–8)#190
Draft
nahiyankhan wants to merge 26 commits into
Draft
Ghost stack 2/6 — Surface model build-out (Phases 1–8)#190nahiyankhan wants to merge 26 commits into
nahiyankhan wants to merge 26 commits into
Conversation
Two regressions from the earlier docs focus pass, surfaced by running the full test suite (the pre-commit hook runs pnpm check, not pnpm test). Remove Ghost's own dogfood .ghost/ package and its verify test: the package cited deleted docs as evidence (fingerprint-evidence-unreachable), and a self-referential fingerprint has been more confusing than useful. Scope terminology-public to genuinely shipped text (README, docs site content, skill bundle, changesets), excluding docs/: the design notes and purposes.md now use 'layers' and 'cascade' as the canonical vocabulary of the surface-model redesign, which the old guard forbade.
Phase 1 of the surface-model cutover (docs/ideas/phase-1-plan.md). Purely additive: a new ghost-core/surfaces/ module mirroring fingerprint/, changing no existing behavior. - schema.ts: Zod for surfaces.yml. Flat-slug ids with the dot excluded, so dotted-id hierarchy is rejected at the schema layer (the tree lives only in parent). Single scalar parent; edge kinds restricted to the fixed Ghost-owned set (composes, governed-by). - types.ts: constants, interfaces, and lint report types reusing the fingerprint facet shape for uniformity. - index.ts + ghost-core/index.ts: re-export under @anarchitecture/ghost/core. - tests: schema accepts a minimal doc and a realistic tree with typed edges; rejects dotted ids, array parents, unknown edge kinds, unknown keys. One case documents that dangling edge refs pass the schema on purpose (a Phase 2 lint concern), so the schema/lint boundary is not crossed in the wrong layer. Graph validation (cycles, dangling refs, reserved core) is Phase 2. No changeset: no user-visible behavior yet.
Add the Phase 2 execution spec (lintGhostSurfaces graph validation + ghost lint dispatch for surfaces.yml; edge cycles allowed, only parent tree-constrained). Also adopt the pre-commit test gate: lefthook.yml now runs just test alongside just check, closing the gap Phase 1 exposed (the check-only hook let two regressions through). Implementation-plan and phase-2-plan process notes updated to reflect that both gates now run automatically.
Phase 2 of the surface-model cutover (docs/ideas/phase-2-plan.md). Additive: adds document-level graph validation and recognizes surfaces.yml in ghost lint. - lint.ts: lintGhostSurfaces validates what the schema cannot see in isolation — parent references exist (with core as the implicit-root exemption), the parent graph is a tree (cycles and self-parent error), edge targets exist (no core exemption), core may not declare a parent, duplicate ids error, and near-miss parent/edge ids warn via a local Levenshtein. Edge cycles are allowed; only parent is tree-constrained. - types.ts: add errors/warnings/info counts to the lint report so it matches the other facet linters and the CLI dispatch return shape. - scan/file-kind.ts: detect surfaces.yml / surfaces.yaml and the ghost.surfaces/v1 literal, dispatch to lintSurfacesFile, mirroring the patterns/resources path. - exports + tests: 12 lint cases covering every rule plus the allowed edge cycle and the schema-failure-as-issues path.
Specs Phase 3, the breaking line: remove topology/applies_to/surface_type/scope from the canonical fingerprint (delete the Scope and Topology schemas) and add a single optional surface: placement per node, validated against surfaces.yml. Maps every removed field to its replacement against the live schema, and scopes the cut to the description facets only — check.applies_to is left for Phase 4/7 because it is coupled to map routing, and pulling it into Phase 3 would leave a half-migrated routing layer. Enumerates the lint rework (placement validation with cross-facet surface input, unplaced warns, near-miss reuse), the consumer ripple (context/graph the largest, kept minimal pending Phase 5), type removals, test migration, the major changeset stub, and explicit out-of-scope.
… test fallout A full read of context/graph.ts shows it is two subsystems: a structure/content graph built from refs (keep, mechanical coordinate-string swap) and an applicability/scope selection machinery that IS the old coordinate model (buildScopes/matchScopes/nodeMatchesTargets/applicabilityFrom*). The original 'map applicability to home surface' instruction would reimplement the selection machinery against placement in the breaking phase only for Phase 5 to discard it — doing the work twice. Revise to make Job 2 compile-dormant and rewrite selection once in Phase 5/7. Also name the expected consequence: path-based selection tests (relay/context) break, and must be migrated or marked pending rather than propped up.
…ase 3) BREAKING: remove topology, applies_to, surface_type, and scope from the canonical fingerprint; coordinates are now a single optional surface: placement per node, validated against surfaces.yml. Schema/types: delete GhostFingerprintScope/Topology/TopologyScope and the topology subtree; add surface: to situations, principles, experience_contracts, patterns, exemplars. Lint: replace topology-ref checking with checkPlacement — unplaced nodes warn (fingerprint-node-unplaced), unknown placements error (fingerprint-surface- unknown) with near-miss suggestions; surface ids are passed in via a new GhostFingerprintLintOptions.surfaceIds (cross-facet, mirroring validate lint). graph.ts: keep the structure/content graph (Job 1, mechanical surface swap); make the path/scope selection machinery (Job 2) compile-dormant — rebuilt against surfaces in Phase 5/7 rather than reimplemented against placement now. Consumers: comparable-fingerprint, package-context, package-review-command, fingerprint-contribution, fingerprint-package-layers, fingerprint-stack updated; map-derived check routing (mapFromFingerprint) and check scope/surface_type grounding made dormant pending Phase 4/7. ghost-ui: migrate the reference bundle to surfaces.yml; drop topology. Tests: migrate fixtures to surface:; rewrite topology/grounding assertions to the dormant behavior; skip path-selection suites (relay, context-entrypoint) and two cli relay cases pending Phase 5/7. Full suite green (410 passed, 31 skipped).
Specs Phase 4: delete the map.md coordinate/routing layer made dormant in Phase 3. Key finding from a full read — the map module is two tangled concerns: the routing layer (MapFrontmatter, getEffectiveMapScopes, MAP_FILENAME, the map.md schema/lint) to delete, and inventory-output types (GitInfo, InventoryOutput, LanguageHistogramEntry, TopLevelEntry) that merely live in map/types.ts and must be relocated, not deleted, since scan/inventory.ts needs them. Plan: relocate the types first as a safe sub-commit, then delete routing and rewire consumers (fingerprint-stack, checks/lint+routing+types, core/check, scope-resolver, file-kind, lint-map, scan-status, fingerprint-package). check routes on applies_to.paths alone; surface-based routing is deferred to Phase 7. Flags reachability checks for scope-resolver and routeGhostValidateForPath before delete-vs-reduce.
…(Phase 4) BREAKING: remove the map.md / ghost.map/v1 coordinate-and-routing system made dormant in Phase 3. - Delete ghost-core/map/ (schema, scopes, the map half of types) and scan/ lint-map.ts. Relocate the inventory-output types (GitInfo, InventoryOutput, LanguageHistogramEntry, TopLevelEntry) that merely lived in map/types.ts to ghost-core/scan-types.ts; scan/inventory.ts is unaffected. - routeGhostValidateForPath now routes on applies_to.paths alone (no map scope resolution); drop routeGhostPathToScopes, matched_scopes, and the options.map check-scope grounding. Surface-based routing is rebuilt in Phase 7. - core/check.ts routes by paths; remove mapFromFingerprint, parseMap, map.md reading, and the map field on the check package/stack types. - Delete dead legacy: core/scope-resolver.ts and scan/fingerprint-set.ts (both unreachable map-scope fingerprint loaders) and their re-exports. - scan-status: drop --include-scopes / scope reporting (map-driven). file-kind: drop the map DetectedFileKind and dispatch. Pull ghost-fleet out of the workspace: a private, map-native relic of a past idea, to be reintroduced later on the surface model. Removed from the root tsconfig references and the cli-manifest dump; package deleted (recoverable from history). Tests: delete map-scopes and scope-resolver tests; retarget checks routing tests to path-only. Full suite green (383 passed, 31 skipped).
…olver Specs Phase 5, the first additive phase: rebuild the dormant selection road on the surface model and ship it as a new gather command (relay's desire done right). Four pieces: a surfaces loader (reads surfaces.yml into the package model — never built; Phases 1-2 did schema+lint only), a deterministic slice resolver (own placed nodes + cascaded ancestors + one-hop typed-edge contributions with provenance, no LLM), a menu emitter (surfaces + descriptions for the host agent to match against), and the gather command (surface to slice, no/unknown surface to menu). Ambiguity returns the menu, never a whole-tree dump. Replaces entrypoint.ts Job 2 (matchScopes/globalFallbackRefs/CAPS). Scoped to the prompt road; path/diff routing is Phase 7. Re-expresses the Phase 3 selection skips against gather. Minor changeset (additive).
Phase 5 — the first additive phase. Rebuilds the dormant selection road on the surface model and ships it as a new gather command (relay's desire done right). - Surfaces loader: read surfaces.yml into the package model (the disk-read step deferred since Phase 1). loadFingerprintPackage now parses it onto LoadedFingerprintPackage.surfaces; FingerprintPackagePaths gains surfaces. - resolveSurfaceSlice (ghost-core/surfaces/resolve.ts): deterministic, no I/O, no LLM. Composes own placed nodes + cascaded ancestors (down-tree only) + one-hop typed-edge contributions, each tagged with provenance (own / ancestor:<id> / edge:<kind>:<id>). Unplaced nodes resolve as core. Checks are excluded — they route by paths (Phase 7), not surface placement. - buildSurfaceMenu (surfaces/menu.ts): deterministic id+description+parent+edges list, always including the implicit core, for the host agent to match against. - gather command: ghost gather <surface> emits the slice (markdown or json); no surface emits the menu (exit 0); unknown surface emits the menu (exit 2). Net-new, not built on relay-config/request-resolution. Tests: resolver (own/cascade/edge/unplaced/empty/no-doc), menu shape, and gather CLI (slice + menu + unknown). Full suite green (397 passed, 31 skipped). Minor changeset (additive).
Specs Phase 6: a one-shot ghost migrate command that moves a legacy .ghost/ onto the surface model. Scope correction — the plan's 'migrate this repo's dogfood .ghost/' no longer applies (deleted in the reset; ghost-ui was hand-migrated in Phase 3), so Phase 6 is only the command + tests, for external users. Key constraint: the current schema rejects legacy fields, so the migrator operates on raw parsed YAML, not the package loader. Derives surfaces.yml from topology.scopes; places single-scope nodes via surface:; drops surface_type (no placement concept) and reports applies_to.paths (Phase 7 binding concern). Core discipline: report-don't-guess — ambiguous/multi-scope nodes are surfaced for human review, never auto-placed. Additive, minor changeset.
Phase 6 — additive. A one-shot migration of a legacy .ghost/ onto the surface model. - migrateLegacyPackage (scan/migrate-legacy.ts): pure transform over raw parsed YAML (the schema rejects legacy fields, so the loader cannot read a legacy package). Derives surfaces.yml from inventory.topology.scopes; places nodes via surface: from a single scope (explicit exemplar scope, or a lone applies_to.scopes entry); strips applies_to/surface_type/scope. Report, don't guess: multi-scope, surface_type-only, and bare nodes are left unplaced and recorded in a MigrationReport, never auto-placed. paths are reported, not converted (binding is Phase 7). Does not mutate input. - looksLegacy: detect a legacy package by topology / node coordinate fields. - ghost migrate [dir] command: --dry-run (print plan + report, write nothing), --force (rewrite in place), --format cli|json. Refuses non-legacy packages. Tests: 11 transform unit tests (surface derivation, single-scope placement, exemplar placement, multi-scope/type-only/bare reporting, topology drop, no-mutation, migrated package passes lint) + 2 CLI round-trip tests (migrate → lint clean → gather places correctly; refuses non-legacy). Full suite green (410 passed, 31 skipped). Minor changeset.
…roads) Specs Phase 7, the largest and least proof-validated cut. Surfaces the core structural tension from reading fingerprint-stack.ts: the current model is merge-centric (loadFingerprintStackForPath walks root-to-leaf, mergeFingerprints unions facets child-wins-by-id, consumers read stack.merged.*). Binding replaces 'merge layers into one fingerprint' with 'resolve path to binding to surface to composed slice' (the Phase 5 resolver output, not a facet union) — the load-bearing reframe to get right before touching consumers. Four steps: binding schema+loader (ghost.binding/v1, .ghost.bind.yml), the path-to-surface resolver (nearest binding wins, explicit beats directory-implied, multi-surface directory requires explicit — report don't guess), wiring the path road (gather --path) and diff road (check/review union of surfaces), and retiring the merge (delete mergeFingerprints/mergeChecks/mergeById and child-wins-by-id, keeping layer discovery as binding discovery). Consumers measured: check (biggest), review-packet, scan-stack, scan-emit; relay left for Phase 8 deletion. Scoped to in-repo contract: . only; external references and relay rewire deferred.
The additive, format-neutral half of Phase 7: turn a filesystem path into a surface, the substrate any check routing needs underneath. - ghost.binding/v1 (ghost-core/binding/): schema, lint, types for .ghost.bind.yml. contract is in-repo '.' only (external refs deferred); paths live on the binding, never the surface. Lint enforces the supported contract and rejects double-bound surfaces. - resolvePathToSurface: pure path-to-surface resolver. Nearest binding wins; explicit beats directory-implied at the same level; a single directory-implied entry binds unconditionally; a multi-entry binding requires a path match (report, don't guess); unbound resolves to core when a root contract exists, else returns null (caller emits the menu). - discoverBindingsForPath (scan/): walk root-to-leaf, collect directory-implied (scoped surfaces.yml) and explicit (.ghost.bind.yml) candidates. - .ghost.bind.yml file-kind detection + lint dispatch. - gather --path <file>: resolve a repo path to its surface via binding, then compose the slice. Verified end-to-end. 14 new tests (7 resolver, 7 schema/lint). Full suite green (424 passed). Minor changeset. Diff road + merge retirement deferred (see phase-7b note).
…unded checks After reading how checks are actually authored (markdown + frontmatter, agent-evaluated, LLM-filtered for relevance), the 'add surface: to Ghost's deterministic detector' sketch is wrong. Settles three decisions: Ghost does not run checks; mimic the established markdown check format rather than compete with it; the differentiator is grounding — when a check flags something, Ghost supplies the why (principles/contracts) and what-to-change (patterns/exemplars) from the surface's gather slice. Ghost owns deterministic path-to-surface routing (the relevance filter, better than an LLM guess) and fingerprint grounding; it never owns the check engine. ghost.validate/v1's regex detector becomes legacy. Leaves four open questions for the 7b build (check placement, grounding emit shape, validate/v1 deprecation, and the still-owed merge retirement), explicitly not improvised here.
Sequences the governance build into four independent cuts: (1) retire the child-wins-by-id merge (Leak E) — the one piece with no dependency on the check-format question, riskiest and most independent, done first and alone; (2) ghost.check/v1 as markdown + frontmatter (name/description/severity/tools/ turn-limit + surface:), mirroring the established agent-check format, parsed and linted but never executed; (3) surface-routed relevance — a diff resolves paths to surfaces (Phase 7a) and selects checks governing those surfaces and ancestors, reusing the Phase 5 cascade and replacing path-glob routing; (4) fingerprint grounding built on review — each flagged surface emits why (principles/contracts) + what (patterns/exemplars). ghost.validate/v1 detector kept parseable but demoted from the governance path; full removal and check migration deferred. Cut 1 first and alone; 2-4 in order.
…tract (7b Cut 1) Retires the fingerprint merge (Leak E). A nested .ghost/ no longer carries its own fingerprint merged into the parent by id — instead a path resolves to the single root contract, used as-is, and nesting binds paths to that contract's surfaces (ghost.binding/v1). One contract, many bindings. - buildFingerprintStack: the root-most layer is the contract; no mergeFingerprints / mergeChecks. stack.merged → stack.contract (the root's fingerprint + checks). - Deleted mergeFingerprints/mergeIntent/mergeInventory/mergeComposition/ mergeBuildingBlocks/mergeSummary/mergeChecks/mergeById/mergeByKey/mergeStrings and the child-wins-by-id provenance. - Consumers rewired: core/check.ts, review-packet.ts, scan-stack-command.ts, fingerprintStackToPackageContext. relay left for Phase 8 deletion (minimal compile fix only). - Public check-report/v1, review, and stack JSON expose (not ) and drop . Fixes Leak E directly: the prior nested fixture had a child disabling an inherited critical check via merge — now the root contract's active check governs and the diff correctly fails. Tests rewritten to the bind-only model. Full suite green (424 passed).
Adds the Ghost check format: markdown + frontmatter, agent-evaluated, never run by Ghost. Mirrors the established .agents/checks format plus a Ghost surface: placement that routes the check. - ghost-core/check/: types, parse (frontmatter splitter), lint, and a typed loader. Frontmatter: name, description, severity (high|medium|low), optional tools / turn-limit, optional surface (flat slug; absent governs core). - lintGhostCheck validates required frontmatter, known severity, flat-slug surface, and a non-empty body; warns when unplaced. No detector, no execution. - file-kind: a markdown file under a checks/ directory lints as a check (detected by location, since the format has no schema: field). 9 tests (parse, lint paths, typed load). Full suite green (433 passed). Minor changeset. Surface-routed relevance (Cut 3) and grounding (Cut 4) next.
Specs Cut 3: the deterministic relevance filter where 7a binding, the Phase 5 cascade, and Cut 2 markdown checks compose. selectChecksForSurfaces (pure) selects checks governing a diff's touched surfaces and their ancestors, reusing ancestorChain from the slice resolver (one cascade for context and governance). Diff road: changed paths to surfaces (binding) to relevant checks. Surfaces the key decision — markdown checks route by surface, legacy validate/v1 detectors keep their path-glob router; add surface routing beside it, do not rip out the legacy path (deprecate by addition). Recommends a checks/ dir loader and a new additive command rather than disturbing check. No grounding (Cut 4), no execution, no validate/v1 removal.
…7b Cut 3) The deterministic relevance filter — where 7a binding, the Phase 5 inheritance, and Cut 2 markdown checks compose. - Extracted ancestorChain/buildParentMap into surfaces/cascade.ts; the slice resolver and check routing now share one inheritance definition (context and governance resolve the same way). - selectChecksForSurfaces (ghost-core/check/route.ts): pure, no LLM. Selects checks governing a diff's touched surfaces and their ancestors; unplaced checks govern core (apply everywhere); provenance tags own vs. ancestor. - loadChecksDir (scan/): reads <package>/checks/*.md, lints each, skips invalid with a reason. - ghost checks --diff: resolve changed paths to surfaces (binding), union, select; prints relevant checks per surface (markdown + json). Additive; the legacy validate/v1 detector path and its router are untouched. 13 tests. Full suite green (440 passed). Minor changeset. Grounding (Cut 4) next.
Specs Cut 4, the final governance cut. groundSurface projects a surface's slice (reusing resolveSurfaceSlice) into why (principles/contracts) + what-to-change (patterns/exemplars with paths), inherited from ancestors the same way context is. Key decision from reading the code: the plan said 'built on review', but review is the legacy merged-stack/validate.yml path — grounding instead extends the Cut 3 ghost checks command (the surface-native command that already resolves surfaces from a diff). Emits a grounding section keyed by touched surface (markdown + json), with --no-grounding for lean output. Ghost never runs checks; review/validate-v1 deprecation deferred.
The final governance cut — the second differentiator. For each touched surface, ghost checks now emits grounding alongside the routed checks: the why (principles + experience contracts) and the what-good-looks-like (patterns + exemplars with paths), so a flagged check can cite the intent it serves and point at an exemplar. - groundSurface (ghost-core/surfaces/ground.ts): pure projection over resolveSurfaceSlice. Maps principles/contracts to why, patterns/exemplars to what; exemplars gathered by the same placement+inheritance rule. Provenance preserved (own vs. ancestor) so brand-wide vs. surface-specific grounding is distinguishable. - ghost checks gains a grounding section (markdown + json), one entry per touched surface; --no-grounding for relevance-only output. - Decision: grounding extends the surface-native ghost checks command, not the legacy review packet (which is the retired merged-stack/validate.yml path). 7 tests (5 grounding unit + 2 CLI). Full suite green (447 passed). Minor changeset. 7b complete; Phase 8 (delete relay + cleanup) remains.
Specs Phase 8 as execution of the settled command fates: delete relay, stack, survey command, diff, describe, plus the relay-only context/ modules; update the skill bundle to teach surfaces; regenerate the manifest; fill the major changeset. Surfaces two entanglements a full read revealed: (1) relay and review share context/ machinery (entrypoint, selected-context) — partition the relay- only modules from the shared ones rather than deleting context/ wholesale, since review still needs them; (2) survey is a command AND a ghost-core/survey module referenced elsewhere — delete only the command surface, flag full module removal as a follow-up. Recommends keeping review/emit (they work on the new contract) and deferring their replacement, validate/v1 removal, and survey-module removal to later cuts. Removes the ./relay public export (breaking, in the major).
…(Phase 8) The final cutover phase — execution of the settled command fates. Deletes the second routing system for good. Commands removed: relay, stack, survey, diff, describe. Their intent lives in the surface model: gather (context), checks (diff-routed governance), bindings (path resolution). - Delete relay.ts, relay-command.ts, relay-runtime-helpers.ts, scan-stack-command.ts, and the describe/diff/survey command blocks in fingerprint-commands.ts. Clean command-discovery (drop dead entries; add gather/checks/migrate). - Partition context/: delete the relay-only modules (relay-config, relay-config-loader, default-relay-config, relay-context, relay-modes, relay-request, relay-request-input, request-resolution, request-stack-document, projection) and the orphaned entrypoint-markdown; keep entrypoint, package-context, selected-context, graph, selection-reasons (review still uses them). - Remove the ./relay public export from package.json and the packed-package smoke check. - Skill bundle: rewrite brief/review/verify/schema/SKILL to teach surfaces, gather, and checks instead of relay/topology/scope. - Tests: delete relay.test.ts and the skipped context-entrypoint test; update help-index, public-exports, and skill-install assertions to the new surface. Kept (per scope): review, emit (work on the contract), ghost-core/survey module, ghost.validate/v1 — their removal is a later cut. Full suite green (429 passed). Major changeset. The cutover is complete.
nahiyankhan
force-pushed
the
ghost/02-surface-model
branch
from
c12f8f1 to
4de14e2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PR 2 of 6 · base: `ghost/01-surface-spec`
The surface-model engine.⚠️ Contains breaking changes (`feat!`).
Contents
Stack order