Epic: the parking ladder — cancellable, pressure-driven eviction (Nominated → Parked-paused → Parked-local → Evicted-remote)
Part of the 2026-07 core-ops overhaul (deep-research synthesis). Tier 2. Size Epic.
Depends on: ram-ledger (hard, phases 2–4), epic-session-op-log (soft — phase 1 ships standalone; cancel becomes an op-cancel once the op log lands).
Related: eviction-finalize-host-durable (D5 durability of the banked diff), epic-transactional-snapshots-revive (rung-4 one-RPC Revive), prefault-admission-control (rung-4 tail), telemetry-restoration (rung metrics), epic-rpc-consolidation (the new host verbs land on the consolidated surface if it ships first), queue-fairness, epic-binding-epoch-delivery (its Phase 4 deletes the host-side idle detection plane this epic's rung-skip/pressure-gate interlocks live in today — coordinate the handoff).
Cross-references: epic-binding-epoch-delivery = #542 · epic-rpc-consolidation = #549 · epic-session-op-log = #543 · epic-transactional-snapshots-revive = #544 · eviction-finalize-host-durable = #529 · generation-purge = #530 · prefault-admission-control = #534 · queue-fairness = #537 · ram-ledger = #540 · telemetry-restoration = #526
Problem
Eviction is irreversible from the instant of nomination. The session FSM has no Evicting → Active edge — Evicting can only reach Idle | HostLost | Dead | Completed (crates/engram-core/src/types/session.rs:241, verified). So when the single most common "the user is here, keep the VM" signal arrives — a follow-up prompt during eviction — the coordinator's only move is to wait for a perfectly healthy VM to be snapshotted and destroyed, then rebuild it from the snapshot it just took. The prompt sits in ensure_active_after_evicting_hold (a 250 ms poll, 8 s default budget, crates/engram-coordinator/src/api/snapshot.rs:41,46,548-604), and because the eviction scanner runs on a 10 s tick (crates/engram-coordinator/src/idle_evictor.rs:926), a session can sit at Evicting with a fully live, untouched VM for up to 10 s during which cancelling the nomination would have been one PG write.
The rebuild the user then pays is the full resume pipeline: prod resume p50 12.2 s, p95 89.1 s; user-perceived prompt→run_started p50 ≈24.5 s (prod evidence, 2026-06-17→07-01 window). Worse, the evict-then-immediately-resume shape — precisely the interactive case — collides with the eviction's own session_lease (held through the D5 finalize upload), costing 3–21 s of lease-acquire retries (8 attempts × 3 s, api/snapshot.rs:232). And every full-resume rebuild rewinds the transcript: 45/45 prod resumes rolled back events (median 4, max 62), with visible turn replay.
Meanwhile the descent side fights the latency goal: soft-TTL nomination fires purely on a 300 s clock (crates/engram-host-agent/src/idle_evictor.rs:39) regardless of whether the host needs the RAM. The team's own fix — pressure-aware nomination (ADR 0034 addendum 2026-06-30) — ships default-off behind ENGRAM_IDLE_EVICT_PRESSURE_AWARE (host idle_evictor.rs:119-137), so prod still evicts warm interactive VMs on a clock, for zero density gain, and then makes the returning user pay for it.
Evidence
Prod numbers below are from the 2026-07-01 evidence pass (prod PG session_events, 2026-06-17→2026-07-01, 14-day window; single-tenant dogfood traffic, small n — indicative, not statistically strong).
- Resume latency (idle→active, n=26 excluding instant-reattach outliers): p50 12.2 s, p90 69.4 s, p95 89.1 s, max 299.3 s. Prompt→
run_started (n=21): p50 ≈24.5 s, p90 ≈140 s, max 1703 s.
- Idle dwell before resume: p50 11.9 min, p90 7.3 h — the p50 is squarely rung-1/2 territory; the p90 is why rung-2 dwell must be capped.
- Eviction volume: 105
evicted events / 14 d (~7.5/day); eviction pipeline p50 6.5 s, p95 80 s. 45 resume attempts; 45/45 emitted recovered_from_checkpoint with rolled_back > 0 (median 4, max 62 — visible turn replay).
- Evict-vs-resume lease collision: 3–21 s (8 × 3 s lease-acquire retries against the D5 finalize upload holding the lease) —
crates/engram-coordinator/src/api/snapshot.rs:219-249 (verified: for attempt in 0..8, 3 s default backoff at :231).
- No
Evicting → Active edge: crates/engram-core/src/types/session.rs:241 (verified) — Evicting => matches!(target, Idle | HostLost | Dead | Completed); doc table at :205-209; test edge list at :588-607.
- The hold loop:
ensure_active's Evicting arm dispatches to ensure_active_after_evicting_hold (crates/engram-coordinator/src/api/snapshot.rs:494, fn at :548-604; DEFAULT_RESUME_EVICTING_HOLD_SECS = 8 at :41, 250 ms poll at :46; tests at :2330-2415).
- Nomination-to-pipeline gap: host nominates on a 10 s tick (
crates/engram-host-agent/src/idle_evictor.rs:47); coord handler idle_eviction_candidates (crates/engram-coordinator/src/api/host_http.rs:893, route at api/mod.rs:73) only flips Active → Evicting; the eviction scanner sweeps status='evicting' on its own 10 s tick with a 20-attempt budget → HostLost (crates/engram-coordinator/src/idle_evictor.rs:908-930).
- Pressure-aware nomination exists but ships dark:
pressure_aware_from_env default OFF (crates/engram-host-agent/src/idle_evictor.rs:119-137), mem_pressure_check at :167; the gate is applied in the host nomination tick at crates/engram-host-agent/src/lib.rs:1341-1374 (IdleKind::Soft candidates dropped when not under pressure; Hard always nominated). ADR 0034 addendum (2026-06-30) documents the design and the default-off ship.
- The cheap-pause primitive already exists: diff capture pauses the guest 38 ms (cold-boot
track_dirty_pages) / 53 ms (post-restore) vs 2,090 ms full capture (ADR 0028 P1, dev-vm measured, 256 MiB guest; docs/adr/0028-eviction-durability-under-host-roll.md); the periodic diff-first checkpoint driver lives at crates/engram-host-agent/src/checkpoint.rs (spawn_checkpoint_driver, CheckpointChain).
- FC pause/resume plumbing exists end to end:
FirecrackerClient::pause/resume via PATCH /vm (crates/engram-sandbox-firecracker/src/client.rs:160,432-437), SandboxBackend::pause/resume (defaults Ok(()) — crates/engram-core/src/traits/sandbox.rs:397-411), FC overrides (crates/engram-sandbox-firecracker/src/lib.rs:4216,4233), HostClient::pause/resume (crates/engram-core/src/traits/host_client.rs:363,369). The D5 split (snapshot_begin/snapshot_wait) is the pause-side half of an eviction snapshot (traits/sandbox.rs:243-258; coord call site idle_evictor.rs:229, finish_eviction_background at :442).
cancel_evict does not exist anywhere in the tree (verified by grep).
- Downstream hold loops that exist only because eviction is irreversible: the 60 s @ 500 ms mid-move delivery HOLD (
crates/engram-coordinator/src/api/prompt.rs:137-168), the 5 s @ 200 ms reattach-forward budget (prompt.rs:30-32), the 10 s SEND_PROMPT_ATTACH_WAIT_SECS (crates/engram-host-agent/src/harness.rs:66).
Root cause
Idle is modeled as a cliff, not a ladder. The moment the host nominates, the session is on a one-way conveyor — nominate → snapshot → destroy → Idle → (full rebuild on return) — and every stage is welded to the next inside two non-cancellable pipelines (evict and resume) glued together by status polling and a wall-clock lease. Physically, eviction is reversible right up to destroy(): the VM is untouched at nomination, and even the capture is a 38–53 ms diff pause. The FSM just cannot express "the user came back."
Past fixes were bandages on this shape rather than changes to it: the soft TTL was raised 30 s → 300 s (ADR 0039 #20) because the clock kept evicting live users; the Evicting hold loop was added because prompts bounced 409 for minutes; the pressure-aware addendum was built — and then shipped default-off — because parked residency isn't accounted anywhere, so turning it on fleet-wide would let placement overcommit (the ADR 0046 allocatable_mib honesty problem, → ram-ledger). Each bandage narrowed a symptom; none made eviction cancellable or descent proportional to actual pressure.
Solution
Model idle as descending rungs driven only by the existing memory-pressure signal (mem_pressure_check, host idle_evictor.rs:167), each rung with a cheap ascent. Pressure-driven descent becomes the only mode (the ENGRAM_IDLE_EVICT_PRESSURE_AWARE flag is deleted, not defaulted-on); the soft clock TTL survives only as the candidacy signal (how long since idle), never as a reclaim trigger. Hard-TTL nomination and the coord's 30-min PG backstop (crates/engram-coordinator/src/idle_detect_backstop.rs) stay as the absolute residency ceiling.
The invariant made structural: eviction is reversible up to the moment its resources are actually reclaimed, and it descends only as far as pressure requires.
The rungs (with the feasibility critic's corrections baked in — do not relitigate):
- Rung 1 — Nominated (
status = Evicting, park_rung = 1, VM untouched). Ascent = one PG write over a new Evicting → Active legality edge + a host cancel_evict (clear the eviction-inflight marker, reset idle clocks). Covers the 10 s scanner gap and deletes the 8 s hold loop. Ships standalone first (days). This is the only new FSM edge in the overhaul (resolved decision: no new FSM states; in-flight visibility comes from op rows).
- Rung 2 — Parked-paused (
status = Evicting, park_rung = 2; FC paused after a 38–53 ms diff capture whose upload runs in the background; NBD/netns/vsock/harness all intact in RAM). Ascent = host unpark (FC PATCH /vm state=Resumed) + the same Evicting → Active edge — no harness respawn, no claude --resume, no start_agent, no transcript rewind. Critic corrections (C1), owned honestly: a paused VM frees zero RAM, so rung 2 buys interactive-return latency, not density — it is the state a descending session passes through (the diff is banked, making 2→3 an O(destroy) step) and briefly dwells in under mild pressure; under acute pressure the ladder skips it (1→3 directly). And because idle dwell p90 is 7.3 h, rung-2 dwell is capped (ENGRAM_PARK_PAUSED_MAX_DWELL_SECS, default 900) — multi-hour pauses resume into stale guest TCP (claude's API sockets); recoverable via the CLI's retries but not the clean "<100 ms and nothing notices" story, so we don't let sessions sit there.
- Rung 3 — Parked-local (
status = Idle via the existing Evicting → Idle edge, park_rung = 3; VM destroyed, but the snapshot staging dir — state.bin, rolling memfile + diff chain, sidecar — and NBD backing state are retained on host NVMe). Ascent = normal resume path, but placement pins the parking host and the restore reads only local artifacts: target ~0.5–2 s. This is NEW machinery — the critic is explicit that live_attach.rs reattaches to live processes; "destroy but retain the data plane" is not that code. Device slots (NbdSlotAllocator, crates/engram-host-agent/src/disk_daemon/slot.rs) are a bounded pool: retain best-effort, release under slot pressure (ascent re-acquires).
- Rung 4 — Evicted-remote (today's Idle,
park_rung = 0): the full restore, the only rung that pays GCS — now the rare case for interactive sessions. Made faster separately by epic-transactional-snapshots-revive and prefault-admission-control.
Descent 2→3 frees RAM (the pressure signal's actual target); 3→4 frees disk + slots. Because the diff was banked and uploaded at rung 2, descent 2→3 is destroy-only and 3→4 is a pure local delete — the durable snapshot already exists. Rung-2/3 residency is charged against allocatable_mib (ADR 0046, docs/adr/0046-placement-memory-reservation.md:176) via ram-ledger — this is the hard dependency that makes pressure-only mode safe to be the only mode.
Dropped from v1 (critic consensus C10): the speculative ensure_rung typing-signal hint RPC. Rungs 1–2 cover the common case without it.
Schema (new migration; migrations are checksum-immutable, so a new file — next free number, deploy/migrations/0077_park_rung.sql at time of writing, re-check the high-water mark at branch time)
-- Parking-ladder rung bookkeeping (ADR 00XX). park_rung is meaningful
-- only while status IN ('evicting','idle'): 0 = none (running / plain
-- idle), 1 = nominated, 2 = parked-paused, 3 = parked-local.
ALTER TABLE sessions ADD COLUMN park_rung SMALLINT NOT NULL DEFAULT 0;
ALTER TABLE sessions ADD COLUMN parked_at TIMESTAMPTZ;
ALTER TABLE sessions ADD COLUMN parked_host_id UUID;
CREATE INDEX sessions_parked_idx ON sessions (park_rung) WHERE park_rung > 0;
Wire/host-verb changes (clean break; coord + host roll together — mind WIRE_VERSION skew during the roll)
New HostClient / SandboxBackend verbs (defaults Err(SandboxError::InvalidSpec(...)), mirroring snapshot_begin — NOT the silent Ok(()) shape pause/resume use today):
cancel_evict(sandbox_id) — clear the hub's eviction-inflight marker + reset idle clocks (so the next tick doesn't instantly re-nominate a just-cancelled session). Host-agent side: HarnessHub (crates/engram-host-agent/src/harness.rs).
park(sandbox_id) -> ParkOut { snapshot_id } — pause + diff capture + background upload, leaving the VM paused (the D5 snapshot_begin pipeline minus the destroy handoff; confirm during implementation whether snapshot_begin currently resumes or leaves paused, and factor a shared helper in pooled_backend.rs rather than a second pipeline).
unpark(sandbox_id) — FC PATCH /vm state=Resumed (+ any guest clock re-step the resumed guest needs — unverified, see below) + mark the hub entry live again.
destroy_retain_local(sandbox_id) -> ParkedLocalOut — tear down FC process/jail/netns, retain the staging dir + NBD local state, write a parked.json record next to the artifacts (re-advertised in heartbeat after a host-agent restart, like CheckpointRecord).
Interlocks
- Race safety without the op log (phase 1): cancel first
try_acquires the session_lease with zero retries. Lease free ⇒ the scanner hasn't started ⇒ cancel_evict on the host first (clock reset closes the re-nomination race), then CAS Evicting → Active. Lease held ⇒ pipeline mid-flight ⇒ fall back to the existing hold (phase 1) / unpark at the rung-2 boundary (phase 2). The pipeline's existing K5 status re-check under the lease (idle_evictor.rs:37-39 EvictOutcome doc) makes a lost race a no-op — verify it re-reads status after lease acquire, add if missing. Once epic-session-op-log lands, cancel becomes cancel(evict-op) and this bespoke interlock retires.
- Scanner semantics:
Evicting no longer implies "pipeline pending"; the eviction scanner becomes the descent driver, keyed on park_rung: rung 1 → run descent when pressure demands (or hard-TTL/backstop); rung 2 → enforce the dwell cap; the 20-attempt → HostLost budget counts only actual transition attempts, never parked dwell.
- Checkpoint driver: the periodic ADR 0028 checkpoint driver must skip parked sandboxes (a paused VM has no new dirty state; the in-flight-capture guard at
checkpoint.rs:219 area already rejects concurrent captures — extend it to park_rung >= 2).
- Idle detection:
HarnessHub::idle_sandboxes (harness.rs:816) must skip sandboxes at rung ≥ 1 (they are already descending; re-nomination is noise). Coordination with epic-binding-epoch-delivery: that epic's Phase 4 deletes HarnessHub::idle_sandboxes and the host eviction tick (crates/engram-host-agent/src/lib.rs:1252-1440) outright and moves idle detection coordinator-side (idle_detector.rs) — which is also where today's pressure-gate anchors (host idle_evictor.rs:119-137, lib.rs:1341-1374) live. This epic's "skip rung ≥ 1" interlock and the pressure gate must therefore be expressed as detector-agnostic requirements, not as calls into the soon-to-be-deleted host code: whichever detector exists at land time (host-side idle_sandboxes today, coordinator-side idle_detector.rs after binding-epoch-delivery Phase 4 lands) must skip descending sessions, and the pressure gate moves with detection rather than being pinned to the host tick.
- Host restart / node death: rung-2 VMs die with the host-agent process boundary like any live VM (survivor reattach applies as today); rung-3 artifacts survive a pod roll (
parked.json re-advertised) but die with node replacement — the session is then a plain rung-4 Idle and resume falls back to GCS. No new durability class is invented; the banked snapshot is already remote.
Implementation plan
ADR first (repo convention): author docs/adr/00XX-parking-ladder.md (next free number; 0067 at time of writing) as Proposed before phase 1 code; update it between phases with divergences; flip to Accepted with the commit chain at the end. Each phase is one PR on its own worktree.
Phase 1 — Rung 1: cancellable nomination (ships standalone, days)
crates/engram-core/src/types/session.rs: add Active to the Evicting arm of can_transition_to (:241), update the doc table (:205-209), extend the legal-edge test list (:588-607) with (Evicting, Active).
crates/engram-core/src/traits/host_client.rs + traits/sandbox.rs: add cancel_evict with an Err(InvalidSpec) default; implement in PooledBackend (clear mark_eviction_inflight state + reset the hub idle clocks for the sandbox); wire through the host-agent gRPC server (crates/engram-host-agent/src/grpc_server.rs) and the coord-side host client + engram-protocol (WIRE_VERSION bump).
crates/engram-coordinator/src/api/snapshot.rs: rewrite the SessionState::Evicting arm of ensure_active (:494) as cancel_evict_and_reactivate: zero-retry SessionLeaseGuard::try_acquire → on success: re-read status, host.cancel_evict(sandbox_id), transition_session(Evicting → Active), emit StatusChanged, release; on lease-held: fall back to the existing ensure_active_after_evicting_hold (deleted in phase 2).
- Verify/add the pipeline's status re-check under the lease in
evict_session_to_state (crates/engram-coordinator/src/idle_evictor.rs:95-140): after lease acquire, a row no longer Evicting ⇒ EvictOutcome::Skipped.
- Metrics:
engram_evict_cancels_total, cancel-path latency histogram (rides telemetry-restoration's export, but the instruments land here).
- Tests (below) +
just check + sqlx cache; single PR.
Phase 2 — Rung 2: parked-paused + dwell cap
- Migration
0077_park_rung.sql (above); crates/engram-postgres/src/lib.rs row mapping + MetadataStore accessors (set_park_rung, list_parked_over_dwell); regen .sqlx/.
- Host:
park/unpark verbs — factor the pause+diff+background-upload out of the post-eviction-finalize snapshot_begin path (the EvictionFinalizeRecord pipeline), not the current D5 shape: eviction-finalize-host-durable lands first via the op-log dependency chain, so by the time this phase is implemented snapshot_begin is already the finalize-record-backed pipeline, and park factors out of that (crates/engram-host-agent/src/pooled_backend.rs); park leaves the VM paused; the banked capture lands as a normal ADR 0028 checkpoint record so 2→3 needs no second capture. Guard the checkpoint driver and idle_sandboxes against parked sandboxes.
- Coord descent driver (
crates/engram-coordinator/src/idle_evictor.rs): a claimed Evicting session under pressure descends 1→2 (host.park → park_rung=2, parked_at=now()), or 1→3 directly under acute pressure (pre-phase-3: 1→4, the composed pipeline, unchanged). Scanner enforces ENGRAM_PARK_PAUSED_MAX_DWELL_SECS (default 900): over-dwell rung-2 sessions descend (2→3 once phase 3 lands; 2→4 until then).
- Ascent: extend
cancel_evict_and_reactivate — park_rung=2 ⇒ host.unpark → Evicting → Active → deliver. No resume_session, no lease-retry ladder, no rewind.
- Delete
ensure_active_after_evicting_hold (+_for, constants :39-55, env knob, tests :2330-2415 — replaced by cancel/unpark tests in the same PR) — unless already deleted by epic-session-op-log Phase 1; whichever lands second ports the replacement tests.
- ADR 0052 idle-drain note: the drain-claude-before-snapshot grace (
IDLE_DRAIN_GRACE_SECS, harness.rs:68-75) is for the terminal eviction capture; park does not drain (the whole point is the harness stays alive). Descent 2→3/4 reuses the already-banked capture, so the drain special-case narrows to the direct-to-rung-4 path (record in the ADR whether it can be deleted outright — clean break preferred).
Phase 3 — Rung 3: parked-local
- Host:
destroy_retain_local on PooledBackend — destroy the VM/jail/netns, retain staging dir + NBD local state, write parked.json; heartbeat advertises parked-local residency (disk bytes + slot holds) for ram-ledger; release retained slots under slot-pool pressure.
- Coord: descent 2→3 (and 1→3 under acute pressure) sets
park_rung=3, parked_host_id and transitions Evicting → Idle (existing edge — rung-3 ascent is a resume, so Idle stays honest).
- Ascent:
resume_session fast path — park_rung=3 ⇒ placement pins parked_host_id; restore must read zero GCS (local staging already present; assert via metrics). If the pinned host is gone, clear the park columns and fall through to rung 4.
- GC / descent 3→4: delete local artifacts +
parked.json, release slots, park_rung=0. Orphan-reap integration: host restart re-advertises parked.json; a record with no matching PG row is swept (orphan_reap.rs pattern).
Phase 4 — Pressure-only becomes the only mode; clock-TTL reclaim deleted
- Requires
ram-ledger merged (parked + resident RAM charged to allocatable_mib) — otherwise the pressure gate lies and placement overcommits.
crates/engram-host-agent/src/idle_evictor.rs: delete pressure_aware_from_env + the ENGRAM_IDLE_EVICT_PRESSURE_AWARE knob (:119-137); crates/engram-host-agent/src/lib.rs:1341-1374: the pressure gate becomes unconditional (Soft candidates nominate only under pressure; Hard always). ENGRAM_IDLE_EVICT_MEM_FLOOR_PCT stays tunable. Soft TTL remains the candidacy age only.
- Delete the phase-1 lease-held hold fallback if any remnant survived phase 2.
- Prod validation window against the baselines below; flip the ADR to
Accepted.
What gets deleted
ensure_active_after_evicting_hold / _for, DEFAULT_RESUME_EVICTING_HOLD_SECS, RESUME_EVICTING_POLL_INTERVAL, resume_evicting_hold_from_env, ENGRAM_RESUME_EVICTING_HOLD_SECS — crates/engram-coordinator/src/api/snapshot.rs:39-55, 494, 526-604 + tests :2330-2415 (phase 2, replacements land in the same PR).
- The evict-vs-resume lease collision class for interactive returns: rung-1/2 ascents never call
resume_session, so the 8×3 s retry ladder (api/snapshot.rs:219-249) stops firing on that path (the ladder itself stays for genuine rung-4 concurrency until epic-session-op-log retires it).
- Clock-TTL reclaim: the unconditional soft-TTL nomination path and the
ENGRAM_IDLE_EVICT_PRESSURE_AWARE flag + pressure_aware_from_env (crates/engram-host-agent/src/idle_evictor.rs:119-137; gate at lib.rs:1341-1374 becomes unconditional) (phase 4).
- The transcript-rewind-on-every-return pathology for rungs 1–2 (45/45 prod resumes rolled back events): parked ascents never leave VM state, so
recovered_from_checkpoint/rewind simply doesn't run on them (the rung-1-rewind code itself stays for rung-4 resumes).
- The idle-drain special case narrows (see phase 2 step 6); ADR decides delete-vs-narrow.
- NOT deleted (explicitly): hard-TTL nomination, the coord 30-min
idle_detect_backstop, the eviction scanner (it becomes the descent driver), ADR 0028 periodic checkpoints, the session_lease (until the op log subsumes it).
Acceptance criteria
Baselines: prod 2026-06-17→07-01 (resume p50 12.2 s / p95 89.1 s; prompt→run_started p50 ≈24.5 s; ~7.5 evictions/day; lease collision 3–21 s; 45/45 resumes rewound).
- Phase 1: a prompt arriving while
status='evicting' with no lease held reaches run_started with no VM teardown and no snapshot restore (assert: no resumed/recovered_from_checkpoint event); cancel-path added latency < 1 s. A prompt racing the in-flight pipeline still lands correctly (falls back, no orphan, no double-resume).
- Phase 2: ascent from parked-paused completes
Evicting → Active + delivery in < 1 s end-to-end (host-side unpark < 200 ms), vs the 12.2 s p50 baseline; zero transcript rewind on rung-1/2 ascents; no session dwells parked-paused longer than ENGRAM_PARK_PAUSED_MAX_DWELL_SECS + one scanner tick; under acute memory pressure the ladder skips rung 2 (observable via engram_park_descents_total{from="1",to="3"}).
- Phase 3: same-host rung-3 ascent < 2 s with zero GCS reads on the restore path (assert via metrics/logs); parking-host-gone falls back to rung 4 cleanly; rung-3 artifacts survive a host-agent pod restart (re-advertised) and their loss on node death degrades to rung 4, never to an error.
- Phase 4: on a host with free RAM above the floor, soft-idle sessions are never evicted (
IDLE_EVICT_KEPT_RESIDENT_TOTAL rises; eviction count drops materially vs ~7.5/day); the lease-collision class (3–21 s waits) is absent from rung-1/2 ascents; placement never overcommits against parked residency (ram-ledger gauges reconcile).
- Rung-2/3 ascent-latency targets (<100 ms host-side unpark, ~0.5–1 s local restore) are engineering estimates, not promises — measure and record actuals in the ADR (the critic specifically flagged the <100 ms headline as over-credited; the honest headline is "interactive returns stop paying the 12–89 s rebuild").
Testing & CI
- engram-core (
just check lane): transition-table unit tests — extend the edge list at session.rs:588-607 with (Evicting, Active); assert (Idle, Active) remains illegal (no new edges beyond the pinned one).
- coordinator live-PG (new
crates/engram-coordinator/tests/parking_ladder_live_pg.rs, mirroring placement_reservation_live_pg.rs shape): cancel-vs-scanner race (lease interlock: exactly one wins, loser no-ops), dwell-cap descent, scanner budget not counting parked dwell, rung-3 host-gone fallback. Keep SQLX_OFFLINE cache in sync; new migration files only — applied migrations are checksum-immutable.
- FC integration (new
crates/engram-sandbox-firecracker/tests/park_unpark.rs, #[ignore]'d, Linux+KVM): one park→unpark cycle on a tiny guest — assert the diff is banked, the guest process survives (marker file/uptime), and unpark resumes execution; one park→destroy_retain_local→local-restore cycle. Must be wired into ci.yml's --test list (repo rule: FC tests not in the list never run). Size to the property, not realism: small guest, one cycle, no long sleeps — do not reinflate the FC lane.
- e2e stack (
test-e2e-stack): prompt-during-evicting scenario — nominate, immediately prompt, assert active-without-rebuild. The deleted evicting-hold tests (api/snapshot.rs:2330-2415) are replaced by cancel-path coverage in the same change — never delete e2e/regression coverage without a landing replacement.
- Cross-compile check: host-agent changes are Linux-heavy —
nix develop -c cargo clippy --target aarch64-unknown-linux-musl -p engram-host-agent --all-targets (macOS clippy can't see cfg(target_os = "linux") paths).
Guardrails
- Repo conventions: ADR bookends (Proposed before phase-1 code → Accepted with the commit chain); one phase = one PR on a dedicated worktree; one logical change per commit;
just check before every commit; validate any ci.yml edit with actionlint/yaml.safe_load; no type/lint laundering; clean breaks over shims — the wire bump and flag deletion are intentional breaks (zero users), but coord rolls before hosts, so sequence the WIRE_VERSION bump to avoid the known serving-hosts-dropped skew wedge.
- VZ / Process backend story (decision 11): rung 1 (
cancel_evict) is control-plane-only and must work on all backends — implement it for VZ and Process (it's a hub-state operation, not a VMM one). Rungs 2–3 (park/unpark/destroy_retain_local) are FC-only: the trait defaults return Err(InvalidSpec) (mirror snapshot_begin — never a silent Ok(())), and the descent driver explicitly catches InvalidSpec and falls through to the composed full eviction (rung 4), exactly as the D5 branch does today (idle_evictor.rs:234-243). VZ has pause/resume (engram-sandbox-vz/src/vm.rs:303,313) but no diff-snapshot machinery — do not fork the design to add a VZ park path; production (FC) drives the design.
- Pinned decisions — do not relitigate: no
ensure_rung typing-signal hint in v1 (critic C10); no new FSM states — Evicting → Active is the only new edge, rungs live in park_rung (decision 5); rung 2 is latency-only, capped, and skippable (critic C1); rung 3 is new machinery, not live_attach reuse; capture jobs, warm-VM pre-spawn, and File-mode memory paths are other issues' scope (File mode gets deleted in generation-purge, not enabled here).
- Do not touch: ADR 0028 checkpoint-chain semantics beyond the skip-parked guard; the hard-TTL/backstop ceiling (it is what makes suppressing soft nominations safe); existing e2e eviction/resume coverage except with same-change replacements.
- Unverified — confirm during implementation: whether the D5
snapshot_begin pipeline leaves the VM paused or resumes it before the destroy handoff (determines how much of park is a refactor vs new code); whether a long-paused FC guest needs an explicit clock re-step on Resumed (vs KVM clock catch-up) — research the FC spec, don't guess-and-check; guest TCP staleness after multi-hour pause is the reasoned basis for the dwell-cap default (900 s) — tune from prod data, and record the measured value in the ADR.
Epic: the parking ladder — cancellable, pressure-driven eviction (Nominated → Parked-paused → Parked-local → Evicted-remote)
Problem
Eviction is irreversible from the instant of nomination. The session FSM has no
Evicting → Activeedge —Evictingcan only reachIdle | HostLost | Dead | Completed(crates/engram-core/src/types/session.rs:241, verified). So when the single most common "the user is here, keep the VM" signal arrives — a follow-up prompt during eviction — the coordinator's only move is to wait for a perfectly healthy VM to be snapshotted and destroyed, then rebuild it from the snapshot it just took. The prompt sits inensure_active_after_evicting_hold(a 250 ms poll, 8 s default budget,crates/engram-coordinator/src/api/snapshot.rs:41,46,548-604), and because the eviction scanner runs on a 10 s tick (crates/engram-coordinator/src/idle_evictor.rs:926), a session can sit atEvictingwith a fully live, untouched VM for up to 10 s during which cancelling the nomination would have been one PG write.The rebuild the user then pays is the full resume pipeline: prod resume p50 12.2 s, p95 89.1 s; user-perceived prompt→
run_startedp50 ≈24.5 s (prod evidence, 2026-06-17→07-01 window). Worse, the evict-then-immediately-resume shape — precisely the interactive case — collides with the eviction's ownsession_lease(held through the D5 finalize upload), costing 3–21 s of lease-acquire retries (8 attempts × 3 s,api/snapshot.rs:232). And every full-resume rebuild rewinds the transcript: 45/45 prod resumes rolled back events (median 4, max 62), with visible turn replay.Meanwhile the descent side fights the latency goal: soft-TTL nomination fires purely on a 300 s clock (
crates/engram-host-agent/src/idle_evictor.rs:39) regardless of whether the host needs the RAM. The team's own fix — pressure-aware nomination (ADR 0034 addendum 2026-06-30) — ships default-off behindENGRAM_IDLE_EVICT_PRESSURE_AWARE(hostidle_evictor.rs:119-137), so prod still evicts warm interactive VMs on a clock, for zero density gain, and then makes the returning user pay for it.Evidence
Prod numbers below are from the 2026-07-01 evidence pass (prod PG
session_events, 2026-06-17→2026-07-01, 14-day window; single-tenant dogfood traffic, small n — indicative, not statistically strong).run_started(n=21): p50 ≈24.5 s, p90 ≈140 s, max 1703 s.evictedevents / 14 d (~7.5/day); eviction pipeline p50 6.5 s, p95 80 s. 45 resume attempts; 45/45 emittedrecovered_from_checkpointwithrolled_back > 0(median 4, max 62 — visible turn replay).crates/engram-coordinator/src/api/snapshot.rs:219-249(verified:for attempt in 0..8, 3 s default backoff at :231).Evicting → Activeedge:crates/engram-core/src/types/session.rs:241(verified) —Evicting => matches!(target, Idle | HostLost | Dead | Completed); doc table at :205-209; test edge list at :588-607.ensure_active's Evicting arm dispatches toensure_active_after_evicting_hold(crates/engram-coordinator/src/api/snapshot.rs:494, fn at :548-604;DEFAULT_RESUME_EVICTING_HOLD_SECS = 8at :41, 250 ms poll at :46; tests at :2330-2415).crates/engram-host-agent/src/idle_evictor.rs:47); coord handleridle_eviction_candidates(crates/engram-coordinator/src/api/host_http.rs:893, route atapi/mod.rs:73) only flipsActive → Evicting; the eviction scanner sweepsstatus='evicting'on its own 10 s tick with a 20-attempt budget →HostLost(crates/engram-coordinator/src/idle_evictor.rs:908-930).pressure_aware_from_envdefault OFF (crates/engram-host-agent/src/idle_evictor.rs:119-137),mem_pressure_checkat :167; the gate is applied in the host nomination tick atcrates/engram-host-agent/src/lib.rs:1341-1374(IdleKind::Softcandidates dropped when not under pressure;Hardalways nominated). ADR 0034 addendum (2026-06-30) documents the design and the default-off ship.track_dirty_pages) / 53 ms (post-restore) vs 2,090 ms full capture (ADR 0028 P1, dev-vm measured, 256 MiB guest;docs/adr/0028-eviction-durability-under-host-roll.md); the periodic diff-first checkpoint driver lives atcrates/engram-host-agent/src/checkpoint.rs(spawn_checkpoint_driver,CheckpointChain).FirecrackerClient::pause/resumeviaPATCH /vm(crates/engram-sandbox-firecracker/src/client.rs:160,432-437),SandboxBackend::pause/resume(defaultsOk(())—crates/engram-core/src/traits/sandbox.rs:397-411), FC overrides (crates/engram-sandbox-firecracker/src/lib.rs:4216,4233),HostClient::pause/resume(crates/engram-core/src/traits/host_client.rs:363,369). The D5 split (snapshot_begin/snapshot_wait) is the pause-side half of an eviction snapshot (traits/sandbox.rs:243-258; coord call siteidle_evictor.rs:229,finish_eviction_backgroundat :442).cancel_evictdoes not exist anywhere in the tree (verified by grep).crates/engram-coordinator/src/api/prompt.rs:137-168), the 5 s @ 200 ms reattach-forward budget (prompt.rs:30-32), the 10 sSEND_PROMPT_ATTACH_WAIT_SECS(crates/engram-host-agent/src/harness.rs:66).Root cause
Idle is modeled as a cliff, not a ladder. The moment the host nominates, the session is on a one-way conveyor — nominate → snapshot → destroy → Idle → (full rebuild on return) — and every stage is welded to the next inside two non-cancellable pipelines (evict and resume) glued together by status polling and a wall-clock lease. Physically, eviction is reversible right up to
destroy(): the VM is untouched at nomination, and even the capture is a 38–53 ms diff pause. The FSM just cannot express "the user came back."Past fixes were bandages on this shape rather than changes to it: the soft TTL was raised 30 s → 300 s (ADR 0039 #20) because the clock kept evicting live users; the Evicting hold loop was added because prompts bounced 409 for minutes; the pressure-aware addendum was built — and then shipped default-off — because parked residency isn't accounted anywhere, so turning it on fleet-wide would let placement overcommit (the ADR 0046
allocatable_mibhonesty problem, →ram-ledger). Each bandage narrowed a symptom; none made eviction cancellable or descent proportional to actual pressure.Solution
Model idle as descending rungs driven only by the existing memory-pressure signal (
mem_pressure_check, hostidle_evictor.rs:167), each rung with a cheap ascent. Pressure-driven descent becomes the only mode (theENGRAM_IDLE_EVICT_PRESSURE_AWAREflag is deleted, not defaulted-on); the soft clock TTL survives only as the candidacy signal (how long since idle), never as a reclaim trigger. Hard-TTL nomination and the coord's 30-min PG backstop (crates/engram-coordinator/src/idle_detect_backstop.rs) stay as the absolute residency ceiling.The invariant made structural: eviction is reversible up to the moment its resources are actually reclaimed, and it descends only as far as pressure requires.
The rungs (with the feasibility critic's corrections baked in — do not relitigate):
status = Evicting,park_rung = 1, VM untouched). Ascent = one PG write over a newEvicting → Activelegality edge + a hostcancel_evict(clear the eviction-inflight marker, reset idle clocks). Covers the 10 s scanner gap and deletes the 8 s hold loop. Ships standalone first (days). This is the only new FSM edge in the overhaul (resolved decision: no new FSM states; in-flight visibility comes from op rows).status = Evicting,park_rung = 2; FC paused after a 38–53 ms diff capture whose upload runs in the background; NBD/netns/vsock/harness all intact in RAM). Ascent = hostunpark(FCPATCH /vm state=Resumed) + the sameEvicting → Activeedge — no harness respawn, noclaude --resume, nostart_agent, no transcript rewind. Critic corrections (C1), owned honestly: a paused VM frees zero RAM, so rung 2 buys interactive-return latency, not density — it is the state a descending session passes through (the diff is banked, making 2→3 an O(destroy) step) and briefly dwells in under mild pressure; under acute pressure the ladder skips it (1→3 directly). And because idle dwell p90 is 7.3 h, rung-2 dwell is capped (ENGRAM_PARK_PAUSED_MAX_DWELL_SECS, default 900) — multi-hour pauses resume into stale guest TCP (claude's API sockets); recoverable via the CLI's retries but not the clean "<100 ms and nothing notices" story, so we don't let sessions sit there.status = Idlevia the existingEvicting → Idleedge,park_rung = 3; VM destroyed, but the snapshot staging dir — state.bin, rolling memfile + diff chain, sidecar — and NBD backing state are retained on host NVMe). Ascent = normal resume path, but placement pins the parking host and the restore reads only local artifacts: target ~0.5–2 s. This is NEW machinery — the critic is explicit thatlive_attach.rsreattaches to live processes; "destroy but retain the data plane" is not that code. Device slots (NbdSlotAllocator,crates/engram-host-agent/src/disk_daemon/slot.rs) are a bounded pool: retain best-effort, release under slot pressure (ascent re-acquires).park_rung = 0): the full restore, the only rung that pays GCS — now the rare case for interactive sessions. Made faster separately byepic-transactional-snapshots-reviveandprefault-admission-control.Descent 2→3 frees RAM (the pressure signal's actual target); 3→4 frees disk + slots. Because the diff was banked and uploaded at rung 2, descent 2→3 is destroy-only and 3→4 is a pure local delete — the durable snapshot already exists. Rung-2/3 residency is charged against
allocatable_mib(ADR 0046,docs/adr/0046-placement-memory-reservation.md:176) viaram-ledger— this is the hard dependency that makes pressure-only mode safe to be the only mode.Dropped from v1 (critic consensus C10): the speculative
ensure_rungtyping-signal hint RPC. Rungs 1–2 cover the common case without it.Schema (new migration; migrations are checksum-immutable, so a new file — next free number,
deploy/migrations/0077_park_rung.sqlat time of writing, re-check the high-water mark at branch time)Wire/host-verb changes (clean break; coord + host roll together — mind WIRE_VERSION skew during the roll)
New
HostClient/SandboxBackendverbs (defaultsErr(SandboxError::InvalidSpec(...)), mirroringsnapshot_begin— NOT the silentOk(())shapepause/resumeuse today):cancel_evict(sandbox_id)— clear the hub's eviction-inflight marker + reset idle clocks (so the next tick doesn't instantly re-nominate a just-cancelled session). Host-agent side:HarnessHub(crates/engram-host-agent/src/harness.rs).park(sandbox_id) -> ParkOut { snapshot_id }— pause + diff capture + background upload, leaving the VM paused (the D5snapshot_beginpipeline minus the destroy handoff; confirm during implementation whethersnapshot_begincurrently resumes or leaves paused, and factor a shared helper inpooled_backend.rsrather than a second pipeline).unpark(sandbox_id)— FCPATCH /vm state=Resumed(+ any guest clock re-step the resumed guest needs — unverified, see below) + mark the hub entry live again.destroy_retain_local(sandbox_id) -> ParkedLocalOut— tear down FC process/jail/netns, retain the staging dir + NBD local state, write aparked.jsonrecord next to the artifacts (re-advertised in heartbeat after a host-agent restart, likeCheckpointRecord).Interlocks
try_acquires thesession_leasewith zero retries. Lease free ⇒ the scanner hasn't started ⇒cancel_evicton the host first (clock reset closes the re-nomination race), then CASEvicting → Active. Lease held ⇒ pipeline mid-flight ⇒ fall back to the existing hold (phase 1) / unpark at the rung-2 boundary (phase 2). The pipeline's existing K5 status re-check under the lease (idle_evictor.rs:37-39EvictOutcome doc) makes a lost race a no-op — verify it re-reads status after lease acquire, add if missing. Onceepic-session-op-loglands, cancel becomescancel(evict-op)and this bespoke interlock retires.Evictingno longer implies "pipeline pending"; the eviction scanner becomes the descent driver, keyed onpark_rung: rung 1 → run descent when pressure demands (or hard-TTL/backstop); rung 2 → enforce the dwell cap; the 20-attempt →HostLostbudget counts only actual transition attempts, never parked dwell.checkpoint.rs:219area already rejects concurrent captures — extend it topark_rung >= 2).HarnessHub::idle_sandboxes(harness.rs:816) must skip sandboxes at rung ≥ 1 (they are already descending; re-nomination is noise). Coordination with epic-binding-epoch-delivery: that epic's Phase 4 deletesHarnessHub::idle_sandboxesand the host eviction tick (crates/engram-host-agent/src/lib.rs:1252-1440) outright and moves idle detection coordinator-side (idle_detector.rs) — which is also where today's pressure-gate anchors (hostidle_evictor.rs:119-137,lib.rs:1341-1374) live. This epic's "skip rung ≥ 1" interlock and the pressure gate must therefore be expressed as detector-agnostic requirements, not as calls into the soon-to-be-deleted host code: whichever detector exists at land time (host-sideidle_sandboxestoday, coordinator-sideidle_detector.rsafter binding-epoch-delivery Phase 4 lands) must skip descending sessions, and the pressure gate moves with detection rather than being pinned to the host tick.parked.jsonre-advertised) but die with node replacement — the session is then a plain rung-4 Idle and resume falls back to GCS. No new durability class is invented; the banked snapshot is already remote.Implementation plan
ADR first (repo convention): author
docs/adr/00XX-parking-ladder.md(next free number; 0067 at time of writing) asProposedbefore phase 1 code; update it between phases with divergences; flip toAcceptedwith the commit chain at the end. Each phase is one PR on its own worktree.Phase 1 — Rung 1: cancellable nomination (ships standalone, days)
crates/engram-core/src/types/session.rs: addActiveto theEvictingarm ofcan_transition_to(:241), update the doc table (:205-209), extend the legal-edge test list (:588-607) with(Evicting, Active).crates/engram-core/src/traits/host_client.rs+traits/sandbox.rs: addcancel_evictwith anErr(InvalidSpec)default; implement inPooledBackend(clearmark_eviction_inflightstate + reset the hub idle clocks for the sandbox); wire through the host-agent gRPC server (crates/engram-host-agent/src/grpc_server.rs) and the coord-side host client +engram-protocol(WIRE_VERSION bump).crates/engram-coordinator/src/api/snapshot.rs: rewrite theSessionState::Evictingarm ofensure_active(:494) ascancel_evict_and_reactivate: zero-retrySessionLeaseGuard::try_acquire→ on success: re-read status,host.cancel_evict(sandbox_id),transition_session(Evicting → Active), emitStatusChanged, release; on lease-held: fall back to the existingensure_active_after_evicting_hold(deleted in phase 2).evict_session_to_state(crates/engram-coordinator/src/idle_evictor.rs:95-140): after lease acquire, a row no longerEvicting⇒EvictOutcome::Skipped.engram_evict_cancels_total, cancel-path latency histogram (ridestelemetry-restoration's export, but the instruments land here).just check+ sqlx cache; single PR.Phase 2 — Rung 2: parked-paused + dwell cap
0077_park_rung.sql(above);crates/engram-postgres/src/lib.rsrow mapping +MetadataStoreaccessors (set_park_rung,list_parked_over_dwell); regen.sqlx/.park/unparkverbs — factor the pause+diff+background-upload out of the post-eviction-finalizesnapshot_beginpath (theEvictionFinalizeRecordpipeline), not the current D5 shape: eviction-finalize-host-durable lands first via the op-log dependency chain, so by the time this phase is implementedsnapshot_beginis already the finalize-record-backed pipeline, andparkfactors out of that (crates/engram-host-agent/src/pooled_backend.rs);parkleaves the VM paused; the banked capture lands as a normal ADR 0028 checkpoint record so 2→3 needs no second capture. Guard the checkpoint driver andidle_sandboxesagainst parked sandboxes.crates/engram-coordinator/src/idle_evictor.rs): a claimedEvictingsession under pressure descends 1→2 (host.park→park_rung=2, parked_at=now()), or 1→3 directly under acute pressure (pre-phase-3: 1→4, the composed pipeline, unchanged). Scanner enforcesENGRAM_PARK_PAUSED_MAX_DWELL_SECS(default 900): over-dwell rung-2 sessions descend (2→3 once phase 3 lands; 2→4 until then).cancel_evict_and_reactivate—park_rung=2⇒host.unpark→Evicting → Active→ deliver. Noresume_session, no lease-retry ladder, no rewind.ensure_active_after_evicting_hold(+_for, constants :39-55, env knob, tests :2330-2415 — replaced by cancel/unpark tests in the same PR) — unless already deleted by epic-session-op-log Phase 1; whichever lands second ports the replacement tests.IDLE_DRAIN_GRACE_SECS,harness.rs:68-75) is for the terminal eviction capture;parkdoes not drain (the whole point is the harness stays alive). Descent 2→3/4 reuses the already-banked capture, so the drain special-case narrows to the direct-to-rung-4 path (record in the ADR whether it can be deleted outright — clean break preferred).Phase 3 — Rung 3: parked-local
destroy_retain_localonPooledBackend— destroy the VM/jail/netns, retain staging dir + NBD local state, writeparked.json; heartbeat advertises parked-local residency (disk bytes + slot holds) forram-ledger; release retained slots under slot-pool pressure.park_rung=3, parked_host_idand transitionsEvicting → Idle(existing edge — rung-3 ascent is a resume, soIdlestays honest).resume_sessionfast path —park_rung=3⇒ placement pinsparked_host_id; restore must read zero GCS (local staging already present; assert via metrics). If the pinned host is gone, clear the park columns and fall through to rung 4.parked.json, release slots,park_rung=0. Orphan-reap integration: host restart re-advertisesparked.json; a record with no matching PG row is swept (orphan_reap.rspattern).Phase 4 — Pressure-only becomes the only mode; clock-TTL reclaim deleted
ram-ledgermerged (parked + resident RAM charged toallocatable_mib) — otherwise the pressure gate lies and placement overcommits.crates/engram-host-agent/src/idle_evictor.rs: deletepressure_aware_from_env+ theENGRAM_IDLE_EVICT_PRESSURE_AWAREknob (:119-137);crates/engram-host-agent/src/lib.rs:1341-1374: the pressure gate becomes unconditional (Soft candidates nominate only under pressure; Hard always).ENGRAM_IDLE_EVICT_MEM_FLOOR_PCTstays tunable. Soft TTL remains the candidacy age only.Accepted.What gets deleted
ensure_active_after_evicting_hold/_for,DEFAULT_RESUME_EVICTING_HOLD_SECS,RESUME_EVICTING_POLL_INTERVAL,resume_evicting_hold_from_env,ENGRAM_RESUME_EVICTING_HOLD_SECS—crates/engram-coordinator/src/api/snapshot.rs:39-55, 494, 526-604+ tests :2330-2415 (phase 2, replacements land in the same PR).resume_session, so the 8×3 s retry ladder (api/snapshot.rs:219-249) stops firing on that path (the ladder itself stays for genuine rung-4 concurrency untilepic-session-op-logretires it).ENGRAM_IDLE_EVICT_PRESSURE_AWAREflag +pressure_aware_from_env(crates/engram-host-agent/src/idle_evictor.rs:119-137; gate atlib.rs:1341-1374becomes unconditional) (phase 4).recovered_from_checkpoint/rewind simply doesn't run on them (the rung-1-rewind code itself stays for rung-4 resumes).idle_detect_backstop, the eviction scanner (it becomes the descent driver), ADR 0028 periodic checkpoints, thesession_lease(until the op log subsumes it).Acceptance criteria
Baselines: prod 2026-06-17→07-01 (resume p50 12.2 s / p95 89.1 s; prompt→run_started p50 ≈24.5 s; ~7.5 evictions/day; lease collision 3–21 s; 45/45 resumes rewound).
status='evicting'with no lease held reachesrun_startedwith no VM teardown and no snapshot restore (assert: noresumed/recovered_from_checkpointevent); cancel-path added latency < 1 s. A prompt racing the in-flight pipeline still lands correctly (falls back, no orphan, no double-resume).Evicting → Active+ delivery in < 1 s end-to-end (host-side unpark < 200 ms), vs the 12.2 s p50 baseline; zero transcript rewind on rung-1/2 ascents; no session dwells parked-paused longer thanENGRAM_PARK_PAUSED_MAX_DWELL_SECS+ one scanner tick; under acute memory pressure the ladder skips rung 2 (observable viaengram_park_descents_total{from="1",to="3"}).IDLE_EVICT_KEPT_RESIDENT_TOTALrises; eviction count drops materially vs ~7.5/day); the lease-collision class (3–21 s waits) is absent from rung-1/2 ascents; placement never overcommits against parked residency (ram-ledger gauges reconcile).Testing & CI
just checklane): transition-table unit tests — extend the edge list atsession.rs:588-607with(Evicting, Active); assert(Idle, Active)remains illegal (no new edges beyond the pinned one).crates/engram-coordinator/tests/parking_ladder_live_pg.rs, mirroringplacement_reservation_live_pg.rsshape): cancel-vs-scanner race (lease interlock: exactly one wins, loser no-ops), dwell-cap descent, scanner budget not counting parked dwell, rung-3 host-gone fallback. KeepSQLX_OFFLINEcache in sync; new migration files only — applied migrations are checksum-immutable.crates/engram-sandbox-firecracker/tests/park_unpark.rs,#[ignore]'d, Linux+KVM): one park→unpark cycle on a tiny guest — assert the diff is banked, the guest process survives (marker file/uptime), and unpark resumes execution; one park→destroy_retain_local→local-restore cycle. Must be wired intoci.yml's--testlist (repo rule: FC tests not in the list never run). Size to the property, not realism: small guest, one cycle, no long sleeps — do not reinflate the FC lane.test-e2e-stack): prompt-during-evicting scenario — nominate, immediately prompt, assert active-without-rebuild. The deleted evicting-hold tests (api/snapshot.rs:2330-2415) are replaced by cancel-path coverage in the same change — never delete e2e/regression coverage without a landing replacement.nix develop -c cargo clippy --target aarch64-unknown-linux-musl -p engram-host-agent --all-targets(macOS clippy can't seecfg(target_os = "linux")paths).Guardrails
just checkbefore every commit; validate anyci.ymledit with actionlint/yaml.safe_load; no type/lint laundering; clean breaks over shims — the wire bump and flag deletion are intentional breaks (zero users), but coord rolls before hosts, so sequence the WIRE_VERSION bump to avoid the known serving-hosts-dropped skew wedge.cancel_evict) is control-plane-only and must work on all backends — implement it for VZ and Process (it's a hub-state operation, not a VMM one). Rungs 2–3 (park/unpark/destroy_retain_local) are FC-only: the trait defaults returnErr(InvalidSpec)(mirrorsnapshot_begin— never a silentOk(())), and the descent driver explicitly catchesInvalidSpecand falls through to the composed full eviction (rung 4), exactly as the D5 branch does today (idle_evictor.rs:234-243). VZ haspause/resume(engram-sandbox-vz/src/vm.rs:303,313) but no diff-snapshot machinery — do not fork the design to add a VZ park path; production (FC) drives the design.ensure_rungtyping-signal hint in v1 (critic C10); no new FSM states —Evicting → Activeis the only new edge, rungs live inpark_rung(decision 5); rung 2 is latency-only, capped, and skippable (critic C1); rung 3 is new machinery, notlive_attachreuse; capture jobs, warm-VM pre-spawn, and File-mode memory paths are other issues' scope (File mode gets deleted ingeneration-purge, not enabled here).snapshot_beginpipeline leaves the VM paused or resumes it before the destroy handoff (determines how much ofparkis a refactor vs new code); whether a long-paused FC guest needs an explicit clock re-step onResumed(vs KVM clock catch-up) — research the FC spec, don't guess-and-check; guest TCP staleness after multi-hour pause is the reasoned basis for the dwell-cap default (900 s) — tune from prod data, and record the measured value in the ADR.