Skip to content

Epic: the parking ladder — cancellable, pressure-driven eviction (Nominated → Parked-paused → Parked-local → Evicted-remote) #545

Description

@nikhilunni

Epic: the parking ladder — cancellable, pressure-driven eviction (Nominated → Parked-paused → Parked-local → Evicted-remote)

Part of the 2026-07 core-ops overhaul (deep-research synthesis). Tier 2. Size Epic.
Depends on: ram-ledger (hard, phases 2–4), epic-session-op-log (soft — phase 1 ships standalone; cancel becomes an op-cancel once the op log lands).
Related: eviction-finalize-host-durable (D5 durability of the banked diff), epic-transactional-snapshots-revive (rung-4 one-RPC Revive), prefault-admission-control (rung-4 tail), telemetry-restoration (rung metrics), epic-rpc-consolidation (the new host verbs land on the consolidated surface if it ships first), queue-fairness, epic-binding-epoch-delivery (its Phase 4 deletes the host-side idle detection plane this epic's rung-skip/pressure-gate interlocks live in today — coordinate the handoff).
Cross-references: epic-binding-epoch-delivery = #542 · epic-rpc-consolidation = #549 · epic-session-op-log = #543 · epic-transactional-snapshots-revive = #544 · eviction-finalize-host-durable = #529 · generation-purge = #530 · prefault-admission-control = #534 · queue-fairness = #537 · ram-ledger = #540 · telemetry-restoration = #526

Problem

Eviction is irreversible from the instant of nomination. The session FSM has no Evicting → Active edge — Evicting can only reach Idle | HostLost | Dead | Completed (crates/engram-core/src/types/session.rs:241, verified). So when the single most common "the user is here, keep the VM" signal arrives — a follow-up prompt during eviction — the coordinator's only move is to wait for a perfectly healthy VM to be snapshotted and destroyed, then rebuild it from the snapshot it just took. The prompt sits in ensure_active_after_evicting_hold (a 250 ms poll, 8 s default budget, crates/engram-coordinator/src/api/snapshot.rs:41,46,548-604), and because the eviction scanner runs on a 10 s tick (crates/engram-coordinator/src/idle_evictor.rs:926), a session can sit at Evicting with a fully live, untouched VM for up to 10 s during which cancelling the nomination would have been one PG write.

The rebuild the user then pays is the full resume pipeline: prod resume p50 12.2 s, p95 89.1 s; user-perceived prompt→run_started p50 ≈24.5 s (prod evidence, 2026-06-17→07-01 window). Worse, the evict-then-immediately-resume shape — precisely the interactive case — collides with the eviction's own session_lease (held through the D5 finalize upload), costing 3–21 s of lease-acquire retries (8 attempts × 3 s, api/snapshot.rs:232). And every full-resume rebuild rewinds the transcript: 45/45 prod resumes rolled back events (median 4, max 62), with visible turn replay.

Meanwhile the descent side fights the latency goal: soft-TTL nomination fires purely on a 300 s clock (crates/engram-host-agent/src/idle_evictor.rs:39) regardless of whether the host needs the RAM. The team's own fix — pressure-aware nomination (ADR 0034 addendum 2026-06-30) — ships default-off behind ENGRAM_IDLE_EVICT_PRESSURE_AWARE (host idle_evictor.rs:119-137), so prod still evicts warm interactive VMs on a clock, for zero density gain, and then makes the returning user pay for it.

Evidence

Prod numbers below are from the 2026-07-01 evidence pass (prod PG session_events, 2026-06-17→2026-07-01, 14-day window; single-tenant dogfood traffic, small n — indicative, not statistically strong).

  • Resume latency (idle→active, n=26 excluding instant-reattach outliers): p50 12.2 s, p90 69.4 s, p95 89.1 s, max 299.3 s. Prompt→run_started (n=21): p50 ≈24.5 s, p90 ≈140 s, max 1703 s.
  • Idle dwell before resume: p50 11.9 min, p90 7.3 h — the p50 is squarely rung-1/2 territory; the p90 is why rung-2 dwell must be capped.
  • Eviction volume: 105 evicted events / 14 d (~7.5/day); eviction pipeline p50 6.5 s, p95 80 s. 45 resume attempts; 45/45 emitted recovered_from_checkpoint with rolled_back > 0 (median 4, max 62 — visible turn replay).
  • Evict-vs-resume lease collision: 3–21 s (8 × 3 s lease-acquire retries against the D5 finalize upload holding the lease) — crates/engram-coordinator/src/api/snapshot.rs:219-249 (verified: for attempt in 0..8, 3 s default backoff at :231).
  • No Evicting → Active edge: crates/engram-core/src/types/session.rs:241 (verified) — Evicting => matches!(target, Idle | HostLost | Dead | Completed); doc table at :205-209; test edge list at :588-607.
  • The hold loop: ensure_active's Evicting arm dispatches to ensure_active_after_evicting_hold (crates/engram-coordinator/src/api/snapshot.rs:494, fn at :548-604; DEFAULT_RESUME_EVICTING_HOLD_SECS = 8 at :41, 250 ms poll at :46; tests at :2330-2415).
  • Nomination-to-pipeline gap: host nominates on a 10 s tick (crates/engram-host-agent/src/idle_evictor.rs:47); coord handler idle_eviction_candidates (crates/engram-coordinator/src/api/host_http.rs:893, route at api/mod.rs:73) only flips Active → Evicting; the eviction scanner sweeps status='evicting' on its own 10 s tick with a 20-attempt budget → HostLost (crates/engram-coordinator/src/idle_evictor.rs:908-930).
  • Pressure-aware nomination exists but ships dark: pressure_aware_from_env default OFF (crates/engram-host-agent/src/idle_evictor.rs:119-137), mem_pressure_check at :167; the gate is applied in the host nomination tick at crates/engram-host-agent/src/lib.rs:1341-1374 (IdleKind::Soft candidates dropped when not under pressure; Hard always nominated). ADR 0034 addendum (2026-06-30) documents the design and the default-off ship.
  • The cheap-pause primitive already exists: diff capture pauses the guest 38 ms (cold-boot track_dirty_pages) / 53 ms (post-restore) vs 2,090 ms full capture (ADR 0028 P1, dev-vm measured, 256 MiB guest; docs/adr/0028-eviction-durability-under-host-roll.md); the periodic diff-first checkpoint driver lives at crates/engram-host-agent/src/checkpoint.rs (spawn_checkpoint_driver, CheckpointChain).
  • FC pause/resume plumbing exists end to end: FirecrackerClient::pause/resume via PATCH /vm (crates/engram-sandbox-firecracker/src/client.rs:160,432-437), SandboxBackend::pause/resume (defaults Ok(()) — crates/engram-core/src/traits/sandbox.rs:397-411), FC overrides (crates/engram-sandbox-firecracker/src/lib.rs:4216,4233), HostClient::pause/resume (crates/engram-core/src/traits/host_client.rs:363,369). The D5 split (snapshot_begin/snapshot_wait) is the pause-side half of an eviction snapshot (traits/sandbox.rs:243-258; coord call site idle_evictor.rs:229, finish_eviction_background at :442).
  • cancel_evict does not exist anywhere in the tree (verified by grep).
  • Downstream hold loops that exist only because eviction is irreversible: the 60 s @ 500 ms mid-move delivery HOLD (crates/engram-coordinator/src/api/prompt.rs:137-168), the 5 s @ 200 ms reattach-forward budget (prompt.rs:30-32), the 10 s SEND_PROMPT_ATTACH_WAIT_SECS (crates/engram-host-agent/src/harness.rs:66).

Root cause

Idle is modeled as a cliff, not a ladder. The moment the host nominates, the session is on a one-way conveyor — nominate → snapshot → destroy → Idle → (full rebuild on return) — and every stage is welded to the next inside two non-cancellable pipelines (evict and resume) glued together by status polling and a wall-clock lease. Physically, eviction is reversible right up to destroy(): the VM is untouched at nomination, and even the capture is a 38–53 ms diff pause. The FSM just cannot express "the user came back."

Past fixes were bandages on this shape rather than changes to it: the soft TTL was raised 30 s → 300 s (ADR 0039 #20) because the clock kept evicting live users; the Evicting hold loop was added because prompts bounced 409 for minutes; the pressure-aware addendum was built — and then shipped default-off — because parked residency isn't accounted anywhere, so turning it on fleet-wide would let placement overcommit (the ADR 0046 allocatable_mib honesty problem, → ram-ledger). Each bandage narrowed a symptom; none made eviction cancellable or descent proportional to actual pressure.

Solution

Model idle as descending rungs driven only by the existing memory-pressure signal (mem_pressure_check, host idle_evictor.rs:167), each rung with a cheap ascent. Pressure-driven descent becomes the only mode (the ENGRAM_IDLE_EVICT_PRESSURE_AWARE flag is deleted, not defaulted-on); the soft clock TTL survives only as the candidacy signal (how long since idle), never as a reclaim trigger. Hard-TTL nomination and the coord's 30-min PG backstop (crates/engram-coordinator/src/idle_detect_backstop.rs) stay as the absolute residency ceiling.

The invariant made structural: eviction is reversible up to the moment its resources are actually reclaimed, and it descends only as far as pressure requires.

The rungs (with the feasibility critic's corrections baked in — do not relitigate):

  1. Rung 1 — Nominated (status = Evicting, park_rung = 1, VM untouched). Ascent = one PG write over a new Evicting → Active legality edge + a host cancel_evict (clear the eviction-inflight marker, reset idle clocks). Covers the 10 s scanner gap and deletes the 8 s hold loop. Ships standalone first (days). This is the only new FSM edge in the overhaul (resolved decision: no new FSM states; in-flight visibility comes from op rows).
  2. Rung 2 — Parked-paused (status = Evicting, park_rung = 2; FC paused after a 38–53 ms diff capture whose upload runs in the background; NBD/netns/vsock/harness all intact in RAM). Ascent = host unpark (FC PATCH /vm state=Resumed) + the same Evicting → Active edge — no harness respawn, no claude --resume, no start_agent, no transcript rewind. Critic corrections (C1), owned honestly: a paused VM frees zero RAM, so rung 2 buys interactive-return latency, not density — it is the state a descending session passes through (the diff is banked, making 2→3 an O(destroy) step) and briefly dwells in under mild pressure; under acute pressure the ladder skips it (1→3 directly). And because idle dwell p90 is 7.3 h, rung-2 dwell is capped (ENGRAM_PARK_PAUSED_MAX_DWELL_SECS, default 900) — multi-hour pauses resume into stale guest TCP (claude's API sockets); recoverable via the CLI's retries but not the clean "<100 ms and nothing notices" story, so we don't let sessions sit there.
  3. Rung 3 — Parked-local (status = Idle via the existing Evicting → Idle edge, park_rung = 3; VM destroyed, but the snapshot staging dir — state.bin, rolling memfile + diff chain, sidecar — and NBD backing state are retained on host NVMe). Ascent = normal resume path, but placement pins the parking host and the restore reads only local artifacts: target ~0.5–2 s. This is NEW machinery — the critic is explicit that live_attach.rs reattaches to live processes; "destroy but retain the data plane" is not that code. Device slots (NbdSlotAllocator, crates/engram-host-agent/src/disk_daemon/slot.rs) are a bounded pool: retain best-effort, release under slot pressure (ascent re-acquires).
  4. Rung 4 — Evicted-remote (today's Idle, park_rung = 0): the full restore, the only rung that pays GCS — now the rare case for interactive sessions. Made faster separately by epic-transactional-snapshots-revive and prefault-admission-control.

Descent 2→3 frees RAM (the pressure signal's actual target); 3→4 frees disk + slots. Because the diff was banked and uploaded at rung 2, descent 2→3 is destroy-only and 3→4 is a pure local delete — the durable snapshot already exists. Rung-2/3 residency is charged against allocatable_mib (ADR 0046, docs/adr/0046-placement-memory-reservation.md:176) via ram-ledger — this is the hard dependency that makes pressure-only mode safe to be the only mode.

Dropped from v1 (critic consensus C10): the speculative ensure_rung typing-signal hint RPC. Rungs 1–2 cover the common case without it.

Schema (new migration; migrations are checksum-immutable, so a new file — next free number, deploy/migrations/0077_park_rung.sql at time of writing, re-check the high-water mark at branch time)

-- Parking-ladder rung bookkeeping (ADR 00XX). park_rung is meaningful
-- only while status IN ('evicting','idle'): 0 = none (running / plain
-- idle), 1 = nominated, 2 = parked-paused, 3 = parked-local.
ALTER TABLE sessions ADD COLUMN park_rung SMALLINT NOT NULL DEFAULT 0;
ALTER TABLE sessions ADD COLUMN parked_at TIMESTAMPTZ;
ALTER TABLE sessions ADD COLUMN parked_host_id UUID;
CREATE INDEX sessions_parked_idx ON sessions (park_rung) WHERE park_rung > 0;

Wire/host-verb changes (clean break; coord + host roll together — mind WIRE_VERSION skew during the roll)

New HostClient / SandboxBackend verbs (defaults Err(SandboxError::InvalidSpec(...)), mirroring snapshot_begin — NOT the silent Ok(()) shape pause/resume use today):

  • cancel_evict(sandbox_id) — clear the hub's eviction-inflight marker + reset idle clocks (so the next tick doesn't instantly re-nominate a just-cancelled session). Host-agent side: HarnessHub (crates/engram-host-agent/src/harness.rs).
  • park(sandbox_id) -> ParkOut { snapshot_id } — pause + diff capture + background upload, leaving the VM paused (the D5 snapshot_begin pipeline minus the destroy handoff; confirm during implementation whether snapshot_begin currently resumes or leaves paused, and factor a shared helper in pooled_backend.rs rather than a second pipeline).
  • unpark(sandbox_id) — FC PATCH /vm state=Resumed (+ any guest clock re-step the resumed guest needs — unverified, see below) + mark the hub entry live again.
  • destroy_retain_local(sandbox_id) -> ParkedLocalOut — tear down FC process/jail/netns, retain the staging dir + NBD local state, write a parked.json record next to the artifacts (re-advertised in heartbeat after a host-agent restart, like CheckpointRecord).

Interlocks

  • Race safety without the op log (phase 1): cancel first try_acquires the session_lease with zero retries. Lease free ⇒ the scanner hasn't started ⇒ cancel_evict on the host first (clock reset closes the re-nomination race), then CAS Evicting → Active. Lease held ⇒ pipeline mid-flight ⇒ fall back to the existing hold (phase 1) / unpark at the rung-2 boundary (phase 2). The pipeline's existing K5 status re-check under the lease (idle_evictor.rs:37-39 EvictOutcome doc) makes a lost race a no-op — verify it re-reads status after lease acquire, add if missing. Once epic-session-op-log lands, cancel becomes cancel(evict-op) and this bespoke interlock retires.
  • Scanner semantics: Evicting no longer implies "pipeline pending"; the eviction scanner becomes the descent driver, keyed on park_rung: rung 1 → run descent when pressure demands (or hard-TTL/backstop); rung 2 → enforce the dwell cap; the 20-attempt → HostLost budget counts only actual transition attempts, never parked dwell.
  • Checkpoint driver: the periodic ADR 0028 checkpoint driver must skip parked sandboxes (a paused VM has no new dirty state; the in-flight-capture guard at checkpoint.rs:219 area already rejects concurrent captures — extend it to park_rung >= 2).
  • Idle detection: HarnessHub::idle_sandboxes (harness.rs:816) must skip sandboxes at rung ≥ 1 (they are already descending; re-nomination is noise). Coordination with epic-binding-epoch-delivery: that epic's Phase 4 deletes HarnessHub::idle_sandboxes and the host eviction tick (crates/engram-host-agent/src/lib.rs:1252-1440) outright and moves idle detection coordinator-side (idle_detector.rs) — which is also where today's pressure-gate anchors (host idle_evictor.rs:119-137, lib.rs:1341-1374) live. This epic's "skip rung ≥ 1" interlock and the pressure gate must therefore be expressed as detector-agnostic requirements, not as calls into the soon-to-be-deleted host code: whichever detector exists at land time (host-side idle_sandboxes today, coordinator-side idle_detector.rs after binding-epoch-delivery Phase 4 lands) must skip descending sessions, and the pressure gate moves with detection rather than being pinned to the host tick.
  • Host restart / node death: rung-2 VMs die with the host-agent process boundary like any live VM (survivor reattach applies as today); rung-3 artifacts survive a pod roll (parked.json re-advertised) but die with node replacement — the session is then a plain rung-4 Idle and resume falls back to GCS. No new durability class is invented; the banked snapshot is already remote.

Implementation plan

ADR first (repo convention): author docs/adr/00XX-parking-ladder.md (next free number; 0067 at time of writing) as Proposed before phase 1 code; update it between phases with divergences; flip to Accepted with the commit chain at the end. Each phase is one PR on its own worktree.

Phase 1 — Rung 1: cancellable nomination (ships standalone, days)

  1. crates/engram-core/src/types/session.rs: add Active to the Evicting arm of can_transition_to (:241), update the doc table (:205-209), extend the legal-edge test list (:588-607) with (Evicting, Active).
  2. crates/engram-core/src/traits/host_client.rs + traits/sandbox.rs: add cancel_evict with an Err(InvalidSpec) default; implement in PooledBackend (clear mark_eviction_inflight state + reset the hub idle clocks for the sandbox); wire through the host-agent gRPC server (crates/engram-host-agent/src/grpc_server.rs) and the coord-side host client + engram-protocol (WIRE_VERSION bump).
  3. crates/engram-coordinator/src/api/snapshot.rs: rewrite the SessionState::Evicting arm of ensure_active (:494) as cancel_evict_and_reactivate: zero-retry SessionLeaseGuard::try_acquire → on success: re-read status, host.cancel_evict(sandbox_id), transition_session(Evicting → Active), emit StatusChanged, release; on lease-held: fall back to the existing ensure_active_after_evicting_hold (deleted in phase 2).
  4. Verify/add the pipeline's status re-check under the lease in evict_session_to_state (crates/engram-coordinator/src/idle_evictor.rs:95-140): after lease acquire, a row no longer Evicting ⇒ EvictOutcome::Skipped.
  5. Metrics: engram_evict_cancels_total, cancel-path latency histogram (rides telemetry-restoration's export, but the instruments land here).
  6. Tests (below) + just check + sqlx cache; single PR.

Phase 2 — Rung 2: parked-paused + dwell cap

  1. Migration 0077_park_rung.sql (above); crates/engram-postgres/src/lib.rs row mapping + MetadataStore accessors (set_park_rung, list_parked_over_dwell); regen .sqlx/.
  2. Host: park/unpark verbs — factor the pause+diff+background-upload out of the post-eviction-finalize snapshot_begin path (the EvictionFinalizeRecord pipeline), not the current D5 shape: eviction-finalize-host-durable lands first via the op-log dependency chain, so by the time this phase is implemented snapshot_begin is already the finalize-record-backed pipeline, and park factors out of that (crates/engram-host-agent/src/pooled_backend.rs); park leaves the VM paused; the banked capture lands as a normal ADR 0028 checkpoint record so 2→3 needs no second capture. Guard the checkpoint driver and idle_sandboxes against parked sandboxes.
  3. Coord descent driver (crates/engram-coordinator/src/idle_evictor.rs): a claimed Evicting session under pressure descends 1→2 (host.park → park_rung=2, parked_at=now()), or 1→3 directly under acute pressure (pre-phase-3: 1→4, the composed pipeline, unchanged). Scanner enforces ENGRAM_PARK_PAUSED_MAX_DWELL_SECS (default 900): over-dwell rung-2 sessions descend (2→3 once phase 3 lands; 2→4 until then).
  4. Ascent: extend cancel_evict_and_reactivate — park_rung=2 ⇒ host.unpark → Evicting → Active → deliver. No resume_session, no lease-retry ladder, no rewind.
  5. Delete ensure_active_after_evicting_hold (+_for, constants :39-55, env knob, tests :2330-2415 — replaced by cancel/unpark tests in the same PR) — unless already deleted by epic-session-op-log Phase 1; whichever lands second ports the replacement tests.
  6. ADR 0052 idle-drain note: the drain-claude-before-snapshot grace (IDLE_DRAIN_GRACE_SECS, harness.rs:68-75) is for the terminal eviction capture; park does not drain (the whole point is the harness stays alive). Descent 2→3/4 reuses the already-banked capture, so the drain special-case narrows to the direct-to-rung-4 path (record in the ADR whether it can be deleted outright — clean break preferred).

Phase 3 — Rung 3: parked-local

  1. Host: destroy_retain_local on PooledBackend — destroy the VM/jail/netns, retain staging dir + NBD local state, write parked.json; heartbeat advertises parked-local residency (disk bytes + slot holds) for ram-ledger; release retained slots under slot-pool pressure.
  2. Coord: descent 2→3 (and 1→3 under acute pressure) sets park_rung=3, parked_host_id and transitions Evicting → Idle (existing edge — rung-3 ascent is a resume, so Idle stays honest).
  3. Ascent: resume_session fast path — park_rung=3 ⇒ placement pins parked_host_id; restore must read zero GCS (local staging already present; assert via metrics). If the pinned host is gone, clear the park columns and fall through to rung 4.
  4. GC / descent 3→4: delete local artifacts + parked.json, release slots, park_rung=0. Orphan-reap integration: host restart re-advertises parked.json; a record with no matching PG row is swept (orphan_reap.rs pattern).

Phase 4 — Pressure-only becomes the only mode; clock-TTL reclaim deleted

  1. Requires ram-ledger merged (parked + resident RAM charged to allocatable_mib) — otherwise the pressure gate lies and placement overcommits.
  2. crates/engram-host-agent/src/idle_evictor.rs: delete pressure_aware_from_env + the ENGRAM_IDLE_EVICT_PRESSURE_AWARE knob (:119-137); crates/engram-host-agent/src/lib.rs:1341-1374: the pressure gate becomes unconditional (Soft candidates nominate only under pressure; Hard always). ENGRAM_IDLE_EVICT_MEM_FLOOR_PCT stays tunable. Soft TTL remains the candidacy age only.
  3. Delete the phase-1 lease-held hold fallback if any remnant survived phase 2.
  4. Prod validation window against the baselines below; flip the ADR to Accepted.

What gets deleted

  • ensure_active_after_evicting_hold / _for, DEFAULT_RESUME_EVICTING_HOLD_SECS, RESUME_EVICTING_POLL_INTERVAL, resume_evicting_hold_from_env, ENGRAM_RESUME_EVICTING_HOLD_SECS — crates/engram-coordinator/src/api/snapshot.rs:39-55, 494, 526-604 + tests :2330-2415 (phase 2, replacements land in the same PR).
  • The evict-vs-resume lease collision class for interactive returns: rung-1/2 ascents never call resume_session, so the 8×3 s retry ladder (api/snapshot.rs:219-249) stops firing on that path (the ladder itself stays for genuine rung-4 concurrency until epic-session-op-log retires it).
  • Clock-TTL reclaim: the unconditional soft-TTL nomination path and the ENGRAM_IDLE_EVICT_PRESSURE_AWARE flag + pressure_aware_from_env (crates/engram-host-agent/src/idle_evictor.rs:119-137; gate at lib.rs:1341-1374 becomes unconditional) (phase 4).
  • The transcript-rewind-on-every-return pathology for rungs 1–2 (45/45 prod resumes rolled back events): parked ascents never leave VM state, so recovered_from_checkpoint/rewind simply doesn't run on them (the rung-1-rewind code itself stays for rung-4 resumes).
  • The idle-drain special case narrows (see phase 2 step 6); ADR decides delete-vs-narrow.
  • NOT deleted (explicitly): hard-TTL nomination, the coord 30-min idle_detect_backstop, the eviction scanner (it becomes the descent driver), ADR 0028 periodic checkpoints, the session_lease (until the op log subsumes it).

Acceptance criteria

Baselines: prod 2026-06-17→07-01 (resume p50 12.2 s / p95 89.1 s; prompt→run_started p50 ≈24.5 s; ~7.5 evictions/day; lease collision 3–21 s; 45/45 resumes rewound).

  • Phase 1: a prompt arriving while status='evicting' with no lease held reaches run_started with no VM teardown and no snapshot restore (assert: no resumed/recovered_from_checkpoint event); cancel-path added latency < 1 s. A prompt racing the in-flight pipeline still lands correctly (falls back, no orphan, no double-resume).
  • Phase 2: ascent from parked-paused completes Evicting → Active + delivery in < 1 s end-to-end (host-side unpark < 200 ms), vs the 12.2 s p50 baseline; zero transcript rewind on rung-1/2 ascents; no session dwells parked-paused longer than ENGRAM_PARK_PAUSED_MAX_DWELL_SECS + one scanner tick; under acute memory pressure the ladder skips rung 2 (observable via engram_park_descents_total{from="1",to="3"}).
  • Phase 3: same-host rung-3 ascent < 2 s with zero GCS reads on the restore path (assert via metrics/logs); parking-host-gone falls back to rung 4 cleanly; rung-3 artifacts survive a host-agent pod restart (re-advertised) and their loss on node death degrades to rung 4, never to an error.
  • Phase 4: on a host with free RAM above the floor, soft-idle sessions are never evicted (IDLE_EVICT_KEPT_RESIDENT_TOTAL rises; eviction count drops materially vs ~7.5/day); the lease-collision class (3–21 s waits) is absent from rung-1/2 ascents; placement never overcommits against parked residency (ram-ledger gauges reconcile).
  • Rung-2/3 ascent-latency targets (<100 ms host-side unpark, ~0.5–1 s local restore) are engineering estimates, not promises — measure and record actuals in the ADR (the critic specifically flagged the <100 ms headline as over-credited; the honest headline is "interactive returns stop paying the 12–89 s rebuild").

Testing & CI

  • engram-core (just check lane): transition-table unit tests — extend the edge list at session.rs:588-607 with (Evicting, Active); assert (Idle, Active) remains illegal (no new edges beyond the pinned one).
  • coordinator live-PG (new crates/engram-coordinator/tests/parking_ladder_live_pg.rs, mirroring placement_reservation_live_pg.rs shape): cancel-vs-scanner race (lease interlock: exactly one wins, loser no-ops), dwell-cap descent, scanner budget not counting parked dwell, rung-3 host-gone fallback. Keep SQLX_OFFLINE cache in sync; new migration files only — applied migrations are checksum-immutable.
  • FC integration (new crates/engram-sandbox-firecracker/tests/park_unpark.rs, #[ignore]'d, Linux+KVM): one park→unpark cycle on a tiny guest — assert the diff is banked, the guest process survives (marker file/uptime), and unpark resumes execution; one park→destroy_retain_local→local-restore cycle. Must be wired into ci.yml's --test list (repo rule: FC tests not in the list never run). Size to the property, not realism: small guest, one cycle, no long sleeps — do not reinflate the FC lane.
  • e2e stack (test-e2e-stack): prompt-during-evicting scenario — nominate, immediately prompt, assert active-without-rebuild. The deleted evicting-hold tests (api/snapshot.rs:2330-2415) are replaced by cancel-path coverage in the same change — never delete e2e/regression coverage without a landing replacement.
  • Cross-compile check: host-agent changes are Linux-heavy — nix develop -c cargo clippy --target aarch64-unknown-linux-musl -p engram-host-agent --all-targets (macOS clippy can't see cfg(target_os = "linux") paths).

Guardrails

  • Repo conventions: ADR bookends (Proposed before phase-1 code → Accepted with the commit chain); one phase = one PR on a dedicated worktree; one logical change per commit; just check before every commit; validate any ci.yml edit with actionlint/yaml.safe_load; no type/lint laundering; clean breaks over shims — the wire bump and flag deletion are intentional breaks (zero users), but coord rolls before hosts, so sequence the WIRE_VERSION bump to avoid the known serving-hosts-dropped skew wedge.
  • VZ / Process backend story (decision 11): rung 1 (cancel_evict) is control-plane-only and must work on all backends — implement it for VZ and Process (it's a hub-state operation, not a VMM one). Rungs 2–3 (park/unpark/destroy_retain_local) are FC-only: the trait defaults return Err(InvalidSpec) (mirror snapshot_begin — never a silent Ok(())), and the descent driver explicitly catches InvalidSpec and falls through to the composed full eviction (rung 4), exactly as the D5 branch does today (idle_evictor.rs:234-243). VZ has pause/resume (engram-sandbox-vz/src/vm.rs:303,313) but no diff-snapshot machinery — do not fork the design to add a VZ park path; production (FC) drives the design.
  • Pinned decisions — do not relitigate: no ensure_rung typing-signal hint in v1 (critic C10); no new FSM states — Evicting → Active is the only new edge, rungs live in park_rung (decision 5); rung 2 is latency-only, capped, and skippable (critic C1); rung 3 is new machinery, not live_attach reuse; capture jobs, warm-VM pre-spawn, and File-mode memory paths are other issues' scope (File mode gets deleted in generation-purge, not enabled here).
  • Do not touch: ADR 0028 checkpoint-chain semantics beyond the skip-parked guard; the hard-TTL/backstop ceiling (it is what makes suppressing soft nominations safe); existing e2e eviction/resume coverage except with same-change replacements.
  • Unverified — confirm during implementation: whether the D5 snapshot_begin pipeline leaves the VM paused or resumes it before the destroy handoff (determines how much of park is a refactor vs new code); whether a long-paused FC guest needs an explicit clock re-step on Resumed (vs KVM clock catch-up) — research the FC spec, don't guess-and-check; guest TCP staleness after multi-hour pause is the reasoned basis for the dwell-cap default (900 s) — tune from prod data, and record the measured value in the ADR.

Activity

  1. added
    core-ops-overhaul2026-07 core-ops overhaul (deep-research synthesis)
    tier-2Core-ops overhaul: phased epic (ADR bookends)
    epicMulti-phase epic
    on Jul 1, 2026
  2. nikhilunni commented on Jul 7, 2026

    @nikhilunni
    ContributorAuthor

    Status (2026-07-07): Rung 1 SHIPPED (PR #581 — cancel-evict 18s → 0.043s prod-verified). Rung 2 SHIPPED (PR #585 + #596 + firecracker fork PR #14 for the upstream vsock RX-gate bug — un-pause 0.042s prod-verified). The evicting-hold deletion landed via #543's kernel (PR #599). Remaining: Phase 3 (rung-3 parked-local — executes under #548's peer/local machinery per the cross-epic split), Phase 4 (pressure-only mode + delete clock-TTL reclaim; needs the #540 RAM-ledger gauges reconciled against parked residency).

  3. added theissue type on Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    core-ops-overhaul2026-07 core-ops overhaul (deep-research synthesis)epicMulti-phase epictier-2Core-ops overhaul: phased epic (ADR bookends)

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions