Skip to content

feat(bin): keep the supervisor's record of a task past cleanup - #4553

Closed
avi2d wants to merge 4 commits into
kunchenguid:mainfrom
avi2d:fm/firstmate-keep-task-record-01
Closed

avi2d wants to merge 4 commits into
kunchenguid:mainfrom
avi2d:fm/firstmate-keep-task-record-01

Conversation

@avi2d

@avi2d avi2d commented Sep 15, 2026

Copy link
Copy Markdown

The problem

When a task is torn down, firstmate removes state/<id>.meta, state/<id>.status, and state/<id>.busy-state. The worker's own transcript survives under its harness session directory, but the supervisor's view of the task does not: which model ran it, on which backend and endpoint, under which delivery mode and merge posture, the status-event stream, and the turn-activity record.

The cost is measured rather than theoretical. Seven tasks finished in one home on the day this was written and each took its record with it. A separate reconstruction of 29 tasks across three repositories could go no further back for exactly this reason. The loss is irreversible and the copy is cheap.

What this does

Cleanup keeps doing its job, and the evidence now outlives it. bin/fm-task-record-lib.sh copies the three record-bearing files into data/<id>/record/ before the first removal, wired into every path that removes them:

  • the main teardown tail
  • the remote second mate teardown, ahead of the registry-row removal
  • the interrupted-cleanup replay in bin/fm-backlog-transition-lib.sh, which removes the meta at the next session start

That directory already holds the task's brief and already survives cleanup, so a finished task ends up with one place holding what it was asked to do and what actually ran for it.

Design notes

The meta is retained whole rather than filtered. A filter is a list that rots, and the next field added to the record would be dropped silently, which is the failure this change exists to end. Nothing secret rides along: every secret firstmate holds for a task lives in a file of its own, and none of those is copied.

What bounds it. The record is four files per task: the meta, the status stream, the turn-activity record, and a small provenance file. It rides an existing growth axis rather than adding one, since data/<id>/ already exists per task and already retains a brief many times larger. Re-running cleanup replaces the record rather than appending to it, and staging left by a killed run is swept on the next attempt.

Retention fails closed. Every copy is read back and compared against its source, the record is asserted complete before success, and a cleanup that cannot retain refuses with every record still in place rather than removing them anyway. Silent loss is invisible until someone goes looking, by which time it is gone. A comparison that cannot run is reported separately from a mismatch, because the two ask for different repairs.

Re-retention never degrades a complete record. If retention succeeds, the sources are removed, and a later step fails, the retry runs with those sources absent. Rather than staging empty files over good evidence, retention carries the existing record's files forward, gated on the existing record's meta being byte-identical to the live one so no other incarnation can donate evidence.

Safety

This touches a safety-critical path, so to state it explicitly: the landed-work and uncommitted-work gates are untouched and I verified it. Retention reads state/ and writes data/, sits after those gates on the path that was already going to succeed, and changes no gate logic. Every refusal case in the suite still holds, including truly unpushed work, dirty worktrees, an unreachable forge, and live index locks.

Verification

  • Full teardown suite green, including four new behavioral tests for retention, and new coverage in tests/fm-backlog-atomicity.test.sh for the replay path.
  • Remote second mate lifecycle end-to-end green, covering the remote retention call site.
  • Each new check was watched failing against a real violation before being trusted: removing the retention call turned the new tests red, corrupting a copy after writing made teardown refuse, and removing the remote-path call turned that assertion red.
  • bin/fm-lint.sh clean with the pinned ShellCheck 0.11.0 and actionlint 1.7.12.

Review, test, document, and lint ran through the no-mistakes pipeline and were green. Push and PR were completed outside that run: the run had already failed at push before a fork existed, and it then refused to resume because its preserved head differed from the clean local head. The branch published here is that exact preserved head, with no pipeline fix commit dropped.

Cleanup removed state/<id>.meta, state/<id>.status, and state/<id>.busy-state.
The worker's own transcript survives under its harness session directory, but
firstmate's side of the run did not: which model ran it, on which backend and
endpoint, under which delivery mode and merge posture, the status events, and
the last turn-activity record. Seven tasks finished in one home on the day this
was written and each took that with it, and a reconstruction of 29 tasks across
three repositories could go no further back for exactly this reason. The loss is
irreversible and the copy is cheap.

Copy those three files to data/<id>/record/ before the first removal. That
directory already holds the task's brief and already survives cleanup, so a
finished task ends up with one place holding what it was asked to do and what
actually ran for it. Cleanup keeps doing its job; the evidence outlives it.

The meta is retained whole rather than filtered. A filter is a list that rots:
the next field added to the record would be dropped silently, which is the
failure this path exists to end. Nothing secret rides along, because every
secret firstmate holds for a task lives in a file of its own and none of those
is copied.

The status stream is the only part a task controls the size of, so it is capped
at the newest 400 lines with the number of dropped ones disclosed in the record.
Everything else is fixed-size, and no new growth axis appears, since data/<id>/
already exists per task and already retains a brief many times this size.

Retention fails closed. Every copy is read back and compared against its source,
the record is asserted complete before success, and a cleanup that cannot retain
refuses with every record still in place rather than removing them anyway -
silent loss is invisible until someone goes looking, by which time it is gone.
It runs on the path that was already going to succeed, so the landed-work and
uncommitted-work gates are untouched.
@avi2d avi2d closed this Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant