Conversation
Cleanup removed state/<id>.meta, state/<id>.status, and state/<id>.busy-state. The worker's own transcript survives under its harness session directory, but firstmate's side of the run did not: which model ran it, on which backend and endpoint, under which delivery mode and merge posture, the status events, and the last turn-activity record. Seven tasks finished in one home on the day this was written and each took that with it, and a reconstruction of 29 tasks across three repositories could go no further back for exactly this reason. The loss is irreversible and the copy is cheap. Copy those three files to data/<id>/record/ before the first removal. That directory already holds the task's brief and already survives cleanup, so a finished task ends up with one place holding what it was asked to do and what actually ran for it. Cleanup keeps doing its job; the evidence outlives it. The meta is retained whole rather than filtered. A filter is a list that rots: the next field added to the record would be dropped silently, which is the failure this path exists to end. Nothing secret rides along, because every secret firstmate holds for a task lives in a file of its own and none of those is copied. The status stream is the only part a task controls the size of, so it is capped at the newest 400 lines with the number of dropped ones disclosed in the record. Everything else is fixed-size, and no new growth axis appears, since data/<id>/ already exists per task and already retains a brief many times this size. Retention fails closed. Every copy is read back and compared against its source, the record is asserted complete before success, and a cleanup that cannot retain refuses with every record still in place rather than removing them anyway - silent loss is invisible until someone goes looking, by which time it is gone. It runs on the path that was already going to succeed, so the landed-work and uncommitted-work gates are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
When a task is torn down, firstmate removes
state/<id>.meta,state/<id>.status, andstate/<id>.busy-state. The worker's own transcript survives under its harness session directory, but the supervisor's view of the task does not: which model ran it, on which backend and endpoint, under which delivery mode and merge posture, the status-event stream, and the turn-activity record.The cost is measured rather than theoretical. Seven tasks finished in one home on the day this was written and each took its record with it. A separate reconstruction of 29 tasks across three repositories could go no further back for exactly this reason. The loss is irreversible and the copy is cheap.
What this does
Cleanup keeps doing its job, and the evidence now outlives it.
bin/fm-task-record-lib.shcopies the three record-bearing files intodata/<id>/record/before the first removal, wired into every path that removes them:bin/fm-backlog-transition-lib.sh, which removes the meta at the next session startThat directory already holds the task's brief and already survives cleanup, so a finished task ends up with one place holding what it was asked to do and what actually ran for it.
Design notes
The meta is retained whole rather than filtered. A filter is a list that rots, and the next field added to the record would be dropped silently, which is the failure this change exists to end. Nothing secret rides along: every secret firstmate holds for a task lives in a file of its own, and none of those is copied.
What bounds it. The record is four files per task: the meta, the status stream, the turn-activity record, and a small provenance file. It rides an existing growth axis rather than adding one, since
data/<id>/already exists per task and already retains a brief many times larger. Re-running cleanup replaces the record rather than appending to it, and staging left by a killed run is swept on the next attempt.Retention fails closed. Every copy is read back and compared against its source, the record is asserted complete before success, and a cleanup that cannot retain refuses with every record still in place rather than removing them anyway. Silent loss is invisible until someone goes looking, by which time it is gone. A comparison that cannot run is reported separately from a mismatch, because the two ask for different repairs.
Re-retention never degrades a complete record. If retention succeeds, the sources are removed, and a later step fails, the retry runs with those sources absent. Rather than staging empty files over good evidence, retention carries the existing record's files forward, gated on the existing record's meta being byte-identical to the live one so no other incarnation can donate evidence.
Safety
This touches a safety-critical path, so to state it explicitly: the landed-work and uncommitted-work gates are untouched and I verified it. Retention reads
state/and writesdata/, sits after those gates on the path that was already going to succeed, and changes no gate logic. Every refusal case in the suite still holds, including truly unpushed work, dirty worktrees, an unreachable forge, and live index locks.Verification
tests/fm-backlog-atomicity.test.shfor the replay path.bin/fm-lint.shclean with the pinned ShellCheck 0.11.0 and actionlint 1.7.12.Review, test, document, and lint ran through the no-mistakes pipeline and were green. Push and PR were completed outside that run: the run had already failed at push before a fork existed, and it then refused to resume because its preserved head differed from the clean local head. The branch published here is that exact preserved head, with no pipeline fix commit dropped.