Replace draft/bench/improve loop with acumen epoch + knowledge wiki - #13
Open
PauBadiaM wants to merge 10 commits into
Open
Replace draft/bench/improve loop with acumen epoch + knowledge wiki#13PauBadiaM wants to merge 10 commits into
acumen epoch + knowledge wiki#13PauBadiaM wants to merge 10 commits into
Conversation
Reshape the optimization loop into a resumable training epoch. `acumen epoch` benches the current arm on the training split, distils each task's runs into a persistent per-task wiki (observations.md + hypothesis.md, one [version][model] block per epoch from noskill up), creates or improves the skill from that wiki, then benches the new version on the held-out split. - New `wiki.py`: cumulative, resume-safe per-task notes (one agent per task, in parallel); terse-by-design prompt; filtered-source isolation. - Rework `improve.py`: reads the wiki + parent skill + staged train transcripts + filtered package source; creates v1 when no version exists, improves after. - New `epoch.py` + `acumen epoch`; add `acumen wiki`; retire `acumen draft`. - Rename the held-out split `test` -> `valid` (reserving `test` for a future train/valid/test split) across paths, tasks, report, check, review, taskgen, scaffold, prompts, and tests. - bench: free codex sandbox preflight probe so a namespace/sandbox failure is reported up front instead of per paid run; raise the auth-probe budget cap (PREFLIGHT_MAX_USD) so premium models can pass preflight. - Rewrite the README (minimal) and the shipped acumen self-skill for the new loop. Validated end-to-end on a toy package across six models; the skill lifted the weakest model's held-out pass rate without regressing the others. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #13 +/- ##
==========================================
- Coverage 78.44% 77.83% -0.61%
==========================================
Files 30 31 +1
Lines 5336 5821 +485
==========================================
+ Hits 4186 4531 +345
- Misses 1150 1290 +140
🚀 New features to boost your workflow:
|
…age sum The transcript/trajectory footer for Claude runs summed the session file's per-message usage. Each Claude turn re-sends the whole conversation, so every turn's usage already counts the cached prefix; summing across turns overcounts badly (a real run showed input/cached/output of 22/230109/2999 and a priced cost of $0.53 vs the $0.32 the run was actually billed). result.json was already correct — it records the authoritative ResultMessage.usage, whose priced cost matches Claude's own total_cost_usd. Thread that authoritative usage into the trajectory so the footer matches result.json: build_trajectory / from_claude_transcript / from_claude_records take a `usage` override, and the runner and LiveLog.finalize pass ResultMessage.usage. The per-message sum remains only as a fallback for rendering a bare transcript with no result. Also fix a shadowing bug where the loop's per-message `usage` clobbered the new parameter. Codex is unaffected: it reports usage once per turn.completed as a single total, which the mapper takes directly (never summed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`acumen fit` drives the epoch loop like training a model: it runs `acumen epoch` back to back, tracks held-out validation mean success, and stops on `--patience` (default 2, no strict improvement over the best-so-far) or the `--max-epochs` hard cap (default 10); `--epochs N` runs exactly N and disables early stopping. - New `training.py`: build the per-epoch training curve from the runs tree (epoch N's train arm = v(N-1)/noskill, valid arm = vN), write `training.csv` (mean success and mean cost per run, overall and per model, train and valid), and the pure early-stop rule (`patience_exhausted`, `epochs_since_best`, `best_version`). Rebuilt from disk each epoch, so `fit` is resumable. - Factor `_run_one_epoch` + `_prepare_pass` out of `_cmd_epoch` so `epoch` and `fit` share one epoch implementation; `fit` prepares the target/rates once. - Each epoch prints a tqdm-style line with train/valid success, best/patience. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
acumen bundled its own skill (guidance for using acumen) under `_skills/`, exposed via the `acumen-install-skills` console script. It had drifted out of date with the epoch/wiki/fit workflow, so drop it: delete `src/acumen/_skills/`, its `[project.scripts]` entry, and the README install note. Unrelated: `acumen ship` — which packages a *target* package's skill into a `<dist>-install-skills` script (`SHIP_INSTALL_TEMPLATE`, ship.py) — is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
codex exec prints this to stderr whenever its stdin is not a TTY (acumen wires it to /dev/null), then reads immediate EOF. It is harmless but reads as a confusing prompt during a run, so drop it in the stderr drain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`acumen epoch` and `acumen fit` now show one in-place progress bar per phase (bench train, wiki, an "inferring skill" spinner, bench valid), each filling per finished run with a running mean success rate and cumulative cost, in place of the per-run scrolling logs. - Bar unit is one matrix cell (model x task x rep); success = passed/done. - Bars by default; --verbose restores the full scrolling logs. - Output auto-detects: live \r bars on a TTY with --stream off, else a plain throttled-line fallback so files/CI stay readable. Genuine harness failures (provider-exhausted / sandbox-blocked) still surface to stderr. - bench/wiki/improve standalone commands are unchanged. update_wiki gains an optional on_plan(total) callback so the wiki bar has an accurate denominator (recorded tasks are skipped without a callback). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_cmd_fit called build_training_rows(runs, cfg, tasks) but the function takes two args and ignores tasks, so every fit run crashed while writing the training curve — after a full epoch had already been spent. The unit tests call it with two args, so the suite stayed green. Drop the stray argument at both call sites and add a regression test that drives the post-epoch path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex logs its own tracing to stderr (rollout writes after a session ends, sandbox-denied commands under approval_policy=never, malformed apply_patch attempts). None signal an acumen fault and all are kept in the run transcript, but each line is unique so StderrFilter never dedupes them — they flood the console and corrupt the \r progress bars. Drop lines matching the Codex tracing shape in _drain_stderr, keeping them in the noise sink so _sandbox_failure still detects real bwrap/landlock errors (which are bare and still print). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The phase bar mixed conventions without saying so: success was a running mean pass-rate while cost was a cumulative sum. Spell both out — "success N% (mean)" and "$X total" — so the numbers are unambiguous. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The improve step reported cost and turns but not how long it took, unlike the phase bars. Time the call and include it — "(2m41s, $5.24, 79 turns)" in bar mode, and "over N turns in 2m41s" in verbose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reshapes acumen's optimization loop from the manual
draft→bench→improvecycle into a single, resumable training epoch.acumen epochruns one epoch end to end:noskillbaseline onvalid).wiki/<task>/observations.md+hypothesis.md(one terse[version][model]block per epoch, fromnoskillup).noskillwiki + filtered source) or improve the latest, reading the wiki + parent skill + staged train transcripts.It resolves state from disk, so a crashed epoch resumes on the right step and a finished one starts the next.
Changes
wiki.py— cumulative, resume-safe per-task knowledge base; parallel agents; brevity-enforced prompt; filtered-source +discover_skills=Falseisolation;.armsmarker for idempotency.improve.py— wiki-driven, create-or-improve, filtered package source; keeps the held-out-split guard hook (renamedfind_valid_access/make_valid_guard).epoch.py+acumen epoch; addacumen wiki; retireacumen draft.test→valid(reservingtestfor a future train/valid/test split) across paths, tasks, report, check, review, taskgen, scaffold, prompts, and tests.codex sandboxpreflight probe so a namespace/sandbox failure is reported up front instead of once per paid run; raise the auth-probe budget cap (PREFLIGHT_MAX_USD) so premium models can pass preflight (pre-existing bug that also blockedacumen benchwith Opus)._skills/data/**) updated to the new loop.Validation
sensorloom) across six models (opus/sonnet/haiku + sol/terra/luna) and six generated tasks: the created skill lifted the weakest model's held-out pass rate (haiku 4/6 → 6/6 onvalid) without regressing the others.🤖 Generated with Claude Code