Repository navigation
fix(skills): leave agent-enhanced files uncommitted for human review (fixes #738) - #743
Daniel-park10 wants to merge 2 commits into
Conversation
mlieberman85
left a comment
There was a problem hiding this comment.
Thanks @Daniel-park10. This takes option 1 from #738, and the new text matches framework-design 15.7. One ordering problem remains: in both skills, the PR is opened while the agent's edits are still uncommitted, so the PR doesn't contain them. That matters most in /darnit-remediate, so I'm requesting changes.
Tested: tests/darnit/server/ on main (aae02e4) with this branch merged in: 249 passed. The added lines are ASCII. Neither the other skills nor the docs still tell the agent to make a commit. The ordering problem below comes from reading the skill text against create_remediation_pr and git_state.py; I didn't run an agent through it.
Blocking: the PR doesn't contain the edits. create_remediation_pr pushes the run's branch as committed, so files left uncommitted are not in the PR.
- In
/darnit-remediate, step 3b fixes broken syntax, such as malformed workflow YAML, and step 4 then offers the PR. The PR goes out with the broken file the agent just found. - In
/darnit-comply, steps 7-9 judge, confirm and re-audit the enhanced working-tree files. The PR then carries the generated versions, which weren't the ones judged.
The person can't simply commit the files and then ask for the PR either. create_remediation_pr refuses a branch with a commit that lacks the Darnit-Remediation-Run trailer (git_operations.py:537, via foreign_commits in git_state.py:182), so a plain commit blocks the tool.
Asking the person to add the trailer to their own commit would get past the check, but the trailer is what marks content remediation wrote, so that would make the record wrong.
Before offering the PR, both skills should list the files left uncommitted and ask the person to review them. Then the agent should say plainly that the PR will not contain them: the person commits and pushes them to the PR branch after it is opened. In /darnit-comply, add that the judgments describe the uncommitted versions. For /darnit-remediate, a generated file that needed a syntax fix is a template bug, and the summary should say so rather than offer a PR that carries the broken file without a warning. If you think the tools should instead commit person-approved edits, raise it on #738, since that changes what the trailer means.
Smaller:
- The darnit-comply text says
commit_remediation_changes"commits only unchanged files from the run manifest". "Files whose content still matches what remediation wrote (framework-design 15.7)" would be clearer, because "unchanged" can be read as "files the run didn't change". test_remediation_skills_instruct_uncommitted_edits_for_reviewchecks only for two phrases. Once the ordering is fixed, a check that each skill names the uncommitted files before it offers the PR would guard what matters.
Drafted with AI assistance (Claude), reviewed and filed by me.
|
@mlieberman85 thanks for pointing the ordering problem with create_remediation_pr. In /darnit-comply, it explicitly notes that the judgments describe these uncommitted versions, and updated the wording for commit_remediation_changes to match framework-design 15.7. In /darnit-remediate, any syntax fix is now explicitly called out as a template bug before PR creation. Also added a test in test_procedures.py ensuring the uncommitted file warning precedes the PR offer. |
e256a6f to
b3a25f1
Compare
mlieberman85
left a comment
There was a problem hiding this comment.
Thanks @Daniel-park10. Since the change is in what the agent does, I tested it with an agent.
Tested on main (c39bb37), and on main with this branch merged in. Each side ran /darnit:darnit-comply headless in Claude Code on whiskeytastingfoundation/wtf-frontend at 939e838. Setup on both sides:
- a checkout with no remote;
- darnit in a throwaway HOME, with no platform token;
- recorded answers, every file change approved, and no PR wanted.
Unlike our usual replay, the agent could edit files in the checkout, so step 6 ran.
| main | here | |
|---|---|---|
| files the agent enhanced in step 6 | 6 | 5 |
commit_remediation_changes calls |
2, both refused: "Conflict: files changed after remediation wrote them. Nothing was committed." | none |
| enhanced files at the end | uncommitted | uncommitted |
| final report | lists them as uncommitted and suggests git add -A && git commit |
lists them under "Uncommitted files for your review", says that a PR from the branch would not contain them until they are committed and pushed, and that the judgments describe the working-tree versions |
Both sides had the same remediation outcomes, the same 20 changed files and the same final audit. I ran one run per side, and the agent's wording varies from run to run.
Full suite on the merged tree: 6000 passed, 11 skipped; ruff check and validate_sync.py pass.
Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.
|
Tested merged with current main on a VM: skill tests pass. Only CHANGELOG.md conflicts no please rebase and I will merge. Assisted by Claude Code |
…ixes darnitdevorg#738) In /darnit-comply and /darnit-remediate, instruct the agent to leave enhanced files and syntax fixes uncommitted in the working tree and report them to the person for review and commit. This preserves darnit's rule that it never commits content it did not write (framework-design 15.6 / 15.7) and eliminates instructions to 'make a new commit' where MCP agents lack shell access and commit_remediation_changes deliberately commits only unmodified manifest files. Add regression test in test_procedures.py asserting no skill instructs the agent to make a new commit. Fixes darnitdevorg#738 Signed-off-by: Daniel-park10 <dev10.tennyson@gmail.com>
…itdevorg#738) Instruct /darnit-comply and /darnit-remediate to list uncommitted working-tree files for review before offering PR creation, and state plainly that the pull request will not contain them. Because create_remediation_pr pushes only what remediation committed, uncommitted files must be reviewed, committed, and pushed to the PR branch by the maintainer after it is opened. In /darnit-comply, clarify that step 7-8 judgments describe these uncommitted versions and update the commit_remediation_changes wording to match framework design 15.7. In /darnit-remediate, explicitly flag generated template syntax fixes as template bugs before offering the PR. Add procedure tests ensuring uncommitted file warnings precede the PR creation call. Signed-off-by: Daniel-park10 <dev10.tennyson@gmail.com>
b3a25f1 to
43a4af9
Compare
|
@Marc-cn rebased and resolved the CHANGELOG.md conflict |
mlieberman85
left a comment
There was a problem hiding this comment.
Thanks @Daniel-park10. The rebase dismissed my approval, so here it is again. Compared with b3a25f1, only the CHANGELOG context changed: the skill text and the tests are the ones I ran an agent through last time. I didn't rerun the agent. CI is green on this head, which is on current main (714b3e6).
Drafted with AI assistance (Claude), reviewed and filed by me.
Summary
/darnit-complyand/darnit-remediatepreviously asked the agent to make a new commit after enhancing generated templates or fixing syntax. In headless MCP setups without shell access, the agent has no git commit tool. On top of that,commit_remediation_changesintentionally commits only files that match what remediation wrote (framework-design 15.6 / 15.7), so any edited file gets flagged as a conflict and left uncommitted.Following Option 1 from #738, this updates the skills so the agent leaves enhanced files uncommitted in the working tree and surfaces them in the final report for the person to review and commit. This preserves darnit's core rule that it never commits content it did not write. Also bumped the skill metadata versions, added regression tests in
test_procedures.py, and updatedCHANGELOG.md.Fixes #738
Type of Change
Framework Changes Checklist
If this PR modifies the darnit framework (
packages/darnit/):docs/architecture/framework-design.md) if behavior changeduv run python scripts/validate_sync.py --verboseand it passesControl/TOML Changes Checklist
If this PR modifies controls or TOML configuration:
Testing
uv run pytest tests/ -v)uv run ruff check .)AI assistance
Additional Notes