docs(autotrain): continuous-openui-20260808 cycles 1-8 closeout - #1498
Open
Tyler-R-Kendrick wants to merge 8 commits into
Open
docs(autotrain): continuous-openui-20260808 cycles 1-8 closeout#1498Tyler-R-Kendrick wants to merge 8 commits into
Tyler-R-Kendrick wants to merge 8 commits into
Conversation
added 3 commits
August 8, 2026 07:36
…p-20260808-continuous-openui-202608-1211eecb-c1 closeout
…p-20260808-continuous-openui-202608-1211eecb-c2 closeout
…p-20260808-continuous-openui-202608-1211eecb-c3 closeout
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughThe PR adds continuous OpenUI screening fixtures for cycles 1–6, campaign notes for cycles 1–5, and five version-history records. The records capture metrics, screening outcomes, checkpoint paths, and fixture-only non-promotion status. ChangesContinuous OpenUI screening records
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
added 3 commits
August 8, 2026 07:48
…p-20260808-continuous-openui-202608-1211eecb-c4 closeout
…p-20260808-continuous-openui-202608-1211eecb-c5 closeout
…p-20260808-continuous-openui-202608-1211eecb-c6 closeout
added 2 commits
August 8, 2026 07:57
…p-20260808-continuous-openui-202608-1211eecb-c7 closeout
…p-20260808-continuous-openui-202608-1211eecb-c8 closeout
Tyler-R-Kendrick
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
README.md (1)
795-825: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDuplicate "Continuous autotrain note" headings in
README.mdanddocs/MODEL_CARD.md. Both files repeat the exact heading## Continuous autotrain note (2026-08-08)for every campaign entry, which markdownlint flags as MD024. Include the campaign id in the heading text to make each heading unique. This is a pre-existing convention from earlier cycles, not new to this PR, so treat it as optional cleanup.
README.md#L795-L825: change each heading to include the campaign suffix, for example## Continuous autotrain note (2026-08-08, c1).docs/MODEL_CARD.md#L1451-L1481: apply the same per-campaign heading suffix.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 795 - 825, Optionally update each repeated “Continuous autotrain note (2026-08-08)” heading to include its campaign suffix, such as c1 through c5, so headings are unique. Apply this to README.md lines 795-825 and docs/MODEL_CARD.md lines 1451-1481, preserving the corresponding campaign entries and content.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.json`:
- Around line 4-10: Add the declared primary metric key
smoke.structural_similarity with a null value to candidate_metrics in both
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.json
(lines 4-10) and
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c6-results.json
(lines 4-10). Preserve the existing unqualified structural_similarity alias and
all other candidate metrics.
---
Nitpick comments:
In `@README.md`:
- Around line 795-825: Optionally update each repeated “Continuous autotrain
note (2026-08-08)” heading to include its campaign suffix, such as c1 through
c5, so headings are unique. Apply this to README.md lines 795-825 and
docs/MODEL_CARD.md lines 1451-1481, preserving the corresponding campaign
entries and content.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 385b6948-e70e-4dcb-b842-e74fae48c6c5
📒 Files selected for processing (15)
README.mddocs/MODEL_CARD.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c4-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c4-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c6-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c6-results.mdsrc/slm_training/resources/versions.json
Comment on lines
+4
to
+10
| "candidate_metrics": { | ||
| "binder_reference_f1": null, | ||
| "latency_ms_p50": null, | ||
| "meaningful_program_rate": null, | ||
| "parse_rate": null, | ||
| "structural_similarity": null | ||
| }, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Tracked JSON rows matching continuous-loop/result names:\n'
git ls-files | grep 'docs/design/continuous-loop' || true
for f in \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c4-results.json \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.json \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c6-results.json
do
if [ -f "$f" ]; then
echo "--- $f ---"
python3 - <<'PY' "$f"
import json, sys
p=sys.argv[1]
data=json.load(open(p))
print("primary_metric:", data.get("primary_metric"))
print("candidate_metrics:")
for k,v in (data.get("candidate_metrics") or {}).items():
print(" ", k, "=", v)
print("other top keys:", list(data.keys()))
PY
else
echo "MISSING $f"
fi
done
echo
echo "All docs/design JSON primary_metric values and candidate_keys summaries:"
python3 - <<'PY'
import json
from pathlib import Path
for p in sorted(Path("docs/design").glob("*.json")):
try:
data=json.loads(p.read_text())
except Exception as e:
print(f"ERROR {p}: {e}")
continue
pm=data.get("primary_metric")
keys=list(data.get("candidate_metrics", {}).keys())
if pm == "smoke.structural_similarity" or (keys and any(k.startswith("smoke.") for k in keys)):
print(f"{p}: primary_metric={pm}; candidate_keys={keys}")
PYRepository: Tyler-R-Kendrick/slm-training
Length of output: 19568
Add smoke.structural_similarity to these candidate metric maps.
For structual_similarity with this result family, primary_metric: smoke.structural_similarity should map to candidate_metrics.smoke.structural_similarity. Keep the existing unqualified aliases if that is the current output contract, but record the declared primary key with null while the score is unavailable:
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c6-results.json
📍 Affects 2 files
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.json#L4-L10(this comment)docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c6-results.json#L4-L10
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.json`
around lines 4 - 10, Add the declared primary metric key
smoke.structural_similarity with a null value to candidate_metrics in both
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.json
(lines 4-10) and
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c6-results.json
(lines 4-10). Preserve the existing unqualified structural_similarity alias and
all other candidate metrics.
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scheduled
autotraincontinuous-loop run (continuous-openui-20260808), fixture-scale screening cycles againstwf_smoke_v2train data and thee938_role_safe_all_targets_v2eval snapshot (smokesuite), obeying the 3-minuteMAX_RUN_MINUTEShard run cap per experiment....-c1)...-c2)smoke.structural_similarity0.327 → 0.383), enqueued as champion candidate. Positive, but no tracked code delta — no stack layer....-c3)binder_reference_f1regressed 0.822 → 0.6. Non-positive....-c4)...-c5)component-planarm measurement incomplete (missing scoreboard); queuedretry_measurement....-c6)decode_timeout_count=3) while the matched control completed — a real harness residual worth tracking, not a model regression. Self-healed, no stack layer....-c7)...-c8)Per this repo's
autotrain/SDLC continuous-mode rules, stacked PR layers are reserved for positive-result cycles that carry a tracked code delta. All eight cycles here are documentation-only closeouts of fixture screening runs (including one positive metric win that didn't survive fresh-seed confirmation), so they land as plain commits on this branch rather than separate stack layers.Changes
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c{1..8}-results.{md,json}— per-cycle experiment evidence (documenting-experiment-results).docs/MODEL_CARD.md,README.md— continuous-autotrain checkpoint provenance notes (fixture/scratch honesty stubs, not ship promotions).src/slm_training/resources/versions.json—no-bump:history entries forharness.experiments.slm228_spectral_dispositioncovering the doc-only README/MODEL_CARD touches (verify_version_stampscontract).Test plan
python -m scripts.verify_version_stamps --check --stagedpasses for each committest_verify_checkpoint_references,test_versioning/*) passpython -m scripts.autoresearch status --loop-id continuous-openui-20260808 --matrix --last 8printed and reviewed after each cycleFollow-ups (not in this PR)
model_runtimefamily) worth a dedicatedimprove-openui-harnessesinvestigation.versions.jsonno-bump entry, so each cycle currently needs a manual bump to passverify_version_stamps; consider teachingSELF_HEAL_DOCUMENTto do this automatically.