docs(autotrain): continuous-openui-20260808 c1-c5 closeout (non-positive screening) - #1495
docs(autotrain): continuous-openui-20260808 c1-c5 closeout (non-positive screening)#1495Tyler-R-Kendrick wants to merge 9 commits into
Conversation
Cycle 1 (continuous-loop-20260808-continuous-openui-202608-1211eecb-c1) non-positive: fixture insufficient_n (n=3<20) and null primary-metric delta between bounds/control arms; honest ship gates correctly reject on smoke scale. Records scratch checkpoint provenance in MODEL_CARD and README (no-bump: SLM-228 behavior unchanged).
Cycle 2 (continuous-loop-20260808-continuous-openui-202608-1211eecb-c2) positive on structural_similarity (0.327->0.383) for the component-plan arm vs control; queued as a champion candidate for fresh-seed confirmation, no ship-gate clear yet (fixture insufficient_n, still smoke-scale). No tracked code delta this cycle (knob-only), so no new stack layer per autotrain-iteration-delivery. Records scratch checkpoint provenance (no-bump: SLM-228 behavior unchanged).
Cycle 3 (continuous-loop-20260808-continuous-openui-202608-1211eecb-c3) fresh-seed confirmation of the c2 component-plan champion candidate: rejected (structural_similarity and meaningful_program_rate did not re-hold; binder_reference_f1 regressed 0.822->0.6 vs control). No stack layer (non-positive). Records scratch checkpoint provenance (no-bump: SLM-228 behavior unchanged).
Cycle 4 (continuous-loop-20260808-continuous-openui-202608-1211eecb-c4) batch1 vs control: null primary-metric delta (structural_similarity 0.4167 both arms), fixture insufficient_n. No stack layer (non-positive). Records scratch checkpoint provenance (no-bump: SLM-228 behavior unchanged).
Cycle 5 (continuous-loop-20260808-continuous-openui-202608-1211eecb-c5) incomplete: component-plan arm did not execute (harness measurement gap); control arm completed and fails honest ship gates on fixture scale as expected. Inconclusive, no stack layer. Priority for next cycle: replay the exact frozen control/candidate before a new hypothesis. Records scratch checkpoint provenance (no-bump: SLM-228 behavior unchanged).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 37 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughAdded fixture-only screening records for five 2026-08-08 continuous OpenUI autotrain cycles. The changes add campaign notes, candidate/control metrics, outcomes, rejection reasons, checkpoint provenance, and explicit non-promotion classifications. ChangesContinuous OpenUI screening campaign
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Cycle 6 (continuous-loop-20260808-continuous-openui-202608-1211eecb-c6) bounds vs control on fresh upstream (b12eb2c): non-positive, null primary-metric delta (structural_similarity 0.0964 both arms), fixture insufficient_n. No stack layer. Records scratch checkpoint provenance (no-bump: SLM-228 behavior unchanged).
Cycle 7 (continuous-loop-20260808-continuous-openui-202608-1211eecb-c7) component-plan vs control: non-positive, null primary-metric delta (structural_similarity 0.0575 both arms), fixture insufficient_n. No stack layer. Records scratch checkpoint provenance (no-bump: SLM-228 behavior unchanged).
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
README.md (1)
795-825: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDuplicate headings trigger MD024 in both files.
Each new "Continuous autotrain note" block reuses the same heading text for the date. markdownlint-cli2 flags this as a duplicate heading (MD024) in both files. Include the campaign id or cycle number in the heading text to make each heading unique.
README.md#L795-L825: Append the campaign id (e.g.,## Continuous autotrain note (2026-08-08, c1)) to each of the 5 new headings.docs/MODEL_CARD.md#L1451-L1481: Append the campaign id (e.g.,## Continuous autotrain note (2026-08-08, c1)) to each of the 5 new headings.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 795 - 825, Make all five “Continuous autotrain note” headings unique in README.md (lines 795-825) by appending each corresponding cycle identifier (c1 through c5) to the date; apply the same heading updates in docs/MODEL_CARD.md (lines 1451-1481).Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json`:
- Around line 37-41: Update the reason-producing logic to stop emitting the
misleading fixture_insufficient_n_alone aggregate when other rejection reasons
are present, using a distinct multi-reason code if downstream consumers require
it. Regenerate
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json
(lines 37-41) and its reasons line in
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.md
(line 11); apply the same correction and regeneration to
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json
(lines 37-43) and
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md
(line 11).
- Around line 28-42: Update all six result artifacts to record an independently
reproducible run contract, artifact references, and a machine-readable honest
ship-gate outcome, including recipe and manifest, wall-time cap, source commit,
AgentEvals JSONL, and pinned AgentV bundle. Apply this to
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json
(lines 28-42), c1-results.md (lines 3-15), c2-results.json (lines 28-41),
c2-results.md (lines 3-15), c3-results.json (lines 28-44), and c3-results.md
(lines 3-15); the Markdown files must mirror the fields or link to the complete
result bundle.
In
`@docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json`:
- Around line 28-43: Bind the c3 confirmation record to its parent experiment
contract by adding the locked c2 arm, seed, manifest, endpoint, stopping rule,
family, gate, and expected smoke.structural_similarity threshold to both
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json
(lines 28-43) and
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md
(lines 3-13), or link both reports to the immutable confirmation manifest.
---
Nitpick comments:
In `@README.md`:
- Around line 795-825: Make all five “Continuous autotrain note” headings unique
in README.md (lines 795-825) by appending each corresponding cycle identifier
(c1 through c5) to the date; apply the same heading updates in
docs/MODEL_CARD.md (lines 1451-1481).
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5e2791c7-6dde-4d8b-a138-3950628ccad5
📒 Files selected for processing (13)
README.mddocs/MODEL_CARD.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c4-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c4-results.mddocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.jsondocs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c5-results.mdsrc/slm_training/resources/versions.json
| "cycle_index": 1, | ||
| "cycle_intent": "screening", | ||
| "cycle_role": "screening", | ||
| "evidence_class": "fixture", | ||
| "honesty": "fixture_screening_only_not_ship", | ||
| "loop_id": "continuous-openui-20260808", | ||
| "measurement_complete": true, | ||
| "positive": false, | ||
| "primary_metric": "smoke.structural_similarity", | ||
| "reasons": [ | ||
| "fixture_insufficient_n:c20260808-continuous-openui-202608-1211eecb-c1-control", | ||
| "fixture_insufficient_n:c20260808-continuous-openui-202608-1211eecb-c1-bounds", | ||
| "primary_metric_null_or_worse:smoke.structural_similarity:control=0.057499999999999996 candidate=0.057499999999999996 improvement=0.0", | ||
| "fixture_insufficient_n_alone" | ||
| ], |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Record the run contract and honest gate result.
These records contain aggregate metrics and free-form reasons, but they do not carry or link to the recipe and manifest, wall-time cap, source commit, AgentEvals JSONL, pinned AgentV bundle, or a machine-readable honest ship-gate result. The JSON records mark measurement_complete: true, and the Markdown files present closeout summaries. The supplied artifacts are therefore not independently reproducible or auditable.
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json: add the run contract, artifact references, and explicit gate outcome.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.md: mirror those fields or link to the complete result bundle.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.json: add the run contract, artifact references, and explicit gate outcome.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.md: mirror those fields or link to the complete result bundle.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json: add the run contract, artifact references, and explicit gate outcome.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md: mirror those fields or link to the complete result bundle.
As per coding guidelines, no train or evaluation run is complete without recipe metadata, honest gate outcomes, AgentEvals JSONL, and a pinned AgentV SDK result bundle.
📍 Affects 6 files
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json#L28-L42(this comment)docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.md#L3-L15docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.json#L28-L41docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c2-results.md#L3-L15docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json#L28-L44docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md#L3-L15
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json`
around lines 28 - 42, Update all six result artifacts to record an independently
reproducible run contract, artifact references, and a machine-readable honest
ship-gate outcome, including recipe and manifest, wall-time cap, source commit,
AgentEvals JSONL, and pinned AgentV bundle. Apply this to
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json
(lines 28-42), c1-results.md (lines 3-15), c2-results.json (lines 28-41),
c2-results.md (lines 3-15), c3-results.json (lines 28-44), and c3-results.md
(lines 3-15); the Markdown files must mirror the fields or link to the complete
result bundle.
Source: Coding guidelines
| "reasons": [ | ||
| "fixture_insufficient_n:c20260808-continuous-openui-202608-1211eecb-c1-control", | ||
| "fixture_insufficient_n:c20260808-continuous-openui-202608-1211eecb-c1-bounds", | ||
| "primary_metric_null_or_worse:smoke.structural_similarity:control=0.057499999999999996 candidate=0.057499999999999996 improvement=0.0", | ||
| "fixture_insufficient_n_alone" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Remove the misleading fixture_insufficient_n_alone reason.
The token says that insufficient sample size was the only rejection reason. The c1 record also reports a zero primary-metric delta. The c3 record also reports a binder F1 regression and a confirmation rejection. This creates contradictory historical evidence and can misclassify the run downstream.
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json: remove or replace the aggregate_alonereason.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.md: regenerate the reasons line from the corrected JSON.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json: remove or replace the aggregate_alonereason.docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md: regenerate the reasons line from the corrected JSON.
Fix the reason producer before regenerating the records. Use a distinct multi-reason code if downstream consumers need that distinction.
📍 Affects 4 files
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json#L37-L41(this comment)docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.md#L11-L11docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json#L37-L43docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md#L11-L11
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json`
around lines 37 - 41, Update the reason-producing logic to stop emitting the
misleading fixture_insufficient_n_alone aggregate when other rejection reasons
are present, using a distinct multi-reason code if downstream consumers require
it. Regenerate
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.json
(lines 37-41) and its reasons line in
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c1-results.md
(line 11); apply the same correction and regeneration to
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json
(lines 37-43) and
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md
(line 11).
| "cycle_index": 3, | ||
| "cycle_intent": "confirm", | ||
| "cycle_role": "screening", | ||
| "evidence_class": "fixture", | ||
| "honesty": "fixture_screening_only_not_ship", | ||
| "loop_id": "continuous-openui-20260808", | ||
| "measurement_complete": true, | ||
| "positive": false, | ||
| "primary_metric": "smoke.structural_similarity", | ||
| "reasons": [ | ||
| "fixture_insufficient_n:c20260808-continuous-openui-202608-1211eecb-c3-control", | ||
| "fixture_insufficient_n:c20260808-continuous-openui-202608-1211eecb-c3-confirm", | ||
| "non_regression_fail:binder_reference_f1:0.8222222222222223->0.6", | ||
| "primary_metric_null_or_worse:smoke.structural_similarity:control=0.19083333333333333 candidate=0.19083333333333333 improvement=0.0", | ||
| "fixture_insufficient_n_alone", | ||
| "confirmation_rejected:primary_quality_not_reheld" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rtk rg -n -C 8 'ExperimentCampaignV1|cycle_intent|confirmation_rejected|parent_cycle|manifest|seed|stopping|family|gate' docs src scriptsRepository: Tyler-R-Kendrick/slm-training
Length of output: 208
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- available tools ---\n'
which rg sed python3 jq git 2>/dev/null || true
printf '\n--- target files ---\n'
for f in \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3 成果.md \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md
do
echo
if [ -f "$f" ]; then
echo "FOUND $f"
wc -l "$f"
sed -n '1,120p' "$f"
else
echo "MISSING $f"
fi
done
printf '\n--- search terms from root (without rtk) ---\n'
rg -n -C 5 'ExperimentCampaignV1|cycle_intent|confirmation_rejected|parent_cycle|manifest|seed|stopping|family|gate|continuous-openui-20260808|1211eecb-c3|structural_similarity|binder_reference_f1' docs src scripts 2>/dev/null || true
printf '\n--- tracked docs/design continuous-loop files ---\n'
git ls-files 'docs/design/continuous-loop*.json' 'docs/design/continuous-loop*.md' | sortRepository: Tyler-R-Kendrick/slm-training
Length of output: 258
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- available tools ---'
command -v rg || true
command -v sed || true
command -v python3 || true
command -v jq || true
command -v git || true
echo
echo '--- target files ---'
for f in \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3.md
do
echo
if [ -f "$f" ]; then
echo "FOUND $f"
wc -l "$f"
sed -n '1,160p' "$f"
else
echo "MISSING $f"
fi
done
echo
echo '--- search terms from root (without rtk) ---'
rg -n -C 5 'ExperimentCampaignV1|cycle_intent|confirmation_rejected|parent_cycle|manifest|seed|stopping|family|gate|continuous-openui-20260808|1211eecb-c3|structural_similarity|binder_reference_f1' docs src scripts 2>/dev/null || true
echo
echo '--- tracked docs/design continuous-loop files ---'
git ls-files 'docs/design/continuous-loop*.json' 'docs/design/continuous-loop*.md' | sortRepository: Tyler-R-Kendrick/slm-training
Length of output: 50386
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- target files ---'
for f in \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3.md
do
echo
if [ -f "$f" ]; then
echo "FOUND $f"
wc -l "$f"
sed -n '1,160p' "$f"
else
echo "MISSING $f"
fi
done
echo
echo '--- deterministic field check for c3 results files ---'
python3 - <<'PY'
import json, pathlib, re
targets = [
"docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json",
"docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md",
]
for path in targets:
p = pathlib.Path(path)
print(f"\n{path} exists={p.exists()}")
if not p.exists():
continue
text = p.read_text(encoding="utf-8")
if path.endswith(".json"):
data = json.loads(text)
print("top-level keys=", sorted(data.keys()))
for key in ["parent_cycle","parent_arm","cycle_intent","positive","primary_metric","reasons","seed","manifest","endpoint","gate","stopping_rule","family","c2"]:
if key in data:
print(f"{key}=", data[key])
else:
print(f"missing {key}")
# Search for the specific c2 primary value mentioned in the review.
print('contains 0.38280000000000003=', '0.38280000000000003' in text)
print('contains 0.3828=', '0.3828' in text)
else:
print('contains cycle_intent=', 'cycle_intent' in text)
print('contains confirmation_rejected=', 'confirmation_rejected' in text)
print('contains 0.38280000000000003=', '0.38280000000000003' in text)
print('contains 0.3828=', '0.3828' in text)
for key in ["parent_cycle","parent_arm","c2","seed","manifest","endpoint","gate","stopping_rule","family"]:
print(f'{key} in text=', key in text or re.search(key + r'[:=]', text) is not None)
echo
echo '--- focused source/docs check for c3-only contract fields ---'
rg -n -C 3 '\b(parent_cycle|parent_arm|cycle_intent|confirmation_rejected|continuous-loop-.*-c3|0\.38280000000000003|control_metrics|candidate_metrics|ExperimentCampaignV1)\b' \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md \
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3.md 2>/dev/null || trueRepository: Tyler-R-Kendrick/slm-training
Length of output: 4466
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
path = Path("docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json")
data = json.loads(path.read_text(encoding="utf-8"))
needed = ["parent_cycle", "parent_arm", "expected_primary_metric", "seed", "manifest_id", "endpoint", "gate", "stopping_rule", "family", "parent"]
for key in needed:
print(f"{key}={data.get(key)!r}")
print(f"all_qualified_expected_keys_present={set(needed).issubset(set(data.keys()) | set(p for p in path.read_text(encoding='utf-8').split()))}")
PYRepository: Tyler-R-Kendrick/slm-training
Length of output: 366
Bind c3 to its parent experiment contract.
campaign_id identifies this record, but cycle_intent: confirm does not identify the c2 arm, seed, manifest, endpoint, stopping rule, family, gate, or expected smoke.structural_similarity threshold. Add the locked reference fields to both the JSON result and markdown report, or link to the immutable confirmation manifest.
📍 Affects 2 files
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json#L28-L43(this comment)docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md#L3-L13
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json`
around lines 28 - 43, Bind the c3 confirmation record to its parent experiment
contract by adding the locked c2 arm, seed, manifest, endpoint, stopping rule,
family, gate, and expected smoke.structural_similarity threshold to both
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.json
(lines 28-43) and
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c3-results.md
(lines 3-13), or link both reports to the immutable confirmation manifest.
Source: Coding guidelines
_append_checkpoint_doc_notes() stamped every cycle's README/MODEL_CARD note with the identical "## Continuous autotrain note (<date>)" heading, which collides (MD024) once more than one cycle lands on the same day — flagged by CodeRabbit on PR #1495. Append the campaign id to the heading so each cycle's note is uniquely addressable, and retrofit the c1-c7 headings already committed on this branch. Also regenerate docs/design/semantic-floor-gate-v1.{json,md} (unrelated pre-existing staleness hit by this commit's broad changed-file test selection): verdict stays inconclusive, only recomputed digests change.
|
$(cat <<'EOF' Fixed (6de37b6): the duplicate Not changing in this PR (the other three "Major" findings):
Happy to open a follow-up harness issue for (1)-(3) if useful, but didn't want to bundle a behavior change into a non-positive screening closeout PR. Generated by Claude Code |
2 participants
Summary
Ran the
autotraincontinuous loop (continuous-openui-20260808) for 5supervised cycles against published fixture train data (
wf_smoke_v2,smoke suite). This PR carries the iron-law documentation and honesty-stub
checkpoint notes for those cycles, per
sdlc'sautotrain-iteration-deliveryprocess. Per that process, stacked PRsopen only for positive-result cycles — none of these five screening
cycles cleared that bar, so this is a single docs-only delivery, not a
training-code stack.
…-c1): bounds vs control, non-positive — fixtureinsufficient_n(n=3<20), null primary-metric delta.…-c2): component-plan vs control — primary-metric win onstructural_similarity(0.327→0.383), queued as a champion candidatefor fresh-seed confirmation; no tracked code delta, so no stack layer.
…-c3): fresh-seed confirmation of the c2 candidate —rejected (quality did not re-hold;
binder_reference_f1regressed0.822→0.6 vs control).
…-c4): batch1 vs control — null primary-metric delta(
structural_similarity0.4167 both arms).…-c5): incomplete — the component-plan arm did not execute(harness measurement gap); control arm completed and correctly fails
honest ship gates on fixture scale.
All cycles correctly reject the honest ship gate at smoke/fixture scale
(
insufficient_n, quality thresholds) — this is expected diagnosticbehavior, not a ship claim.
What's included
docs/design/continuous-loop-20260808-continuous-openui-202608-1211eecb-c{1..5}-results.{md,json}— per-cycle measured results (iron law).
docs/MODEL_CARD.md/README.md— honesty-stub scratch checkpointprovenance notes for each cycle.
src/slm_training/resources/versions.json—no-bumphistory entriesfor
harness.experiments.slm228_spectral_disposition(README/MODEL_CARDtouched, no behavior change).
Test plan
scripts.verify_version_stamps --check --stagedpasses for everycommit in this PR.
results.jsonscoreboard.screening cycle produces a genuine positive result (primary-metric
win with a tracked code delta, ship-quality win, or proven
executable unblock).
🤖 Generated with Claude Code
https://claude.ai/code/session_016QXmZApaPwKQNZH3XENDmm
Generated by Claude Code
Summary by CodeRabbit