SLM-314: preserve multiple hard-valid AST modes with winner-take-all training - #851
SLM-314: preserve multiple hard-valid AST modes with winner-take-all training#851Tyler-R-Kendrick wants to merge 8 commits into
Conversation
- coverage_class per frozen row (OBSERVABLE_PROMPT / OBSERVABLE_REQUEST_ONLY / UNKNOWN); all 51 rows are OBSERVABLE_REQUEST_ONLY pre-inventory. - Matched requests differ only by the production ensure_prompt_inventory suffix; request-derived, no hidden gold channel. - X22 deterministic arm: paired v2-strict delta +0.3125 (5/16 vs 0/16, Wilson [0.142, 0.556]) exceeds the predeclared 0.10 minimum; no regressions on observable rows. - AR tiny baseline: 6/32 decodes completed (all v2-fail partials), 26 decode_timeout (600-step checkpoint constrained-forest cost per SLM-294 evidence); reported unmeasured, not evidence. Rico rows not_run with SLM-294 cost evidence. - 100-record blind slot-observability audit: raw agreement 97%, kappa 0.0 (skew caveat); finding: v2 coverage detector misses inline slot enumeration -> false prompt_contract_unknown (append-only; no metric change). - 8 harness tests; new component harness.experiments.slm301_prompt_observability v1.
…ints - decode_outcome.py taxonomy with strict precedence (fallback never counts as model success); per-request budget/elapsed/forwards/ verifier/fallback/stop-reason records. - eval_runner additive integration: details[] gain decode_outcome, stop_reason, fallback_used; suite metrics gain decode_outcome_counts. - Census: only slm230 committed-SHA verifiable; hash-pinned the two remediated nonzero-timeout checkpoints; scoreboard classes separated (runtime_timeout/fallback/unmeasured/model_behavior). - Preregistered 1x vs 10x budget sweep: all 12 cells not_rerunnable (v0 checkpoints fail current output-contract check); the 10x-flip question is honestly UNANSWERED by re-decode. Recommendation: retrain the remediated recipe on contract v2, then re-eval at both budgets. - 25 new tests; harness.model_build.eval v50; component harness.experiments.slm303_decode_budget_audit v1; gates.ship v3 threshold mirror fixed.
- slm308_distance_oracle.py: bounded reverse-BFS over canonical AST fingerprints via the real extended 11-action transitions; EXACT / BOUNDED / UNKNOWN labels (budget never conflated with distance); cache keyed by action-schema version, grammar sha, inventory, target and state hashes. - tree_edit_diffusion: value_label_mode (bounded_distance default for new configs; pre-field checkpoints inject mutation_count for behavior parity, format stays 2); normalized oracle cost-to-go with UNKNOWN masked out of MSE; pairwise progress margin loss (independently tested); decode paths never read gold distance (audit test). - Matched fixture experiment (mutation_count vs bounded_distance, identical budgets, near-gold + seed-trajectory states): rank corr 0.433->0.505 (+0.072 < 0.10), beam regret 0.889->0.889 (+0.0 < 0.05), Brier 0.031->0.004, UNKNOWN coverage 0.175. Preregistered thresholds written before results; verdict honestly rejected at this budget. - 16 new tests; slm308 component v1; slm299 component v3.
- Proposal instrumentation: per enumerated candidate — action, factor score, applicability, rejection reason (23 machine-readable codes), budget consumption; deterministic order; additive evidence keys. - Distribution audit: training-target vs decode-demand action distributions by source/suite; dead-candidate rate ~0.98, applicable-ADD recall 0.10 baseline; preregistered reweighting rule. - corruption_action_distribution knob (gold corpus untouched; default off = historical uniform; parity tested) and stop_slot_accounting legacy|corrected arm (STOP consumes an expansion slot only when its frozen candidate is retained; deterministic, regression-tested). - Matched 2x2 (ADD-balanced x STOP arms, isolated levers): ADD target share 0.177 -> 0.328 (T1 +0.150 >= 0.10 both STOP arms), recall 0.10 -> 0.16, corrected STOP budget <= legacy; verdicts adopted per preregistered rules; loss reweighting deferred to preserve isolation. - 10 new tests; slm299 component v4; slm310 component v1.
- State-source harness: gold_only (existing corruption chain), seedward
(offline oracle-guided walk from the seed toward gold, strictly
distance-decreasing valid intermediates), on_policy (immutable
content-addressed beam-trajectory snapshots: wrong states, verifier
failures, abstentions); explicit provenance + gold-visibility policy
per row; sha256 rows + tamper-evident manifest; fail-closed leakage
guards (train vs held-out AST fingerprints).
- Matched {gold_only, seedward, on_policy, mixed} arms with predeclared
weights/caps, identical model/steps/optimizer/seeds; evaluated on
held-out seed trajectories. Verdict: rejected per preregistered
primary gate (beam regret improvement 0.0 < 0.05); secondary signal
(value rank corr -0.258 -> +0.258 for non-gold arms) recorded as
wiring evidence only.
- 9 new tests; slm312 component v1.
…training
- Frozen multi-mode dataset (8 prompts x 2 verifier-accepted modes;
canonical alpha-invariant AST fingerprints, sha256 rows + tamper-evident
manifest; prompt count declared below ship-gate scale).
- WTA/MCL loss: backpropagate only the min-loss mode per example +
preregistered floor term (eps=0.1) on losing modes; durable per-example
selection telemetry. No set-FTPO machinery.
- Two-mode synthetic fixture proves single-gold CE collapses
(p(mode B)=0.00004) while WTA retains both modes (p(B)=0.0083,
p(A)=0.826 dominant); multi-gold spreads 50% mass to invalid hybrids.
- Matched arms {single_gold, multi_gold, wta} identical budgets: coverage
0.750/0.500/0.500, hard-valid decode 1.0, v2 regression 0.000.
Verdict rejected per preregistered coverage gate (WTA gain -0.25);
at this scale single-gold lifts the sibling mode too.
- 16 new tests; slm314 component v1.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (49)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Tyler-R-Kendrick
left a comment
Tyler-R-Kendrick
left a comment
There was a problem hiding this comment.
Not merging — branch is stale and merge is blocked well beyond versions.json. Findings from review of f6b1731 against current main:
-
Stale-branch ship-gate regression. The PR changes
MEANINGFUL_METRIC_POLICY['threshold_version']fromopenui_ship_gates_v2tov3, but main has been onopenui_ship_gates_v4since #764 ('Make AgentEvals criteria authoritative gates', 5de2958). Merging would revert the gate policy v4→v3 — exactly the gate-weakening the repo laws forbid. -
14 files conflict, including code-semantics files (merge-tree vs main):
src/slm_training/harnesses/model_build/eval_runner.py(7 newer main commits since base 528feeb),ship_gates.py,models/tree_edit_diffusion.py(5),models/checkpoint_migrate.py(2),evals/agentv.py,runtime/telemetry/trace.py,harnesses/experiments/slm299_edit_reachability.py(5),scripts/run_slm299_reachability_audit.py(3),tests/test_models/test_tree_edit_diffusion.py,tests/test_runtime_trace.py,docs/design/agentv-evaluation.md,docs/design/iter-slm305-edit-language-20260724.md,docs/design/langsmith-telemetry-smoke-20260724.json, andversions.json. Resolving these requires real code decisions, not a versions.json union. -
Partially superseded/duplicated content. This is a 6-ticket stacked PR (SLM-303/305/308/310/312/314). SLM-305 already merged separately as #842 (a3dab82, which also landed
scripts/run_slm303_decode_budget_audit.pyon main); the SLM-308 (#845), SLM-312 (#848), and SLM-303 (#911) sibling PRs were closed. The SLM-305/303 portions here are now duplicate churn against main.
Positive: the SLM-314-specific code itself looks sound — additive decode-outcome taxonomy (fail-closed: fallback outputs can never classify as model_valid), and the PR's own tests pass at its head (75 passed: test_slm314_winner_take_all, test_decode_outcome, test_tree_edit_diffusion, test_eval_gates), and verify_version_stamps + repo_policy are green.
Recommendation: rebase and split — land only the SLM-314-delta files (harnesses/experiments/slm314_winner_take_all.py, scripts/run_slm314_winner_take_all.py, slm314_multimode data, its docs + tests) on top of current main, keeping main's openui_ship_gates_v4 and the newer eval_runner/tree_edit_diffusion work, then resubmit as a single-ticket PR.
|
Closing to drain the stale queue. Review comments above document the still-unique delta (SLM-310/314 harnesses) — re-open as a focused branch rebased on current main. |
1 participant
Summary
SLM-314 / LAR2-05: preserve multiple hard-valid AST modes with winner-take-all training.
Verification
harness.experiments.slm314_winner_take_allv1); repo_policy ok; git diff --check clean; ruff clean.Closes SLM-314.