feat: SLM-446/447/449/450/451/452/453/455/456 — 9 slm-training issues (VCE/PCT/SRP/SGS) - #1494
Open
Tyler-R-Kendrick wants to merge 9 commits into
Open
feat: SLM-446/447/449/450/451/452/453/455/456 — 9 slm-training issues (VCE/PCT/SRP/SGS)#1494Tyler-R-Kendrick wants to merge 9 commits into
Tyler-R-Kendrick wants to merge 9 commits into
Conversation
…E-002) Add RepairResidualV1, a pure/stateless projection of the existing SemanticRepairRecordV1/ConflictSlice into an advisory residual-obligation view (unsatisfied obligations, conflict slice/completeness, protected regions, legal repair-action domain, witness/source ids, tamper-evident residual_hash), optionally reconciling VCE-001's VerifierWitnessV1 unresolved localizations as additional obligations without ever fabricating legal edits, converting UNKNOWN to failure, or letting HEURISTIC evidence authorize repair. Extends the existing semantic_repair owner per the pre-registered ownership-map decision instead of adding a competing registry. Closes SLM-446. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 13 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (20)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…tion harness (VCE-004) Add PlanInterventionRecordV1/apply_plan_intervention to src/slm_training/data/semantic_plan/oracle.py -- an immutable, replayable paired baseline/intervention observation over the existing PlanOracleSubstitutor: before/after plan hashes, the declared-vs-observed changed-factor set (proving an intervention never touches an undeclared factor), a frozen per-comparison identity (model/request/candidate-budget/ search-budget/seed/verifier/hardware), and caller-supplied downstream decision/quality/compute evidence. filter_manifest_safe excludes contamination-bannered records from any manifest -- documented and tested against the gap where apply() intentionally never overwrites plan identity, so a mutated plan's own is_oracle_only cannot be trusted as the contamination signal. Extends the existing plan_oracle_substitutor owner per the pre-registered ownership-map decision instead of adding a competing intervention registry. Closes SLM-447. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
…PCT-002) Add ReplayBundleV1/build_replay_bundle to a new src/slm_training/runtime/telemetry/replay_bundle.py, composing evidence this repo already owns -- DecodeIdentityV1 (models/decode_stats.py) and solver_replay_violations (dsl/solver/replay.py) -- into a compact, content-bound, tamper-evident envelope: request identity, seed, frozen model/checkpoint/evaluator identity, the discrete decoder-choice sequence, legal-domain size/status/digest, redacted artifact state and final evidence, and an explicit replayable/non_replayable_reasons fail-closed verdict. Pure and stateless -- never touches telemetry state, so observability bound or not can never change a bundle for identical inputs. mirror_bundle_best_effort is an optional sink call that never raises (mirrors the existing OTLP-mirror contract in runtime/telemetry/trace.py), and persistence reuses whatever append-only store the caller already has via a minimal duck-typed ReplayBundleStore protocol rather than opening a new one. Extends the pre-registered runtime_telemetry + solver_replay owners (ownership_map.json downstream_extension_map row for PCT-002); introduces no new decode/verifier authority and no competing trace store. Closes SLM-449. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
Add src/slm_training/harnesses/model_build/cold_warm_bench.py: BenchTrialV1/run_cold_trial/run_warm_trial/summarize_trials. A cold trial always spawns a real fresh subprocess (never simulated by clearing caches in-process) and reads back phase markers the child process emits via phase_marker(); a warm trial runs an in-process callable and is labeled steady_state so it can never blend into cold numbers. Reuses the existing MEASUREMENT_STAGES/COMPLETENESS_STATES taxonomy from models/decode_stats.py rather than inventing a second one. A subprocess timeout or non-zero exit records partial_timeout/aborted -- never complete -- and summarize_trials excludes incomplete trials from any percentile. Hardware/runtime identity (HardwareIdentityV1) is captured and persisted on every trial, and control vs. artifact-enabled arms are kept in separate summary buckets by config_hash so mismatched configs are never silently averaged together. Extends the existing bench_* script family and the decode_telemetry owner; introduces no second telemetry authority. Closes SLM-450. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
…provider seam (PCT-005) Add tests/test_dsl/test_completion_artifact_provider_parity.py, a differential/tamper matrix proving StaticCompletionArtifactProvider .load_checked (PCT-004) is byte/digest/decision-identical to the direct load_checked_completion_artifact path it delegates to: - manifest, direct_map, digest, and full StaticLalrAdapter equality between the old and provider-mediated paths - every-prefix accepts() parity (via adapter structural equality) - checkpoint artifact-binding identity matches through the provider - dynamic runtime/scope facts stay absent from the provider payload - every corruption class -- byte-flipped artifact, stale grammar, unsupported manifest/metadata schema, tokenizer row shape abuse, and tampered rule_origins/rule_lengths/state_min_terminals -- fails closed identically through the provider seam (never a partial/degraded CompletionArtifact), with the exact live-construction fallback proven to remain available and correct test_completion_artifact.py already proved most of these vectors through the old direct path only; this closes the "through the provider seam" gap PCT-005 asks for without duplicating existing coverage. Closes SLM-451. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
…-007) Add MechanismActivationV1/build_mechanism_activation directly to src/slm_training/models/decode_stats.py: a common eligible/invoked/ abstained/no_op/changed_top_choice/changed_final_program/forwards/ verified_states/verifier_calls/wall_ms/failure_reason envelope any optional semantic/search/repair/controller mechanism can emit. build_mechanism_activation fails closed on internally inconsistent claims -- an ineligible mechanism cannot be invoked, invoked and abstained are mutually exclusive, and a choice-change claim requires invoked=True and no_op=False -- the concrete guard against representing a wired-but-never-active mechanism as a successful intervention. evidence_class/default_state stay caller-declared advisory fields (MECHANISM_EVIDENCE_CLASSES is an illustrative, non-exhaustive set) since no single canonical disposition enum spanning fixture/measured/ promotable/rejected/blocked exists yet -- SGS-009 (SLM-456) is still Backlog, and the only real precedent today is the duplicated per-experiment Disposition enum pattern in slm160/228/236_*_disposition.py. This does not fabricate that registry ahead of SGS-009; it stays a pure, additive telemetry record extending the existing decode_telemetry owner. Closes SLM-452. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
…(SRP-002) Add src/slm_training/dsl/symbolic_expr_ir.py: a real typed AST for the symbolic-regression pack (VarRef/CoefficientHole/OpApply) plus a deterministic canonical structural codec (serialize_expr/parse_expr), superseding the untyped dict-shaped expression tree in symbolic_regression_pack.py. The Lark surface grammar SRP-001 registered stays as input-surface syntax; this module owns the canonical structural form. Symbol-only by construction: variables are referenced by an opaque index into the problem's variable tuple, never by name, and coefficients are referenced by an opaque index only -- no free numeric literal or string can ever appear in the structural codec's output, and the token grammar makes this a parser-enforced guarantee, not a convention. This makes the codec alpha-renaming safe for free: renaming the underlying problem's variable names never changes a byte of the serialized form, since names never enter the IR. parse_expr/parse_problem_expr take the authorized operator set from the active problem contract (never a hardcoded default) and fail closed on depth/node budget overrun, unknown/unauthorized operators, wrong arity, and any malformed or smuggled token. validate_expr_budget extends the same budget guarantee to directly-constructed trees. Registers a new 'symbolic_expr_ir' ownership-map subsystem per the pre-registered SRP-002 downstream_extension_map row (new_owner_justified: true, layered under output_contract's symbol-only policy, does not fork OUTPUT_CONTRACT_VERSION or reuse production_codec's OpenUI-specific encode/decode logic). Closes SLM-453. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
Add src/slm_training/data/progspec/synthesis_capability_report.py: SyGuSCapabilityReportV1/build_sygus_capability_report classify every requirement-bearing element of a VerifiedSynthesisProblemV1 (runtime symbols, verification requirements, prompt-side facts/ambiguity groups, objective terms, search budget, evidence provenance) as representable, non_representable, or extension_required -- never silently dropped. Fidelity boundary kept narrow and demonstrable: a runtime symbol's existence/name is representable as an opaque SMT-LIB String-sorted declaration; a symbolic_regression pack's finite operator vocabulary (dsl/symbolic_expr_ir.py, SRP-002) is representable as a synth-fun grammar that mirrors its BINARY_OPERATORS/UNARY_OPERATORS exactly. Everything else -- this repo's own verifier gate/evaluator identities, prompt ambiguity/preference semantics, multi-objective optimization, resource budgets, provenance metadata -- has no SyGuS-IF equivalent and is reported with a typed status + reason, reusing the same idiom dsl/grammar_capabilities.py's UnsupportedCapabilityV1 already uses. No solver dependency is introduced and no external solver is ever called, so nothing here can widen exact compiler support. The emitted sygus_program is explicitly inspired-interoperability, not standards conformance (no (check-synth), no example-table constraints) -- conformance/conformance_note make that distinction machine-readable. Round-trip fixtures cover both required fragments: a finite symbolic-regression grammar and a restricted OpenUI symbol-existence fragment. Registers a new 'sygus_capability_report' ownership-map subsystem; the pre-registered SGS-008 downstream-extension row predicted extending dsl_pack_registry_canonical/completion_domain directly, but the implementation is a small self-contained module over the verified_synthesis_problem envelope instead -- documented as a deliberate divergence in the new subsystem's notes. Closes SLM-455. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
… (SGS-009)
Add src/slm_training/harnesses/experiments/mechanism_disposition_report.py:
MechanismDispositionRecordV1/MechanismDispositionReportV1/SupersessionEntryV1,
generalizing the SLM-160/SPV4-02 SPVMechanismDisposition narrative
precedent into reusable code for any mechanism, not just the SPV family.
Reuses that module's Disposition enum directly instead of redefining the
same seven values a second time.
This is the canonical EVIDENCE_CLASSES registry PCT-007 (decode_stats.py's
MECHANISM_EVIDENCE_CLASSES) named as still-Backlog when it stayed a
caller-declared advisory field instead of a closed registry.
build_mechanism_disposition_record fails closed on both acceptance
criteria: adopt_primary/adopt_optional requires evidence_class in
{measured,promotable,ship} plus non-empty activation_evidence -- a
wired-but-unmeasured mechanism can never be recorded as adopted -- and
evidence_class='fixture' can never pair with default_state='on', so
fixture-only results can never promote a default/checkpoint.
append_supersession is a pure, append-only function linking two
same-mechanism_id records; the superseded record is never rewritten or
removed from a report, only pointed at, so negative/null results stay
linked and visible. build_disposition_report/to_markdown prove the
closeout-from-structured-records capability the issue asks for.
Scope note: ships the reusable schema/validation/generator proven against
the issue's own fixture-matrix validation ask (known positive/null/
blocked mechanisms), not a completed audit of every real mechanism across
the current SGS/VCE/PCT/SRP/SIE/RSP backlog -- that data-entry pass is
follow-on work, mirroring how SLM-160's own audit was a separate pass
from building its harness.
Closes SLM-456.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg
Tyler-R-Kendrick
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nine related Linear issues from the
slm-trainingteam, all landed on this session's single designated branch (extends the same PR rather than opening additional ones, per this session's branch policy).SLM-446 (VCE-002) — Unify repair residuals with existing SemanticRepairRecordV1 and conflict slices
RepairResidualV1/project_repair_residualinharnesses/distill/semantic_repair.py.SLM-447 (VCE-004) — Promote existing PlanOracleSubstitutor into a full factor-wise intervention harness
PlanInterventionRecordV1/apply_plan_interventionindata/semantic_plan/oracle.py.SLM-449 (PCT-002) — Add hermetic replay traces and an optional observability sink
ReplayBundleV1/build_replay_bundlein newruntime/telemetry/replay_bundle.py.SLM-450 (PCT-003) — Implement an honest end-to-end cold and warm benchmark harness
BenchTrialV1/run_cold_trial/run_warm_trialin newharnesses/model_build/cold_warm_bench.py.SLM-451 (PCT-005) — Prove OpenUI artifact byte, digest, decision, and tamper parity through the provider seam
New
tests/test_dsl/test_completion_artifact_provider_parity.py.SLM-452 (PCT-007) — Add mechanism activation, choice-change, and disposition telemetry
MechanismActivationV1/build_mechanism_activationadded directly tomodels/decode_stats.py.SLM-453 (SRP-002) — Implement typed symbolic-expression IR, grammar, codec, and symbol-only contract
New
dsl/symbolic_expr_ir.py: typed AST + deterministic structural codec for the symbolic-regression pack.SLM-455 (SGS-008) — Add fail-closed SyGuS/SemGuS capability reports and conformance fixtures
New
data/progspec/synthesis_capability_report.py: classifies every requirement-bearing element of aVerifiedSynthesisProblemV1as representable/non-representable/extension-required, never silently dropped.SLM-456 (SGS-009) — Generate mechanism disposition and stale-evidence supersession reports
New
harnesses/experiments/mechanism_disposition_report.py:MechanismDispositionRecordV1/MechanismDispositionReportV1/SupersessionEntryV1, generalizing the SLM-160/SPV4-02SPVMechanismDispositionnarrative precedent into reusable code for any mechanism (reuses that module'sDispositionenum directly). This is the canonicalEVIDENCE_CLASSESregistry PCT-007 named as still-Backlog. Fails closed on both acceptance criteria:adopt_primary/adopt_optionalrequires measured evidence + non-empty activation evidence, andevidence_class="fixture"can never pair withdefault_state="on". Supersession is a pure append-only pointer between same-mechanism records — a superseded record is never rewritten or removed, only linked, so negative/null results stay visible. Ships the reusable schema/generator proven against a fixture matrix (the issue's own validation ask), not a completed audit of every real mechanism in the current backlog — documented as follow-on work.All nine extend or newly register their
ownership_map.jsonowners — no undocumented registries or authorities introduced. SLM-448 (VCE-006) was scoped but returned to Backlog with a documented comment. SLM-454 (SGS-006) was already completed by a separate session before this PR started.Test plan
pytest tests/test_harnesses/distill/test_semantic_repair.py— 20 passedpytest tests/test_data/test_semantic_plan_extraction/test_oracle.py— 16 passedpytest tests/test_runtime/test_replay_bundle.py— 11 passedpytest tests/test_harnesses/model_build/test_cold_warm_bench.py— 11 passedpytest tests/test_dsl/test_completion_artifact_provider_parity.py— 14 passedpytest tests/test_models/test_decode_stats.py— 34 passedpytest tests/test_dsl/test_symbolic_expr_ir.py tests/test_dsl/test_symbolic_regression_pack.py— 60 passedpytest tests/test_data/test_progspec/(full dir) — 132 passedpytest tests/test_harnesses/experiments/test_mechanism_disposition_report.py tests/test_harnesses/experiments/test_slm160_spv_disposition.py— 32 passedpytest tests/test_dsl/test_operator_registry.py tests/test_dsl/test_completion_artifact.py tests/test_dsl/test_completion_artifact_provider.py tests/test_dsl/test_static_control_domain.py— all greenpytest tests/test_runtime_trace.py tests/test_dsl/test_solver_replay.py tests/test_evals/test_semantic_failure.py— all greenpytest tests/test_harnesses/distill(full suite) — 187 passed, 4 pre-existing failures unrelated to this changepython -m scripts.verify_version_stamps --check— okpython -m scripts.verify_ownership_map— okpython -m scripts.refresh_test_cases --check --changed— no fixture driftruff checkon changed files — clean🤖 Generated with Claude Code
https://claude.ai/code/session_01Fssz1GjfLHoUSeZuP1K3zg