feat(tier-check): validate the submitted SDK release against the target spec version#402
Draft
rinaldofesta wants to merge 1 commit into
Draft
Conversation
…et spec version The spec-tracking check compares the latest spec release with the first SDK release published after it and passes on a gap of <= 30 days. That validates nothing about a tier submission: the measured release can be an unrelated patch, the spec version is not the one the submission claims, and 30 days matches no documented requirement (SEP-1730 says "before the new spec version release" for Tier 1 and "within six months" for Tier 2). The Java SDK Tier 2 assessment showed a 9d-gap PASS while the submitted v1.0.0 shipped 250 days after the 2025-06-18 spec it claims. computeTier also only blocks Tier 1 on status 'fail', so a skipped result (API error, missing releases) scores green. Add --sdk-release-tag to pin the check to the exact submitted release (getReleaseByTag, no normalization); the existing --spec-version flag now also pins the target spec release. Replace the 30-day constant with the SEP-1730 windows (pass <= 0d, partial <= 183d, fail beyond), derived once and exposed as meets_tier1_window / meets_tier2_window, which computeTier reads instead of pattern-matching statuses (same shape as p0's all_p0s_resolved_within_7d/14d). Submitted tags that do not resolve fail with the attempted tag in `reason` (precedent: modelcontextprotocol#372 moved missing prerequisites from skip to fail); unexpected API errors still skip, but skipped now blocks Tier 1 like skipped conformance already does. Both renderers print `reason` when there is no gap to show. Existing SpecTrackingResult fields keep emitting; sdk_release_within_30d is no longer read by tier logic. Refs modelcontextprotocol#167.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #167.
Broken behavior
checkSpecTrackingcompares the latest spec release with the first SDK release published after it and passes on a gap of<= 30days. That measures nothing about a tier submission: the release it finds can be an unrelated patch, the spec version is not the one the submission claims, and the 30-day figure appears nowhere in the tiering requirements (docs/community/sdk-tiers.mdxand SEP-1730 require "before the new spec version release" for Tier 1 and "within six months" for Tier 2). Concrete case, from the Java SDK Tier 2 assessment (modelcontextprotocol/modelcontextprotocol#2301): the scorecard shows a 9d-gap PASS while the submittedv1.0.0was published 2026-02-23, 250 days after the 2025-06-18 spec it claims.A second defect compounds it:
computeTieronly blocks Tier 1 onstatus === 'fail', andtier2Metnever readsspec_tracking, so askippedresult (API error, missing releases) scores green.Change
Per the plan agreed on #167:
--sdk-release-tag <tag>: exact release tag of the submitted SDK version, resolved withgetReleaseByTag, no normalization. A tag that does not resolve fails with the attempted tag inreason(precedent: fix: fail checks whose prerequisite is missing instead of skipping them #372 moved missing prerequisites from skip to fail). Draft releases are rejected.--spec-version(existing flag, feat: add --spec-version option to tier-check #176) now also pins the spec release the gap is measured against. Absent = latest stable, as before. Thedraftalias has no GitHub release to pin, so draft runs keep tracking the latest stable release.<= 0days =pass(Tier 1, before the spec release),<= 183days =partial(Tier 2, within six months), beyond =fail. Derived once and exposed asmeets_tier1_window/meets_tier2_window, whichcomputeTierreads instead of pattern-matching status strings (same shape as p0'sall_p0s_resolved_within_7d/14d).skipped, butskippednow blocks Tier 1 the same way skipped conformance already does.reasonwhen there is no gap to show.SpecTrackingResultfields keep emitting;sdk_release_within_30dis still computed but no longer read by tier logic.Behavior changes to be aware of
partial(blocks Tier 1, satisfies Tier 2). The old 30-day pass window matched no documented requirement.--spec-versionfor scenario filtering now also pin spec-tracking to that version. This is the issue's intent (measure against the spec version the submission claims), but it can change scores for CI that pins older versions.skippedspec_tracking now appears intier1_blockersasspec_tracking (skipped).Validation
npx vitest run: 40 files, 449 tests, 0 failures (baseline before the change: 37 files, 419 tests). The new spec-tracking.test.ts, tier-logic.test.ts and index.test.ts cover the window boundaries (-5, 0, 1, 183, 184 days), missing-tag and draft-release failures, API-error skips, CLI forwarding, and the tier-logic blocker matrix.npm run check(typecheck + eslint + prettier): clean.End to end against the motivating case:
tier-check --repo modelcontextprotocol/java-sdk --sdk-release-tag v1.0.0 --spec-version 2025-06-18 --skip-conformance --output jsonSame command with
--sdk-release-tag v9.9.9(nonexistent), terminal output:Scope cuts, for maintainers to call
--spec-versiongoes throughresolveSpecVersion, whose vocabulary is the released dated versions plusdraft. Widening that vocabulary, or adding a dedicated flag, is your call; I can extend this PR either way.TIER2_WINDOW_DAYS).sdk_release_within_30dis kept for output compatibility; renaming or dropping it is a one-line decision I did not take unilaterally.