Skip to content

feat(tier-check): validate the submitted SDK release against the target spec version#402

Draft
rinaldofesta wants to merge 1 commit into
modelcontextprotocol:mainfrom
rinaldofesta:feat/spec-tracking-submitted-release
Draft

feat(tier-check): validate the submitted SDK release against the target spec version#402
rinaldofesta wants to merge 1 commit into
modelcontextprotocol:mainfrom
rinaldofesta:feat/spec-tracking-submitted-release

Conversation

@rinaldofesta

Copy link
Copy Markdown

Closes #167.

Broken behavior

checkSpecTracking compares the latest spec release with the first SDK release published after it and passes on a gap of <= 30 days. That measures nothing about a tier submission: the release it finds can be an unrelated patch, the spec version is not the one the submission claims, and the 30-day figure appears nowhere in the tiering requirements (docs/community/sdk-tiers.mdx and SEP-1730 require "before the new spec version release" for Tier 1 and "within six months" for Tier 2). Concrete case, from the Java SDK Tier 2 assessment (modelcontextprotocol/modelcontextprotocol#2301): the scorecard shows a 9d-gap PASS while the submitted v1.0.0 was published 2026-02-23, 250 days after the 2025-06-18 spec it claims.

A second defect compounds it: computeTier only blocks Tier 1 on status === 'fail', and tier2Met never reads spec_tracking, so a skipped result (API error, missing releases) scores green.

Change

Per the plan agreed on #167:

  • --sdk-release-tag <tag>: exact release tag of the submitted SDK version, resolved with getReleaseByTag, no normalization. A tag that does not resolve fails with the attempted tag in reason (precedent: fix: fail checks whose prerequisite is missing instead of skipping them #372 moved missing prerequisites from skip to fail). Draft releases are rejected.
  • --spec-version (existing flag, feat: add --spec-version option to tier-check #176) now also pins the spec release the gap is measured against. Absent = latest stable, as before. The draft alias has no GitHub release to pin, so draft runs keep tracking the latest stable release.
  • SEP-1730 windows replace the 30d constant: gap <= 0 days = pass (Tier 1, before the spec release), <= 183 days = partial (Tier 2, within six months), beyond = fail. Derived once and exposed as meets_tier1_window / meets_tier2_window, which computeTier reads instead of pattern-matching status strings (same shape as p0's all_p0s_resolved_within_7d/14d).
  • Unexpected API errors still produce skipped, but skipped now blocks Tier 1 the same way skipped conformance already does.
  • The markdown and terminal renderers print reason when there is no gap to show.
  • Output stays backward compatible: all existing SpecTrackingResult fields keep emitting; sdk_release_within_30d is still computed but no longer read by tier logic.
  • Docs: the mcp-sdk-tier-audit skill's CLI table gains both flags; report template row 4b gains a PARTIAL slot.

Behavior changes to be aware of

  1. Gaps of 1-30 days used to pass; they are now partial (blocks Tier 1, satisfies Tier 2). The old 30-day pass window matched no documented requirement.
  2. Invocations that already pass --spec-version for scenario filtering now also pin spec-tracking to that version. This is the issue's intent (measure against the spec version the submission claims), but it can change scores for CI that pins older versions.
  3. skipped spec_tracking now appears in tier1_blockers as spec_tracking (skipped).

Validation

  • npx vitest run: 40 files, 449 tests, 0 failures (baseline before the change: 37 files, 419 tests). The new spec-tracking.test.ts, tier-logic.test.ts and index.test.ts cover the window boundaries (-5, 0, 1, 183, 184 days), missing-tag and draft-release failures, API-error skips, CLI forwarding, and the tier-logic blocker matrix.

  • npm run check (typecheck + eslint + prettier): clean.

  • End to end against the motivating case:

    tier-check --repo modelcontextprotocol/java-sdk --sdk-release-tag v1.0.0 --spec-version 2025-06-18 --skip-conformance --output json

    "spec_tracking": {
      "status": "fail",
      "latest_spec_release": "2025-06-18T20:09:59Z",
      "latest_sdk_release": "2026-02-23T13:20:26Z",
      "sdk_release_within_30d": false,
      "days_gap": 250,
      "target_spec_tag": "2025-06-18",
      "submitted_sdk_tag": "v1.0.0",
      "meets_tier1_window": false,
      "meets_tier2_window": false
    }

    Same command with --sdk-release-tag v9.9.9 (nonexistent), terminal output:

    ✗ Spec Tracking  sdk release tag not found: v9.9.9
    

Scope cuts, for maintainers to call

  • The check accepts any exact spec release tag including prereleases (unit-tested), but the CLI cannot reach the RC case: --spec-version goes through resolveSpecVersion, whose vocabulary is the released dated versions plus draft. Widening that vocabulary, or adding a dedicated flag, is your call; I can extend this PR either way.
  • 183 days is my reading of "within six months"; a different constant is a one-line change (TIER2_WINDOW_DAYS).
  • sdk_release_within_30d is kept for output compatibility; renaming or dropping it is a one-line decision I did not take unilaterally.
  • Not run: the GitHub Action path and the full mcp-sdk-tier-audit skill flow.

…et spec version

The spec-tracking check compares the latest spec release with the first
SDK release published after it and passes on a gap of <= 30 days. That
validates nothing about a tier submission: the measured release can be
an unrelated patch, the spec version is not the one the submission
claims, and 30 days matches no documented requirement (SEP-1730 says
"before the new spec version release" for Tier 1 and "within six
months" for Tier 2). The Java SDK Tier 2 assessment showed a 9d-gap
PASS while the submitted v1.0.0 shipped 250 days after the 2025-06-18
spec it claims. computeTier also only blocks Tier 1 on status 'fail',
so a skipped result (API error, missing releases) scores green.

Add --sdk-release-tag to pin the check to the exact submitted release
(getReleaseByTag, no normalization); the existing --spec-version flag
now also pins the target spec release. Replace the 30-day constant with
the SEP-1730 windows (pass <= 0d, partial <= 183d, fail beyond),
derived once and exposed as meets_tier1_window / meets_tier2_window,
which computeTier reads instead of pattern-matching statuses (same
shape as p0's all_p0s_resolved_within_7d/14d). Submitted tags that do
not resolve fail with the attempted tag in `reason` (precedent: modelcontextprotocol#372
moved missing prerequisites from skip to fail); unexpected API errors
still skip, but skipped now blocks Tier 1 like skipped conformance
already does. Both renderers print `reason` when there is no gap to
show. Existing SpecTrackingResult fields keep emitting;
sdk_release_within_30d is no longer read by tier logic.

Refs modelcontextprotocol#167.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spec tracking check should validate the submitted SDK release, not first-release-after-spec

1 participant