Run every referee scenario in the server conformance leg

## Motivation and Context

The server conformance leg ran the referee's default `active` suite, which leaves out the extension-tagged scenarios,
while the client leg has selected every scenario with `--suite all`. The Skills server implementation
therefore never met the referee's `sep-2640-skills-*` server scenarios, although the conformance server already serves
the `conformance/skills` directory for them, and the `all` suite's other additions (stateless server, SSE polling, caching,
standard header validation, `input-required` results) were not run either.

The server leg now selects `--suite all` the way the client leg does, with a requirement set or a single scenario
still replacing the selection. Of the 34 scenarios this adds, 23 pass. The rest reach three areas the SDK does not implement
and are listed in `expected_failures.yml` the way the TypeScript and Python SDKs list theirs: the Tasks extension (SEP-2663);
the server side of SEP-2243 custom headers, where the server does not bind `Mcp-Param-*` headers to `x-mcp-header` tool parameters
and the conformance server exposes no such tool, so the referee reports the scenario's checks as untestable; and SEP-1699 resumability,
where a POST SSE stream carries no priming event or `retry` field, which the referee reports as warnings that its baseline counts as failures.
`tasks-status-notifications` is left out of the baseline because the referee runs no check in it, and the baseline check would flag
an entry for a passing scenario as stale.

## How Has This Been Tested?

test/conformance/server_runner_test.rb, and `bundle exec rake conformance:test`: the server leg runs 65 scenarios,
the 11 baselined failures reported as expected and the three `sep-2640-skills-*` scenarios passing, where it ran 31 before the change.
`REQUIREMENTS=2026-07-28` still runs the frozen set without a suite.

## Breaking Changes

None.

Author: koic

.github/workflows/conformance.yml

@@ -29,7 +29,7 @@ jobs:
       - run: bundle exec rake conformance:test
 
   # Frozen requirement sets for SEP-1730 tier scoring: each spec revision's scenarios run at
-  # that revision's own wire version, unlike the moving active suite above.
+  # that revision's own wire version, unlike the moving full suite above.
   requirements-conformance:
     runs-on: ubuntu-latest
     continue-on-error: true

conformance/README.md

@@ -15,10 +15,12 @@ Validates the Ruby SDK's conformance to the MCP specification using [`@modelcont
 bundle exec rake conformance:test
 ```
 
-Runs both server and client conformance tests in sequence. Server conformance starts the
-conformance server and tests it. Client conformance spawns test servers for each scenario
-and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
-are allowed to fail without affecting the exit code.
+Runs both server and client conformance tests in sequence. Server conformance starts the conformance server and tests it.
+Client conformance spawns test servers for each scenario and invokes `conformance/client.rb` against them.
+Both legs select every scenario the referee ships (`--suite all`), the extension-tagged ones included,
+so the Skills server and client implementations are checked. Scenarios listed in `expected_failures.yml`,
+which cover the areas the SDK does not implement, are allowed to fail without affecting the exit code;
+the referee flags an entry as stale once its scenario passes.
 
 ### Environment variables
 
@@ -55,7 +57,7 @@ bundle exec rake conformance:server PORT=3000
 bundle exec rake conformance:server
 
 # Terminal 2: run all scenarios
-npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp
+npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --suite all
 
 # Terminal 2: run a single scenario
 npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping

conformance/expected_failures.yml

@@ -1,4 +1,23 @@
-server: []
+server:
+  # SEP-1699 resumability: the server sends no priming event or `retry` field on a POST SSE stream and replays
+  # nothing on `Last-Event-ID`; the referee warns about both, and the baseline counts a warning as a failure.
+  - server-sse-polling
+  # SEP-2243 custom headers: the server does not bind `Mcp-Param-*` headers to `x-mcp-header` tool parameters
+  # (the client side sends them), so the conformance server exposes no such tool and the referee reports
+  # the scenario's checks as untestable.
+  - http-custom-header-server-validation
+  # SEP-2663 (io.modelcontextprotocol/tasks): the server does not implement the tasks extension.
+  # `tasks-status-notifications` is intentionally absent: the referee runs no check in it, and a baseline entry
+  # for a scenario with no failing check is flagged stale.
+  - tasks-lifecycle
+  - tasks-capability-negotiation
+  - tasks-wire-fields
+  - tasks-request-state-removal
+  - tasks-mrtr-input
+  - tasks-request-headers
+  - tasks-dispatch-and-envelope
+  - tasks-required-task-error
+  - tasks-mrtr-composition
 client:
   # Unimplemented authorization extensions.
   - auth/dpop

conformance/server_runner.rb

@@ -33,9 +33,17 @@ def build_command
       expected_failures_yml = File.expand_path("expected_failures.yml", __dir__)
 
       npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance@alpha", "server", "--url", "http://localhost:#{@port}/mcp"]
-      npx_command += ["--scenario", @scenario] if @scenario
+      # A requirement set already fixes which scenarios run, so it replaces the suite selection.
+      # The referee's default `active` suite leaves out the extension-tagged scenarios (Skills, Tasks), which `all` selects,
+      # as the client leg does; the areas the SDK does not implement are listed in expected_failures.yml.
+      npx_command += if @requirements
+        ["--requirements", @requirements]
+      elsif @scenario
+        ["--scenario", @scenario]
+      else
+        ["--suite", "all"]
+      end
       npx_command += ["--spec-version", @spec_version] if @spec_version
-      npx_command += ["--requirements", @requirements] if @requirements
       npx_command += ["--verbose"] if @verbose
       npx_command += ["--expected-failures", expected_failures_yml]
       npx_command

test/conformance/server_runner_test.rb

@@ -7,7 +7,7 @@ module Conformance
   class ServerRunnerTest < ActiveSupport::TestCase
     EXPECTED_FAILURES_YML = File.expand_path("../../conformance/expected_failures.yml", __dir__)
 
-    test "build_command runs the active suite by default" do
+    test "build_command runs the full suite by default" do
       command = ServerRunner.new.send(:build_command)
 
       assert_equal(
@@ -18,6 +18,28 @@ class ServerRunnerTest < ActiveSupport::TestCase
           "server",
           "--url",
           "http://localhost:#{Server::DEFAULT_PORT}/mcp",
+          "--suite",
+          "all",
+          "--expected-failures",
+          EXPECTED_FAILURES_YML,
+        ],
+        command,
+      )
+    end
+
+    test "build_command keeps the single scenario selection" do
+      command = ServerRunner.new(scenario: "ping").send(:build_command)
+
+      assert_equal(
+        [
+          "npx",
+          "--yes",
+          "@modelcontextprotocol/conformance@alpha",
+          "server",
+          "--url",
+          "http://localhost:#{Server::DEFAULT_PORT}/mcp",
+          "--scenario",
+          "ping",
           "--expected-failures",
           EXPECTED_FAILURES_YML,
         ],