[RFC] Tests using the Everything server #582
Conversation
- Move auto open disable closer to server startup (`playwright test ...` doesn't need that env) - Emit stdout for operational clarity and debugging - Extract common inspector URL to a variable
Implements basic end-to-end tests for the Everything server, focusing on connection setup and tool functionality validation. Current test coverage includes: - Server connection via STDIO transport - Tools listing and discovery - Single tool execution (structuredOutput) with various input scenarios - Error handling for missing required inputs - Proper disconnect handling with expected network error filtering The Everything server implements many MCP protocol features and serves as a comprehensive example server, but this initial test implementation focuses on establishing the test framework and validating core tool execution. Auth is disabled in the test configuration because the server-generated token would need to be parsed from the emitted startup URL, which is complicated if possible. Includes dedicated Playwright configuration for Everything server testing with appropriate timeouts and debugging setup.
Adds GitHub Actions workflow to run e2e tests against the Everything MCP
server. The workflow clones both inspector and servers repos, builds
the Everything server, and runs Playwright tests against it.
Required solving several CI-specific challenges:
- Repository layout: Used explicit checkout paths to place inspector and servers
repos as siblings, preventing GitHub Actions from nesting servers inside inspector
- Server setup: Everything server needs npm install and build before testing
- Dependency management: Created custom setup-playwright action to handle
package.json location and dependency caching across multiple repos
|
Hi @richardkmichael I like this idea. I wanted to see if you already knew about the Community Working Groups, there is more info here: https://github.com/modelcontextprotocol-community/working-groups The reason why I mention it, is that we've been talking on that Discord about putting together a working group for community-driven reference implementation, validation etc. so I think it could be a good place to have more discussions around stuff like this. |
Thank you! I didn't know about it, and I will take a look. |
Add Playwright e2e tests which connect to the reference Everything server.
This implementation is in a fork:
Motivation and Context
The Inspector lacks automated testing against real MCP servers, making it difficult to catch regressions and validate new functionality. (Much of the UI depends on a connected server.)
The Everything server provides example implementations of many MCP protocol features.
Together, these create an opportunity for a feedback loop to drive MCP specification validation and compliance:
I think there is a lot of potential in this direction.
I'm particularly interested in validation and compliance of MCP clients and servers, and how I can help. I know about the focus on validation in the Roadmap, and SDK compliance spec schema.
How Has This Been Tested?
Running in GitHub Actions, sample run
Request for Comments
Seeking feedback on the concept. But specifically:
Current scope
structuredContentimplements the new MCP 2025-06-18 specification feature)Current limitations
structuredContenttool (can'tnpx ...from a sub-package branch)Next Steps
npx modelcontextprotocol/server-everything(oncestructuredContenttool is merged; otherwise, change the tested tool)
Future Test Coverage
Elicitation, Roots, Change Notifications in various contexts