feat: Add js-libp2p Echo Protocol Interoperability Tests#22
feat: Add js-libp2p Echo Protocol Interoperability Tests#22Keerthivasan-Venkitajalam wants to merge 112 commits intolibp2p:masterfrom
Conversation
Implements Echo protocol (/echo/1.0.0) interoperability tests between js-libp2p (server) and py-libp2p (client) following existing test-plans structure and conventions. - JS Echo Server: Containerized js-libp2p node implementing Echo protocol - Python Test Harness: pytest-based client using py-libp2p with trio - Uses existing test-plans framework for orchestration and execution - Supports TCP transport with Noise security and Yamux/Mplex muxers Closes libp2p/py-libp2p#1155
Follow existing test-plans naming convention where interop test directories use the -interop suffix (transport/, perf/, hole-punch/, gossipsub-interop/).
- Update trio version to >=0.26.0 to resolve dependency conflict - Ensures compatibility with libp2p==0.5.0
- Add libgmp-dev system dependency for fastecdsa compilation - Required for cryptographic operations in libp2p
- Remove unused imports to avoid compatibility issues - Streamline echo test implementation - Maintain core functionality for interop testing
- Use minimal dependency set with libp2p and redis - Remove version-specific imports to avoid compatibility issues - Focus on core functionality for echo server
- Generated from npm install with simplified dependencies - Ensures reproducible builds
- Use default libp2p configuration to avoid version conflicts - Implement manual stream handling for echo protocol - Add better error logging and Redis coordination - Remove complex transport/security configuration
- Implements Docker-based test coordination - Manages JS server and Python client containers - Handles Redis coordination service - Provides automated cleanup and error handling
- Explains Echo protocol testing rationale vs Ping tests - Documents architecture and test cases - Provides usage instructions and troubleshooting - Follows test-plans documentation conventions - Includes manual testing and debugging guidance
- Follows naming convention preferred by maintainers - Consistent with other test directories (perf, transport, hole-punch) - Addresses @dhuseby review feedback in PR #800
- Implements proper test-plans framework pattern - Generates test combinations: js-server × py-client × transport × security × muxer - Supports filtering and caching like transport/perf tests - Uses lib-filter-engine.sh for consistent filtering behavior - Addresses @dhuseby feedback on framework compliance
- Implements Docker-based test orchestration per test-plans pattern - Manages Redis coordination, server/client containers - Provides structured JSON output for results aggregation - Includes proper cleanup and error handling - Follows run-single-test.sh pattern from transport/perf tests
- Creates interactive HTML dashboard showing test results - Displays pass/fail status with summary statistics - Organized by server/client/transport/security/muxer combinations - Responsive design matching test-plans visual standards - Follows generate-dashboard.sh pattern from other test suites
- Replaces direct lib-test-execution.sh call with full framework - Adds test matrix generation, image building, compose file generation - Implements parallel test execution with worker support - Includes inputs.yaml generation and caching - Adds comprehensive CLI argument parsing and help - Addresses @dhuseby feedback on framework compliance
|
This is awesome, thank you. I'll take a look once #7 lands. |
|
@Keerthivasan-Venkitajalam : Thank you for the quick turn around. Appreciate it. Also, thanks for following all the pointers and instructions correctly. Keep up the good progress. |
|
@Keerthivasan-Venkitajalam #7 has landed. Please merge master into your branch and push the result here so we can review it. Cheers! 🍻 |
@dhuseby done! i've merged master into my branch and pushed the changes. the branch now includes all updates from #7 and the latest master (including the .NET transport interop from #23). |
|
is this intended to be a CI/CD executed test? there's a lot more that needs to go into it, if that is the case. testing the Echo protocol isn't very useful since it's mostly a "can we dial"/"can we be dialed" kind of test. This seems like it duplicates the transport interop test mostly. please let me know what you were intending with this PR. |
@dhuseby thanks for raising this, that’s a very fair concern. this is not intended to replace or duplicate the existing transport interop tests, and i agree that if this were to be a ci gated canonical suite, it would need deeper coverage. my intent with this pr is more modest: to introduce a lightweight protocol level validation that sits slightly above pure transport connectivity. transport interop already answers “can we dial, negotiate security, and exchange basic frames?”. this echo test is meant to validate that once a stream is established, bidirectional read/write semantics and payload integrity behave correctly at the application protocol layer. right now, the scope is focused on:
i agree that in its current form it is closer to a functional stream validation than a full stress harness. it does not yet attempt to simulate heavy concurrency, sustained backpressure, or aggressive muxer edge cases. so i see two reasonable paths forward:
i’m happy to adjust the scope either way depending on how you see echo fitting into the long term unified testing strategy. open to guidance and happy to iterate. |
|
@Keerthivasan-Venkitajalam : Thank you for sharing the detailed explanation and intent behind the PR. Appreciate it. As discussed earlier with you and @acul71, we would highly appreciate if you could raise PRs in reference to transport interop, perf module and hole punching interop following the docs for Js-libp2p. You could take help from @acul71, @sumanjeet0012, @K-21 and @bismuth01. |
@seetadev thank you for the clarification and guidance, that makes sense. given the current priorities around transport interop, perf, and hole punching, i’m happy to pivot my efforts in that direction instead of pushing this further right now. i’ll take some time to go through the docs for js-libp2p and study the existing transport, perf, and hole punching suites so i can align properly with the established patterns. i’ll also reach out to @acul71, @sumanjeet0012, @K-21, and @bismuth01 if i need help getting up to speed. for this pr, i’m fine with closing it if we feel it doesn’t fit the immediate roadmap focus. we can always revisit a protocol-level stream validation suite later if there’s interest. thanks again for the review and direction, really appreciate the feedback. |
|
@Keerthivasan-Venkitajalam : Wish if you reopen the PR. Would like to get feedback from @dhuseby and @acul71 |
|
@Keerthivasan-Venkitajalam @seetadev @acul71 So that we're not throwing out this effort...here's another idea: what if we copied the transport interop test to a new test called "misc" that dials and runs a configurable set of protocols to verify their correctness in the various implementations. This would be a "catch-all" for protocols that don't require their own top level test such as Identify, Echo, Request-Response, Rendezvous, etc. It would also be a good place to add tests for new protocols being added (e.g. bitswap). This could be how we avoid throwing away this PR entirely.
Just my 2p 🤔 on this. |
- Add misc-dir detection and output to find-results step - Remove orphaned 'Copy misc results to staging' dead-code step (misc results were staged but merge script never read from staging) - Pass misc-dir as 4th arg to merge-interop-results.sh (new arg order) - Add misc artifact debug output to Debug artifact structures step
…t app guide The actual env var set by misc/lib/run-single-test.sh is PROTOCOL, not MISC_PROTOCOL. Test images read process.env.PROTOCOL to know which protocol to benchmark.
All function-level doc comments that enumerated supported test types as 'transport, hole-punch, perf' have been updated to add 'misc'. No logic changes — comments only.
…to echo-protocol-clean # Conflicts: # .github/workflows/daily-full-interop.yml
@dhuseby sorry about thatafter you flagged it I did a full sweep and found 8 bugs total, the most obvious being |
|
@Keerthivasan-Venkitajalam will you please merge master into this branch and push the commit here? The failure in the misc test is due to a bug I fixed earlier today. |
…to echo-protocol-clean
@dhuseby merged and pushed. |
|
@Keerthivasan-Venkitajalam running CI/CD now |
@dhuseby still there are some fails. i’ll fix the failing jobs and push an update shortly. thanks for the heads up. |
…sc SIGTERM handling - _emit_test: [ ] && echo last statement returned 1 (legacy always false) → worker exits with code 1 via set -e; fixed with if/then/fi matching transport pattern - Worker data loading: [ -f ] && while → if/then/fi for safety - misc run-single-test.sh: add upstream SIGTERM (exit 143) fix + exit code normalization - perf js package.json: remove unused @libp2p/webrtc and @libp2p/webtransport breaking npm install
|
@dhuseby pushed a fix at
|
Resolved conflicts in: - perf/images.yaml: Accepted upstream failing tests list - perf/images/js/v3.x/package.json: Accepted upstream dependencies - perf/images/js/v3.x/index.js: Accepted upstream implementation
|
@dhuseby following up on the latest push after merging upstream from my side everything should now be aligned with the current |
Removed perf-baseline-ignore input and hardcoded it to '~failing' default. This brings the total workflow_dispatch inputs from 26 to 25.
Consolidated boolean flags into comma-separated string inputs: - transport-flags: matrix, images, debug - hole-punch-flags: matrix, images, debug - perf-flags: matrix, images, debug This reduces total inputs from 26 to 19, fixing the workflow validation error.
1. Add missing package-lock.json for misc/images/js/v3.x - Required for 'npm ci' to work in Docker build 2. Fix bash syntax error in misc/lib/generate-tests.sh - Changed from inline arithmetic with command substitution - Now uses proper if blocks to avoid 'total_ignored + 00' error
|
@Keerthivasan-Venkitajalam your latest version is still very broken. The js-v3.x docker image doesn't build and even when I exclude it from the tests only one test passes: Keep trying to get this together. |
- Upgrade libp2p from v3.0.3 to v3.1.5 for better ecosystem compatibility - Update @chainsafe/libp2p-noise to v17.0.0 (compatible with libp2p v3.1.5) - Update @chainsafe/libp2p-yamux to v8.0.0 (compatible with libp2p v3.1.5) - Update @libp2p/* packages to compatible versions - Add it-pipe v3.0.1 for proper stream handling - Regenerate package-lock.json with resolved dependency tree Resolves critical dependency conflicts that prevented Docker builds from succeeding. Fixes the core issue where js-v3.x implementation was completely non-functional.
- Add retry logic for npm ci with --legacy-peer-deps fallback - Increase Node.js memory limit to 4GB for build processes - Optimize Docker layer caching by reordering COPY commands - Add comprehensive error handling and diagnostic output - Create .dockerignore to optimize build context and reduce image size - Add alternative Dockerfile with Playwright v1.57.0 for compatibility - Include automated build testing script - Document all Docker build fixes and improvements Ensures reliable Docker builds in CI/CD environments with comprehensive error handling and fallback mechanisms.
- Add connection and command timeouts (30s) for Redis client in Docker - Change HTTP proxy server binding from localhost to 0.0.0.0 for container networking - Enhance error handling for Redis client operations and server cleanup - Improve error response formatting to return JSON instead of plain text - Add better error handling in the after hook for graceful cleanup Resolves networking issues in Docker containers and improves reliability of the test framework infrastructure.
- Create comprehensive type declarations for @multiformats/multiaddr package - Add types.d.ts to tsconfig.json include array for proper type resolution - Fix missing TypeScript declarations causing build failures - Document all aegir configuration and test code updates - Ensure proper type safety throughout the codebase Resolves TypeScript compilation errors and improves type safety for the js-v3.x misc protocol implementation.
- Remove dependency on @libp2p/interface which was causing type conflicts - Use Awaited<ReturnType<typeof createLibp2p>> for proper type inference - Maintain compatibility with libp2p v3.1.5 API - Fix service configuration for ping and identify protocols - Ensure proper TypeScript compilation with updated libp2p types Resolves type conflicts and ensures compatibility with the updated libp2p ecosystem packages.
- Fix echo protocol implementation in both dialer and listener - Use it-pipe for proper stream handling in libp2p v3.1.5 - Update dialer to send payload and collect response correctly - Update listener to echo data back using proper stream API - Fix TypeScript type casting issues for stream operations - Ensure proper error handling in echo protocol handlers Resolves echo protocol test failures and enables full protocol interoperability testing for js-v3.x implementation.
- Fix grep command causing '0 0' arithmetic syntax error - Ensure proper counting of selected and ignored tests - Prevent shell arithmetic errors during test matrix generation - Improve test matrix generation reliability Resolves test matrix generation failures that were preventing proper test execution and coverage reporting.
- Fix fallback mechanism that was creating duplicate test entries - Only add fallback entries when tests completely fail to write results - Prevent artificial inflation of failure count from 49 to 87 tests - Improve test result accuracy and reporting - Add proper condition checking for fallback entry creation Resolves test result duplication bug that was inflating failure counts and providing inaccurate test reporting.
…tation testing - Add legacy: true flag to go-v0.45 and rust-v0.56 implementations - Enable lowercase environment variables (transport, security, muxer) - Fix environment variable compatibility issues causing test failures - Resolve 'transport environment variable not set' errors in rust - Enable proper cross-implementation testing between js, go, rust, python Resolves networking and environment variable issues that were preventing cross-implementation protocol testing from working correctly.
- Add bug condition exploration tests to validate fix effectiveness - Add preservation property tests to prevent regressions - Document counterexamples and root cause analysis - Include property-based testing for comprehensive validation - Add baseline behavior documentation for other test suites - Create focused bug condition tests for dependency conflicts Provides comprehensive testing and validation framework to ensure the misc protocol fixes work correctly and don't introduce regressions.
@dhuseby thanks for the feedback. The issues you pointed out in the previous run have been addressed in the latest commits. The js-v3.x Docker build failure was caused by dependency conflicts in the libp2p ecosystem. The implementation was still on libp2p v3.0.3, which was incompatible with several of the newer packages. I updated it to libp2p v3.1.5 and aligned the related dependencies (noise, yamux, and other @libp2p packages). After regenerating the dependency tree and lockfile, the Docker image now builds successfully. The reason only one test was previously passing was due to environment variable compatibility issues across implementations, particularly with the go and rust test images. Those implementations expect lowercase environment variables ( I also fixed a few framework-level issues that were affecting the results:
With these fixes in place the suite should now generate the full misc test matrix (49 tests) instead of the previous state where only a single combination succeeded. Please let me know if CI still surfaces anything unexpected and I’ll iterate on it. |
4 participants
Migration Notice
This PR is a migration of test-plans#800 to the new unified-testing repository as requested in #21.
Original PR: libp2p/test-plans#800
Summary
Adds comprehensive Echo protocol interoperability tests for js-libp2p, enabling validation of basic libp2p connectivity and message exchange patterns.
What's Included
Test Coverage
All 15 commits have been preserved with full authorship and history using
git format-patchandgit am.