Add plugin-test tool for testing plugin loading

[jpshackelford]

Summary

Add a generalized plugin testing tool that can test plugin loading via the OpenHands Cloud API. This tool can be used to verify that plugins work correctly by launching conversations and checking that the agent response matches expected patterns.

Features

• Automated testing: Launch conversations with any plugin and verify responses
• Flexible matching: Substring or regex pattern matching
• Configurable marketplace: Test plugins from any marketplace repository (defaults to github:OpenHands/extensions)
• CI/CD ready: Exit codes indicate pass/fail for integration into pipelines
• Verbose mode: Detailed output for debugging API interactions

Usage

Command Line

export OPENHANDS_API_KEY="sk-oh-..."

# Test the city-weather plugin
./plugins/plugin-test/scripts/test_plugin.sh \
  --plugin "plugins/city-weather" \
  --message "/city-weather:now Tokyo" \
  --expect "Weather Report"

# Test with custom marketplace
MARKETPLACE_REPO="github:myorg/my-plugins" ./plugins/plugin-test/scripts/test_plugin.sh \
  --plugin "plugins/my-plugin" \
  --message "/my-plugin:test" \
  --expect "success"

Slash Command

/plugin-test:test --plugin "plugins/city-weather" --message "/city-weather:now Tokyo" --expect "Weather Report"

Environment Variables

Variable	Default	Description
OPENHANDS_API_KEY	(required)	Your OpenHands API key
OPENHANDS_URL	https://app.all-hands.dev	OpenHands Cloud URL
MARKETPLACE_REPO	github:OpenHands/extensions	Plugin marketplace repository
MARKETPLACE_REF	main	Git ref for marketplace

Plugin Structure

plugins/plugin-test/
├── .claude-plugin/
│   └── plugin.json      # Plugin manifest
├── commands/
│   └── test.md          # Slash command definition
├── scripts/
│   └── test_plugin.sh   # Test execution script
└── README.md

How It Works

1. Create Conversation: POST to /api/v1/app-conversations with the plugin spec
2. Wait for Sandbox: Poll /api/v1/app-conversations/start-tasks/search until status is READY
3. Fetch Events: Query the Agent Server for conversation events
4. Verify Response: Check if the agent's response matches the expected pattern

Related

This tool is based on the demo script from jpshackelford/openhands-sample-plugins, generalized to work with any plugin and any marketplace.

@jpshackelford can click here to continue refining the PR