Add plugin-test tool for testing plugin loading

.plugin/marketplace.json

@@ -376,6 +376,19 @@
                 "preview",
                 "hosting"
             ]
+        },
+        {
+            "name": "plugin-test",
+            "source": "../plugins/plugin-test",
+            "description": "Test plugin loading via OpenHands Cloud API - launch conversations with plugins and verify responses match expected patterns.",
+            "category": "testing",
+            "keywords": [
+                "test",
+                "plugin",
+                "verification",
+                "api",
+                "conversation"
+            ]
         }
     ]
 }

plugins/plugin-test/.claude-plugin/plugin.json

@@ -0,0 +1,13 @@
+{
+  "name": "plugin-test",
+  "version": "1.0.0",
+  "description": "Test plugin loading via OpenHands Cloud API - launch conversations with plugins and verify responses",
+  "author": {
+    "name": "OpenHands",
+    "email": "contact@all-hands.dev"
+  },
+  "homepage": "https://github.com/OpenHands/extensions",
+  "repository": "https://github.com/OpenHands/extensions",
+  "license": "MIT",
+  "keywords": ["test", "plugin", "verification", "conversation", "api"]
+}

plugins/plugin-test/README.md

@@ -0,0 +1,203 @@
+# Plugin Test Tool
+
+Test plugin loading via the OpenHands Cloud API. Launch conversations with any plugin from a marketplace, wait for the agent to respond, and verify the response matches an expected string or pattern.
+
+## Features
+
+- **Automated testing**: Launch conversations and verify responses programmatically
+- **Flexible matching**: Substring or regex pattern matching
+- **Configurable marketplace**: Test plugins from any marketplace repository
+- **Detailed output**: Verbose mode for debugging API interactions
+- **CI/CD ready**: Exit codes indicate pass/fail for integration into pipelines
+
+## Quick Start
+
+### Prerequisites
+
+- `curl` and `jq` installed
+- OpenHands API key (from [app.all-hands.dev](https://app.all-hands.dev))
+
+### Basic Usage
+
+```bash
+export OH_API_KEY="sk-oh-your-api-key"
+
+# Test the city-weather plugin
+./scripts/test_plugin.sh \
+  --plugin "plugins/city-weather" \
+  --message "/city-weather:now Tokyo" \
+  --expect "Weather Report"
+```
+
+### Using the Slash Command
+
+If you have this plugin loaded, you can invoke it directly:
+
+```
+/plugin-test:test --plugin "plugins/city-weather" --message "/city-weather:now Tokyo" --expect "Weather Report"
+```
+
+---
+
+## Command Reference
+
+```
+Usage: test_plugin.sh [OPTIONS]
+
+Required Arguments:
+  --plugin PATH       Path to plugin within the marketplace repo
+  --message TEXT      Initial message to send
+  --expect STRING     String or regex pattern to expect in the response
+
+Options:
+  --regex             Treat --expect as a regex pattern (default: substring)
+  --open              Open the conversation in a browser when ready
+  --verbose           Show detailed output including API responses
+  --max-wait SECONDS  Maximum time to wait for conversation (default: 180)
+  --poll SECONDS      Polling interval in seconds (default: 3)
+  --help              Show help message
+```
+
+---
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `OH_API_KEY` | (required) | Your OpenHands API key |
+| `OPENHANDS_API_KEY` | - | Alternative name for API key (fallback) |
+| `OPENHANDS_URL` | `https://app.all-hands.dev` | OpenHands Cloud URL |
+| `MARKETPLACE_REPO` | `github:OpenHands/extensions` | Plugin marketplace repository |
+| `MARKETPLACE_REF` | `main` | Git ref (branch/tag) for marketplace |
+
+---
+
+## Examples
+
+### Test city-weather plugin
+
+```bash
+export OH_API_KEY="sk-oh-..."
+
+./scripts/test_plugin.sh \
+  --plugin "plugins/city-weather" \
+  --message "/city-weather:now Tokyo" \
+  --expect "Weather Report"
+```
+
+### Test magic-test plugin with regex
+
+```bash
+./scripts/test_plugin.sh \
+  --plugin "plugins/magic-test" \
+  --message "What happens if I say alakazam?" \
+  --expect "Plugin loaded successfully" \
+  --regex
+```
+
+### Test with custom marketplace
+
+```bash
+export MARKETPLACE_REPO="github:myorg/my-plugins"
+export MARKETPLACE_REF="develop"
+
+./scripts/test_plugin.sh \
+  --plugin "plugins/my-plugin" \
+  --message "/my-plugin:hello" \
+  --expect "Hello World"
+```
+
+### Verbose mode for debugging
+
+```bash
+./scripts/test_plugin.sh \
+  --plugin "plugins/city-weather" \
+  --message "/city-weather:now London" \
+  --expect "Temperature" \
+  --verbose
+```
+
+### Open in browser after test
+
+```bash
+./scripts/test_plugin.sh \
+  --plugin "plugins/city-weather" \
+  --message "/city-weather:now Paris" \
+  --expect "Weather" \
+  --open
+```
+
+---
+
+## CI/CD Integration
+
+The script returns exit codes suitable for CI pipelines:
+
+| Exit Code | Meaning |
+|-----------|---------|
+| 0 | Test passed - response matched expected pattern |
+| 1 | Test failed - response didn't match, or error occurred |
+
+### GitHub Actions Example
+
+```yaml
+- name: Test city-weather plugin
+  env:
+    OH_API_KEY: ${{ secrets.OH_API_KEY }}
+  run: |
+    ./plugins/plugin-test/scripts/test_plugin.sh \
+      --plugin "plugins/city-weather" \
+      --message "/city-weather:now Tokyo" \
+      --expect "Weather Report"
+```
+
+---
+
+## How It Works
+
+1. **Create Conversation**: POST to `/api/v1/app-conversations` with the plugin spec
+2. **Wait for Sandbox**: Poll `/api/v1/app-conversations/start-tasks/search` until status is `READY`
+3. **Fetch Events**: Query the Agent Server for conversation events
+4. **Verify Response**: Check if the agent's response matches the expected pattern
+
+### Architecture
+
+```
+┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
+│   test_plugin   │────▶│   App Server    │────▶│  Agent Server   │
+│     script      │     │  (Cloud URL)    │     │  (In Sandbox)   │
+└─────────────────┘     └─────────────────┘     └─────────────────┘
+        │                      │                        │
+        │ 1. Create conv       │ 2. Start sandbox       │
+        │    with plugin       │    load plugin         │
+        │                      │                        │
+        │ 3. Poll for ready    │                        │
+        │                      │                        │ 4. Agent runs,
+        │ 5. Fetch events ◀────────────────────────────│    responds
+        │                                               │
+        │ 6. Verify response                            │
+        └───────────────────────────────────────────────┘
+```
+
+---
+
+## Plugin Structure
+
+```
+plugin-test/
+├── .claude-plugin/
+│   └── plugin.json      # Plugin manifest
+├── commands/
+│   └── test.md          # Slash command definition
+├── scripts/
+│   └── test_plugin.sh   # Test execution script
+└── README.md
+```
+
+---
+
+## Related Resources
+
+- [OpenHands Cloud](https://app.all-hands.dev)
+- [OpenHands API Documentation](https://docs.openhands.dev)
+- [Plugin Marketplace](https://github.com/OpenHands/extensions)

plugins/plugin-test/commands/test.md

@@ -0,0 +1,103 @@
+---
+allowed-tools: Bash(*)
+argument-hint: --plugin <path> --message <text> --expect <pattern> [--regex] [--open]
+description: Test plugin loading via OpenHands Cloud API - creates a conversation with the specified plugin and verifies the response
+---
+
+# Test Plugin Loading
+
+Launch a conversation with a specified plugin loaded via the OpenHands Cloud API, wait for the agent response, and verify it matches an expected string or pattern.
+
+## Instructions
+
+Run the test script with the provided arguments: **$ARGUMENTS**
+
+```bash
+# Navigate to the plugin-test scripts directory and run the test
+cd /path/to/extensions/plugins/plugin-test/scripts
+
+# Run the test with the provided arguments
+./test_plugin.sh $ARGUMENTS
+```
+
+If the script is not available locally, you can download and run it:
+
+```bash
+curl -sL "https://raw.githubusercontent.com/OpenHands/extensions/main/plugins/plugin-test/scripts/test_plugin.sh" -o /tmp/test_plugin.sh
+chmod +x /tmp/test_plugin.sh
+/tmp/test_plugin.sh $ARGUMENTS
+```
+
+## Required Environment Variables
+
+Before running, ensure these are set:
+
+```bash
+export OH_API_KEY="sk-oh-your-api-key"
+# Alternative: OPENHANDS_API_KEY is also accepted as a fallback
+
+# Optional: override defaults
+export OPENHANDS_URL="https://app.all-hands.dev"
+export MARKETPLACE_REPO="github:OpenHands/extensions"
+export MARKETPLACE_REF="main"
+```
+
+## Arguments
+
+| Argument | Required | Description |
+|----------|----------|-------------|
+| `--plugin PATH` | Yes | Path to plugin within marketplace (e.g., `plugins/city-weather`) |
+| `--message TEXT` | Yes | Initial message to send (e.g., `/city-weather:now Tokyo`) |
+| `--expect STRING` | Yes | String or regex pattern to expect in the response |
+| `--regex` | No | Treat `--expect` as a regex pattern |
+| `--open` | No | Open the conversation in a browser when ready |
+| `--verbose` | No | Show detailed debug output |
+
+## Example Usage
+
+```bash
+# Test the city-weather plugin
+/plugin-test:test --plugin "plugins/city-weather" --message "/city-weather:now Tokyo" --expect "Weather Report"
+
+# Test the magic-test plugin with regex matching
+/plugin-test:test --plugin "plugins/magic-test" --message "alakazam" --expect "Plugin loaded successfully" --regex
+
+# Test with verbose output and open browser
+/plugin-test:test --plugin "plugins/city-weather" --message "/city-weather:now London" --expect "Temperature" --verbose --open
+```
+
+## Expected Output
+
+On success:
+```
+=== Plugin Test ===
+
+Configuration:
+  OpenHands URL:  https://app.all-hands.dev
+  Marketplace:    github:OpenHands/extensions (ref: main)
+  Plugin:         plugins/city-weather
+  Message:        /city-weather:now Tokyo
+  Expect:         Weather Report
+
+[INFO] Creating conversation with plugin: plugins/city-weather
+[INFO] Task created: abc123...
+[INFO] Waiting for conversation to start (max 180s)...
+[PASS] Conversation ready: def456...
+[INFO] Fetching conversation events...
+[INFO] Agent responded (1234 chars)
+[PASS] Response matches expected pattern: "Weather Report"
+
+=== TEST PASSED ===
+
+View conversation:
+  https://app.all-hands.dev/conversations/def456...
+```
+
+On failure, the script exits with code 1 and shows the actual response for debugging.
+
+## Notes
+
+- Sandbox startup typically takes 30-90 seconds
+- The script polls the API every 3 seconds by default
+- Maximum wait time is 180 seconds (configurable with `--max-wait`)
+- Use `--verbose` to see full API responses for debugging