Custom Tests specification documentation

docs/custom-tests-specification/architecture.md

@@ -0,0 +1,133 @@
+# Architecture
+
+#### Understand how QIT runs Custom E2E Tests
+
+### Single Plugin
+
+QIT can run a single plugin in a disposable environment. This is useful for testing a plugin in isolation.
+
+```mermaid
+sequenceDiagram
+    participant QIT
+    participant TestEnvironment
+
+    activate QIT
+
+    %% --- Environment Setup ---
+    Note over QIT,TestEnvironment: Environment Setup Phase
+    QIT->>TestEnvironment: 1. Create Environment
+    activate TestEnvironment
+    Note right of TestEnvironment: Creates a disposable, dockerized environment.<br>Installs WordPress, PHP<br>and the plugin under test.
+
+    QIT->>TestEnvironment: 2. Run Shared Setup (if defined)
+    Note right of TestEnvironment: Executes `lifecycle.sharedSetup` script<br>if defined in plugin-a's qit-e2e.json.
+    %% DB Snapshot/Import steps are omitted for single-plugin tests
+
+    %% --- Test Execution for plugin-a ---
+    Note over QIT,TestEnvironment: Executing tests for plugin-a
+    QIT->>TestEnvironment: 3. Run Plugin A Setup (`lifecycle.setup` in plugin-a's qit-e2e.json)
+    QIT->>TestEnvironment: 4. Run Plugin A E2E test (`test_command` in plugin-a's qit-e2e.json)
+    activate TestEnvironment # LightSkyBlue
+    Note right of TestEnvironment: (Runs plugin-a E2E Tests)
+    deactivate TestEnvironment # LightSkyBlue
+    QIT->>TestEnvironment: 5. Run Plugin A Teardown (`lifecycle.teardown` in plugin-a's qit-e2e.json)
+    QIT->>TestEnvironment: 6. Collect Results for plugin-a (From plugin-a's `test_results` paths)
+    TestEnvironment-->>QIT: (Raw results & logs for plugin-a)
+
+    %% --- Teardown Phase ---
+    Note over QIT,TestEnvironment: Teardown Phase
+    QIT->>TestEnvironment: 7. Run Shared Teardown (if defined)
+    Note right of TestEnvironment: Executes `lifecycle.sharedTeardown` script<br>if defined in plugin-a's qit-e2e.json.
+
+    %% --- Result Processing & Reporting ---
+    Note over QIT,TestEnvironment: Result Processing & Reporting Phase
+    QIT->>QIT: 8. Process Results
+    Note right of QIT: Processes results from plugin-a,<br>captures logs, and prepares the final report.
+
+    QIT->>TestEnvironment: 9. Destroy Environment
+    deactivate TestEnvironment
+
+    Note over QIT: Show Final Status & Report URL
+    deactivate QIT
+```
+
+### Multiple Plugins
+
+QIT can run multiple plugins in a single environment. This is useful for testing compatibility between plugins or themes.
+
+When running in a multi-plugin environment, QIT will create a single environment and run the tests for each plugin in sequence, restoring a snapshot of the database state between each plugin's tests. This ensures that each plugin's tests run in a clean environment, without any interference from the other plugins.
+
+```mermaid
+sequenceDiagram
+    participant QIT
+    participant TestEnvironment
+
+    activate QIT
+
+    %% --- Environment Setup ---
+    Note over QIT,TestEnvironment: Environment Setup Phase
+    QIT->>TestEnvironment: 1. Create Environment
+    activate TestEnvironment
+    Note right of TestEnvironment: Creates a disposable, dockerized environment.<br>Installs WordPress, PHP<br>AND specified plugins (plugin-a, plugin-b, plugin-c)<br>based on CLI options / qit-env.json.
+
+    QIT->>TestEnvironment: 2. Run Shared Setup (if defined)
+    Note right of TestEnvironment: Executes combined `lifecycle.sharedSetup` scripts<br>from relevant plugins (e.g., plugin-a, plugin-b, plugin-c if defined in qit-e2e.json).
+
+    QIT->>TestEnvironment: 3. Snapshot Initial Database State
+    Note right of TestEnvironment: Saves the DB state after shared setup.
+
+    %% --- Test Execution for plugin-a ---
+    Note over QIT,TestEnvironment: Executing tests for plugin-a
+    QIT->>TestEnvironment: Import Initial Database State
+    Note right of TestEnvironment: Resets DB to the state saved after Shared Setup.
+    QIT->>TestEnvironment: Run Plugin-Specific Setup (plugin-a's `lifecycle.setup`)
+    QIT->>TestEnvironment: Run Plugin's Test Command (plugin-a's `test.command`)
+    activate TestEnvironment # LightSkyBlue
+    Note right of TestEnvironment: (plugin-a Test Execution...)
+    deactivate TestEnvironment # LightSkyBlue
+    QIT->>TestEnvironment: Run Plugin-Specific Teardown (plugin-a's `lifecycle.teardown`)
+    QIT->>TestEnvironment: Collect Results for plugin-a (From plugin-a's `test.results` paths)
+    TestEnvironment-->>QIT: (Raw results & logs for plugin-a)
+
+    %% --- Test Execution for plugin-b ---
+    Note over QIT,TestEnvironment: Executing tests for plugin-b
+    QIT->>TestEnvironment: Import Initial Database State
+    Note right of TestEnvironment: Resets DB to the state saved after Shared Setup.
+    QIT->>TestEnvironment: Run Plugin-Specific Setup (plugin-b's `lifecycle.setup`)
+    QIT->>TestEnvironment: Run Plugin's Test Command (plugin-b's `test.command`)
+    activate TestEnvironment # LightSkyBlue
+    Note right of TestEnvironment: (plugin-b Test Execution...)
+    deactivate TestEnvironment # LightSkyBlue
+    QIT->>TestEnvironment: Run Plugin-Specific Teardown (plugin-b's `lifecycle.teardown`)
+    QIT->>TestEnvironment: Collect Results for plugin-b (From plugin-b's `test.results` paths)
+    TestEnvironment-->>QIT: (Raw results & logs for plugin-b)
+
+    %% --- Test Execution for plugin-c ---
+    Note over QIT,TestEnvironment: Executing tests for plugin-c
+    QIT->>TestEnvironment: Import Initial Database State
+    Note right of TestEnvironment: Resets DB to the state saved after Shared Setup.
+    QIT->>TestEnvironment: Run Plugin-Specific Setup (plugin-c's `lifecycle.setup`)
+    QIT->>TestEnvironment: Run Plugin's Test Command (plugin-c's `test.command`)
+    activate TestEnvironment # LightSkyBlue
+    Note right of TestEnvironment: (plugin-c Test Execution...)
+    deactivate TestEnvironment # LightSkyBlue
+    QIT->>TestEnvironment: Run Plugin-Specific Teardown (plugin-c's `lifecycle.teardown`)
+    QIT->>TestEnvironment: Collect Results for plugin-c (From plugin-c's `test.results` paths)
+    TestEnvironment-->>QIT: (Raw results & logs for plugin-c)
+
+    %% --- Final Steps ---
+    Note over QIT,TestEnvironment: Teardown Phase
+    QIT->>TestEnvironment: 4. Run Shared Teardown (if defined)
+    Note right of TestEnvironment: Executes combined `lifecycle.sharedTeardown` scripts<br>from relevant plugins (e.g., plugin-a, plugin-b, plugin-c if defined in qit-e2e.json).
+
+    %% --- Result Processing & Reporting ---
+    Note over QIT,TestEnvironment: Result Processing & Reporting Phase
+    QIT->>QIT: 5. Process & Merge All Results
+    Note right of QIT: Combines results from plugin-a, plugin-b, plugin-c,<br>captures logs, and prepares the final report.
+
+    QIT->>TestEnvironment: 6. Destroy Environment
+    deactivate TestEnvironment
+
+    Note over QIT: Show Final Status & Report URL
+    deactivate QIT
+```

docs/custom-tests-specification/introduction.md

@@ -0,0 +1,23 @@
+# Introduction
+
+**QIT Custom Tests** is a developer-friendly testing solution that lets WordPress plugin and theme authors ship their own end-to-end test suites. It eliminates environment setup hassles so you can focus on writing meaningful tests.
+
+## Key Features
+
+- **Write tests, not infrastructure code**: QIT handles Docker setup, WordPress configuration, and environment teardown
+- **Playwright by default, flexible by design**: Built for Playwright but supports any test framework that outputs CTRF results
+- **Multi-plugin orchestration**: Test multiple plugins in one environment with shared setup and isolated test cycles
+- **Standardized Test Definition**: Each plugin includes a `qit-e2e.json` manifest file, defining how its tests integrate with and are executed by the QIT CLI. This file acts as the key configuration, ensuring compatibility with QIT Custom Tests.
+- **Disposable Docker environments:** QIT builds a fresh WordPress instance for every run, then cleans up automatically.
+- **Standardised reporting:** CTRF results are merged and surfaced consistently in local runs, CI pipelines, and marketplace checks.
+
+## Compatibility Testing
+
+QIT's orchestration system helps plugin and theme developers identify cross-plugin compatibility issues before they affect users. During the shared setup phase, each participating plugin performs its own initialization steps (e.g., a payment gateway plugin connects to its sandbox, another plugin dismisses the onboarding wizard it adds, etc). QIT then captures a database snapshot of this fully prepared environment before systematically running each plugin's test suite in isolation. This approach reveals how plugins interact in real-world conditions while maintaining test isolation and separation of concerns.
+
+- Validate your plugin against popular extensions your users likely have installed
+- Test compatibility with different versions of companion plugins
+- Ensure consistent functionality when multiple plugins modify the same WordPress areas
+
+> **Industry adoption**  
+> Leading plugin marketplaces—such as [WooCommerce.com](https://woocommerce.com)—use QIT to run **quality checks** on extensions submitted to their platforms. Making your E2E tests **compliant with the QIT Custom Test Specification** signals that your plugin is serious about quality and reliability, helps you pass automated reviews, and makes your product stand out in a crowded marketplace.

docs/custom-tests-specification/quick-start.md

@@ -0,0 +1,74 @@
+# Quick Start
+
+### Get started with QIT Custom Tests
+
+Custom End-to-End (E2E) Tests allow you to define and run compatibility tests for WordPress plugins/themes directly in Dockerized environments. Choose your scenario:
+
+#### [1. New Playwright Tests](#new-playwright)
+Create a new test suite from scratch using Playwright.
+
+#### [2. Existing Playwright Tests](#existing-playwright)
+Integrate QIT into an existing project with minimal setup.
+
+#### [3. Other test frameworks](#other-frameworks)
+Use any framework that outputs results in the [CTRF format](https://ctrf.io).
+
+## 1. New Playwright Tests
+
+1. Run the following command to create a new test suite:
+
+   ```qitbash
+   qit scaffold:e2e ./tests/e2e
+   ```
+2. This creates a new directory `tests/e2e` with the following structure:
+
+   ```bash
+   tests/e2e/
+   ├── qit-e2e.json
+   ├── playwright.config.js
+   ├── package.json
+   ├── bootstrap/
+   │   ├── setup.sh
+   │   └── mu-plugin.php
+   └── tests/
+       └── example.spec.ts
+   ```
+
+3. Run the tests:
+
+   ```qitbash
+   qit run:e2e your-plugin-slug ./tests/e2e
+   ```
+
+4. Develop the tests with Playwright. For example, you can spin up a persistent environment for development:
+
+   ```qitbash
+   qit run:e2e your-plugin-slug ./tests/e2e --persistent
+   ```
+
+   Then run the test several times while developing:
+
+   ```qitbash
+   cd tests/e2e
+   QIT_SITE_URL=http://localhost:8080 npx playwright test
+   ```
+
+   You can also run in Playwright Codegen or Headed mode.
+
+   ```qitbash
+   npx playwright codegen http://localhost:8080
+   ```
+
+   Check out the [Playwright documentation](https://playwright.dev/docs/codegen-intro) for more options.
+
+   You can always reset the environment to a clean state by running:
+
+   ```qitbash
+   qit reset
+   ```
+
+   When you are happy with the results, you can publish your tests to QIT with:
+
+   ```qitbash
+   qit publish:e2e your-plugin-slug ./tests/e2e
+   ```