[Question] How to integrate automated testing and bug fixing into Custom Workflows?

[ganchuanman]

Hi OpenSpec team,

Thanks for providing such an amazing tool! I have a few questions regarding Custom Workflows in OpenSpec and would appreciate your guidance.

📝 Context We are currently using OpenSpec to build custom workflows. Since our target code runs on mobile devices, we are looking to establish an automated development-test-feedback loop using a multi-agent approach. Specifically, our plan is to:

Integrate mobile operation agents to execute test cases directly on the device.

Utilize visual understanding (Vision-Language Models) for UI validation to detect visual or functional bugs.

Feed the error context (logs and visual feedback) back to a coding agent (e.g., Claude Code) to autonomously generate and apply bug fixes.

❓ Questions

Automated Testing Integration: Are there recommended best practices or native support within OpenSpec to trigger external mobile testing agents (e.g., after a specific build/deploy node)?

Multi-Agent Bug Fixing Loop: How can we best orchestrate this feedback loop (Testing Agent -> Visual Validation -> Code Agent) within the platform? Is it possible to feed test failure messages and visual context back to an AI node or a specific script to generate a patch and automatically apply it?

💻 Environment (Optional, delete if not needed)

OpenSpec Version: 1.1.1

Target Platform: product run on Mobile (Android/iOS)

OS: develop on macos

This is a hypothetical workflow I've envisioned, but I'm unsure if it's feasible at present.

Looking forward to your reply! If this functionality is not currently supported, we would be very happy to propose it as a Feature Request. Thanks in advance for your time and help!

[1code-async]

Task completed.

The comment has been posted on issue #693. Here's a summary of what the comment covers:

1. Clarifies OpenSpec's role — It's a specification-driven development framework, not a CI/CD pipeline, so it doesn't natively trigger external processes like mobile test agents.

2. What's possible today:

   • Creating a custom schema (e.g., mobile-dev-test) with artifacts for test strategy and visual validation criteria
   • Using multi-tool orchestration with 20+ supported AI coding assistants sharing the same spec context
   • Leveraging the built-in /opsx:verify command as a complementary check

3. What's not currently supported — External process triggering, automated feedback ingestion, and fully autonomous agent-to-agent loops

4. Links to 5 related community issues discussing multi-agent communication, agent hierarchy, and custom agent support

5. Suggests a practical architecture — Use OpenSpec for the spec/planning layer, an external orchestrator (CI/CD, LangGraph, etc.) for the automation loop, and bridge them via the filesystem since OpenSpec's state is entirely file-based

---

View full conversation

Powered by 1Code

[1code-async]

Hi @ganchuanman, thanks for the detailed write-up! This is an interesting use case. Let me provide some context from the codebase that should help clarify what's currently possible and where the boundaries are.

Understanding OpenSpec's Role

OpenSpec is a specification-driven development framework — it orchestrates the planning and specification workflow (proposal → specs → design → tasks → implementation), not the runtime build/test/deploy pipeline itself. It works with AI coding assistants (20+ supported tools) to guide them through structured artifact creation and implementation.

This means OpenSpec doesn't have native hooks for triggering external processes like mobile test agents after a build step. The workflow is artifact-driven, not CI/CD pipeline-driven.

What You Can Do Today

1. Custom Schema for Your Mobile Dev-Test Loop

You can create a custom schema that models your workflow's planning artifacts. For example, you could fork the default spec-driven schema and add artifacts for test strategy and feedback analysis:

openspec schema fork spec-driven mobile-dev-test

Then edit openspec/schemas/mobile-dev-test/schema.yaml to add custom artifacts:

name: mobile-dev-test
version: 1
description: Mobile development with test feedback loop

artifacts:
  - id: proposal
    generates: proposal.md
    requires: []

  - id: specs
    generates: "specs/**/*.md"
    requires: [proposal]

  - id: test-strategy
    generates: test-strategy.md
    description: Mobile test cases and visual validation criteria
    template: test-strategy.md
    instruction: |
      Define test cases for mobile execution, including:
      - UI validation checkpoints (for VLM-based verification)
      - Functional test scenarios
      - Expected visual states per screen
    requires: [specs]

  - id: tasks
    generates: tasks.md
    requires: [specs, test-strategy]

apply:
  requires: [tasks]
  tracks: tasks.md

See docs/customization.md for the full custom schema guide, and schemas/spec-driven/schema.yaml for the default schema structure.

2. Multi-Tool Orchestration

OpenSpec supports 20+ AI coding tools via its adapter architecture (src/core/command-generation/adapters/). You can configure multiple tools simultaneously:

openspec init --tools claude,cursor

Each tool gets the same 10 skill files, so your coding agent and any other AI assistants share the same specification context. This means you could have one agent doing implementation (/opsx:apply) and manually coordinate with another agent for testing — they'd both reference the same specs and tasks.

3. Verification Step

The built-in /opsx:verify command (docs/workflows.md) validates implementation against your specs across completeness, correctness, and coherence. While it doesn't execute tests on a device, it could serve as a complementary check alongside your mobile testing results.

What's Not Currently Supported

Your proposed architecture involves several capabilities that are outside OpenSpec's current scope:

1. Triggering external processes (mobile test agents, build/deploy) — OpenSpec has no event hooks or webhook system for triggering external tools after artifact/workflow transitions
2. Automated feedback ingestion — There's no mechanism to programmatically feed test results (logs, screenshots) back into the workflow as input for the next artifact
3. Closed-loop automation (Test → Detect Bug → Fix → Retest) — The OPSX workflow is designed for human-in-the-loop interaction with AI assistants, not fully autonomous agent-to-agent pipelines

Related Community Discussion

There's significant community interest in adjacent capabilities. These existing issues may be worth following:

• #24798 — Inter-session communication for multi-Claude workflows
• #14109 — MCP-Based Multi-Agent Communication
• #24537 — Agent Hierarchy Dashboard for multi-agent workflows
• #16424 — Expose Agent Context in Hook Event Payloads for Multi-Agent Observability
• #18212 — Support custom agents defined in .claude/agents/

Suggested Approach

Given the current architecture, a practical approach would be:

1. Use OpenSpec for the spec/planning layer — Define your mobile app's behavior specs, test criteria, and implementation tasks using a custom schema
2. Use an external orchestrator for the automation loop — Tools like CI/CD pipelines, custom scripts, or agent frameworks (e.g., LangGraph, CrewAI) can handle the Test Agent → VLM Validation → Code Agent loop
3. Bridge them via the filesystem — OpenSpec's state is entirely filesystem-based (artifact existence = completion), so an external orchestrator could read specs/tasks from openspec/changes/ and feed context to your coding agent

If the automated feedback loop is critical to your workflow, filing this as a Feature Request would be valuable — it would help the maintainers gauge demand for event hooks or external process integration in the OPSX workflow.

[ganchuanman]

Hi @ganchuanman, thanks for the detailed write-up! This is an interesting use case. Let me provide some context from the codebase that should help clarify what's currently possible and where the boundaries are.

Understanding OpenSpec's Role

OpenSpec is a specification-driven development framework — it orchestrates the planning and specification workflow (proposal → specs → design → tasks → implementation), not the runtime build/test/deploy pipeline itself. It works with AI coding assistants (20+ supported tools) to guide them through structured artifact creation and implementation.

This means OpenSpec doesn't have native hooks for triggering external processes like mobile test agents after a build step. The workflow is artifact-driven, not CI/CD pipeline-driven.

What You Can Do Today

1. Custom Schema for Your Mobile Dev-Test Loop

You can create a custom schema that models your workflow's planning artifacts. For example, you could fork the default spec-driven schema and add artifacts for test strategy and feedback analysis:

openspec schema fork spec-driven mobile-dev-test
Then edit openspec/schemas/mobile-dev-test/schema.yaml to add custom artifacts:

name: mobile-dev-test
version: 1
description: Mobile development with test feedback loop

artifacts:

• id: proposal
  generates: proposal.md
  requires: []

• id: specs
  generates: "specs/**/*.md"
  requires: [proposal]

• id: test-strategy
  generates: test-strategy.md
  description: Mobile test cases and visual validation criteria
  template: test-strategy.md
  instruction: |
  Define test cases for mobile execution, including:

  • UI validation checkpoints (for VLM-based verification)
  • Functional test scenarios
  • Expected visual states per screen
    requires: [specs]

• id: tasks
  generates: tasks.md
  requires: [specs, test-strategy]

apply:
requires: [tasks]
tracks: tasks.md
See docs/customization.md for the full custom schema guide, and schemas/spec-driven/schema.yaml for the default schema structure.

2. Multi-Tool Orchestration

OpenSpec supports 20+ AI coding tools via its adapter architecture (src/core/command-generation/adapters/). You can configure multiple tools simultaneously:

openspec init --tools claude,cursor
Each tool gets the same 10 skill files, so your coding agent and any other AI assistants share the same specification context. This means you could have one agent doing implementation (/opsx:apply) and manually coordinate with another agent for testing — they'd both reference the same specs and tasks.

3. Verification Step

The built-in /opsx:verify command (docs/workflows.md) validates implementation against your specs across completeness, correctness, and coherence. While it doesn't execute tests on a device, it could serve as a complementary check alongside your mobile testing results.

What's Not Currently Supported

Your proposed architecture involves several capabilities that are outside OpenSpec's current scope:

1. Triggering external processes (mobile test agents, build/deploy) — OpenSpec has no event hooks or webhook system for triggering external tools after artifact/workflow transitions
2. Automated feedback ingestion — There's no mechanism to programmatically feed test results (logs, screenshots) back into the workflow as input for the next artifact
3. Closed-loop automation (Test → Detect Bug → Fix → Retest) — The OPSX workflow is designed for human-in-the-loop interaction with AI assistants, not fully autonomous agent-to-agent pipelines

Related Community Discussion

There's significant community interest in adjacent capabilities. These existing issues may be worth following:

• #24798 — Inter-session communication for multi-Claude workflows
• #14109 — MCP-Based Multi-Agent Communication
• #24537 — Agent Hierarchy Dashboard for multi-agent workflows
• #16424 — Expose Agent Context in Hook Event Payloads for Multi-Agent Observability
• #18212 — Support custom agents defined in .claude/agents/

Suggested Approach

Given the current architecture, a practical approach would be:

1. Use OpenSpec for the spec/planning layer — Define your mobile app's behavior specs, test criteria, and implementation tasks using a custom schema
2. Use an external orchestrator for the automation loop — Tools like CI/CD pipelines, custom scripts, or agent frameworks (e.g., LangGraph, CrewAI) can handle the Test Agent → VLM Validation → Code Agent loop
3. Bridge them via the filesystem — OpenSpec's state is entirely filesystem-based (artifact existence = completion), so an external orchestrator could read specs/tasks from openspec/changes/ and feed context to your coding agent

If the automated feedback loop is critical to your workflow, filing this as a Feature Request would be valuable — it would help the maintainers gauge demand for event hooks or external process integration in the OPSX workflow.

Got it, that makes perfect sense. Thank you for the clarification!

Just to confirm my understanding of the recommended workflow, the lifecycle should look something like this:

Implementation Phase: Let OpenSpec handle the specification and coding phases up through the opsx:apply step.

External Testing & Fixing Loop: Once the code implementation is applied, we step outside of OpenSpec to trigger our custom multi-agent loop (automated mobile testing -> visual UI validation -> code agent fixing).

Completion & Archiving: After our external testing and fixing loop is fully resolved and passes all cases, we then return to OpenSpec to execute the archive step and wrap up the process.

Is this the correct way to think about the integration and handoff?