feat: parallel-improve workflow for concurrent experiment execution

[lambdabaa]

Summary

• Adds a new parallel-improve workflow that runs N hypotheses concurrently in isolated git worktrees, then selects the best result via tournament-style selection
• Introduces two new workflow primitives: SubgraphForkNode (fan-out subgraphs into per-experiment worktrees) and SelectionNode (compare branches and pick the winner by best_score)
• Extends the executor with _execute_subgraph_fork() which spawns independent WorkflowExecutor instances per branch for full state isolation, and _execute_selection() which merges the winner and finalizes losers as superseded
• Adds ParallelConfig model, create_experiment_worktree(), "superseded" verdict type, checkpoint parallel fields, config parsing, skill export rendering, and graph validation for the new node types

This is a stepping stone toward integrating parallel experimentation into the core improve loop per the design in #987.

Ref: #987

Test plan

• 31 new tests in tests/test_parallel_improve.py covering models, primitives, workflow validation, executor dry-run, helpers, and checkpoint
• All 493 related existing tests pass with no regressions
• ruff check and mypy clean on all modified files
• Skill export generates valid SKILL.md with parallel-specific sections
• Graph validation passes for parallel_improve_workflow()

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 97.36070% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.19%. Comparing base (fcebd56) to head (5dc5643).

Files with missing lines	Patch %	Lines
factory/worktree.py	82.14%	5 Missing ⚠️
factory/workflow/executor.py	97.79%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #992      +/-   ##
==========================================
+ Coverage   88.99%   89.19%   +0.20%     
==========================================
  Files         124      124              
  Lines       14544    14872     +328     
==========================================
+ Hits        12943    13265     +322     
- Misses       1601     1607       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[osilkin98]

@ceo-review

[github-actions]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: QA: CLEAN — 3393 tests pass, 0 failures, ruff clean, mypy clean. Code review found 3 important issues (shared .factory/ symlink race in concurrent branches, fragile fork output scan, missing _parse_parallel tests) but no critical blockers. Adversarial testing verified 11/11 feature surfaces. All deliverables implemented per spec.

QA Analysis

Adversarial QA Report — PR #992: parallel-improve workflow

Date: 2026-07-08
Project type: CLI (Python)
Detected mode: Library / CLI hybrid

Smoke Test

No project-specific smoke test defined in factory.md. Proceeded directly to feature tests after confirming uv sync and factory --help succeed.

Feature Tests

Test 1: Import test

Status: VERIFIED

$ python -c 'from factory.workflow.primitives import SubgraphForkNode, SelectionNode; from factory.models import ParallelConfig; from factory.workflow.definitions import parallel_improve_workflow; from factory.workflow.executor import _parse_hypotheses, _collect_subgraph_nodes; from factory.worktree import create_experiment_worktree; print("ALL IMPORTS OK")'

ALL IMPORTS OK

Test 2: Model validation — ParallelConfig defaults and max

Status: VERIFIED

$ python -c 'from factory.models import ParallelConfig; c=ParallelConfig(); print(f"defaults: {c.parallel_hypotheses}, {c.selection_strategy}"); c2=ParallelConfig(parallel_hypotheses=8); print(f"max: {c2.parallel_hypotheses}")'

defaults: 1, best_score
max: 8

Test 3: Model rejection — ParallelConfig rejects invalid values

Status: VERIFIED

$ python -c '
from factory.models import ParallelConfig
from pydantic import ValidationError
try:
    ParallelConfig(parallel_hypotheses=9)
    print("BUG: should reject 9")
except ValidationError:
    print("OK: rejects 9")
try:
    ParallelConfig(parallel_hypotheses=0)
    print("BUG: should reject 0")
except ValidationError:
    print("OK: rejects 0")'

OK: rejects 9
OK: rejects 0

Test 4: Superseded verdict type

Status: VERIFIED

$ python -c 'from factory.models import ExperimentRecord; from datetime import datetime, timezone; r=ExperimentRecord(id=1, timestamp=datetime.now(tz=timezone.utc), hypothesis="test", change_summary="s", issue_number=None, pr_number=None, score_before=0.5, score_after=0.6, delta=0.1, verdict="superseded", cost_usd=None, notes=""); print(f"verdict: {r.verdict}")'

verdict: superseded

Test 5: Workflow graph validation

Status: VERIFIED

$ python -c 'from factory.workflow.definitions import parallel_improve_workflow; wf=parallel_improve_workflow(); issues=wf.validate_graph(); print(f"issues: {issues}"); assert issues==[], f"Graph invalid: {issues}"; print("GRAPH VALID")'

issues: []
GRAPH VALID

Test 6: Workflow registration

Status: VERIFIED

$ python -c 'from factory.workflow.definitions import register_all; wfs=register_all(); assert "parallel-improve" in wfs; print(f"registered: {list(wfs.keys())}")'

registered: ['build', 'design', 'discover', 'review', 'improve', 'parallel-improve', 'qa', 'deep-qa', 'legacybench', 'featurebench', 'programbench', 'swebench', 'terminalbench', 'research', 'meta', 'refine', 'create', 'skill-refine', 'doc-generate', 'doc-update', 'spec-generate', 'spec-update']

Test 7: Hypothesis parser

Status: VERIFIED

$ python -c '
import tempfile; from pathlib import Path
from factory.workflow.executor import _parse_hypotheses
p = Path(tempfile.mktemp())
p.write_text("## Hypothesis 1\nAdd caching\n\n## Hypothesis 2\nRefactor auth\n\n## Hypothesis 3\nAdd logging\n")
result = _parse_hypotheses(p)
print(f"parsed {len(result)} hypotheses: {result}")
assert len(result) == 3, f"Expected 3, got {len(result)}"
p.unlink()
print("PARSER OK")'

parsed 3 hypotheses: ['## Hypothesis 1\nAdd caching', '## Hypothesis 2\nRefactor auth', '## Hypothesis 3\nAdd logging']
PARSER OK

Test 8: Subgraph collector

Status: VERIFIED

$ python -c '
from factory.workflow.executor import _collect_subgraph_nodes
from factory.workflow.primitives import FnNode, Edge, Workflow
wf = Workflow(name="test", nodes={"a": FnNode(id="a", writes={"x"}), "b": FnNode(id="b", reads={"x"}, writes={"y"}), "c": FnNode(id="c", reads={"y"}, writes={"z"})}, edges=[Edge(source="a",target="b"), Edge(source="b",target="c")], start_node="a")
result = _collect_subgraph_nodes(wf, "a", "c")
print(f"subgraph nodes: {result}")
assert result == {"a", "b", "c"}, f"Expected a,b,c got {result}"
print("SUBGRAPH COLLECTOR OK")'

subgraph nodes: {'c', 'a', 'b'}
SUBGRAPH COLLECTOR OK

Test 9: Dry-run executor (pytest)

Status: VERIFIED

$ pytest tests/test_parallel_improve.py::TestSubgraphForkDryRun -v

tests/test_parallel_improve.py::TestSubgraphForkDryRun::test_dry_run_subgraph_fork PASSED [ 50%]
tests/test_parallel_improve.py::TestSubgraphForkDryRun::test_dry_run_selection PASSED [100%]

============================== 2 passed in 0.09s ===============================

Test 10: CLI mode registration

Status: VERIFIED

$ python -c 'from factory.cli._helpers import CEO_MODES, RUN_MODES; assert "parallel-improve" in CEO_MODES; assert "parallel-improve" in RUN_MODES; print(f"CEO_MODES includes parallel-improve: True"); print(f"RUN_MODES includes parallel-improve: True")'

CEO_MODES includes parallel-improve: True
RUN_MODES includes parallel-improve: True

Test 11: Skill export

Status: VERIFIED

$ python -c '
from factory.workflow.definitions import parallel_improve_workflow
from factory.workflow.skill_export import workflow_to_skill_md
wf = parallel_improve_workflow()
md = workflow_to_skill_md(wf)
assert "Parallel" in md or "parallel" in md
assert "SubgraphForkNode" in md
assert "SelectionNode" in md
print(f"Skill MD length: {len(md)} chars")
print("SKILL EXPORT OK")'

Skill MD length: 6534 chars
SKILL EXPORT OK

Acceptance Criteria Verification

#	Criterion	Status
1	All new symbols import correctly	VERIFIED
2	ParallelConfig defaults and max bounds correct	VERIFIED
3	ParallelConfig rejects out-of-range values	VERIFIED
4	"superseded" verdict type works in ExperimentRecord	VERIFIED
5	parallel-improve workflow graph validates cleanly	VERIFIED
6	parallel-improve registered in workflow registry	VERIFIED
7	_parse_hypotheses correctly splits multi-hypothesis text	VERIFIED
8	_collect_subgraph_nodes correctly traverses graph edges	VERIFIED
9	SubgraphForkNode and SelectionNode dry-run tests pass	VERIFIED
10	parallel-improve in CEO_MODES and RUN_MODES	VERIFIED
11	Skill export generates valid markdown with node types	VERIFIED

Adversarial Verdict: PASS

All 11 tests produced concrete evidence of correct behavior. No tests were skipped. No blockers encountered. The parallel-improve workflow feature is functional across all tested surfaces: models, graph definition, executor utilities, CLI registration, and skill export.

---

Posted by Factory CEO

[osilkin98]

@ceo-review

[github-actions]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: QA: CLEAN — 3432 tests pass, 0 failures, 0 lint/type issues. 63 new tests green. All adversarial exercises pass. Code review found 2 non-blocking improvement opportunities (fragile fork output scan, unused field). No Sacred Rule violations.

QA Analysis

Adversarial QA -- PR #992

Test Results

#	Test	Result	Evidence
1	Import & Instantiation	PASS	SubgraphForkNode(id='fork1', subgraph_entry='entry_node', subgraph_exit='exit_node') and SelectionNode(id='sel1', strategy='best_score') instantiate correctly. Note: PR description mentions subgraph_ids and parallel_config fields which do not exist on SubgraphForkNode; actual fields are subgraph_entry, subgraph_exit, parallelism, worktree_isolated.
2	Pydantic Boundaries	PASS	ParallelConfig rejects parallel_hypotheses=0 (below min 1), parallel_hypotheses=9 (above max 8), extra fields (bogus='x'), and invalid strategy ('random'). All 4/4 boundary checks pass.
3	Workflow Graph Validity	PASS	parallel_improve_workflow() validates clean with 19 nodes and 22 edges. No dangling edges, missing nodes, or unreachable nodes.
4	Verdict Type Expansion	PASS	ExperimentRecord accepts verdict='superseded' alongside existing keep, revert, error. Invalid verdict 'invalid_verdict' is correctly rejected.
5	Worktree Helper	PASS	create_experiment_worktree exists with signature (project_path: Path, exp_id: int, base_commit: str) -> tuple[Path, str].
6	Store finalize_parallel_losers	N/A	Method finalize_parallel_losers does not exist on ExperimentStore. The PR description claims it exists, but the implementation handles loser finalization inline in executor.py:_execute_selection (lines 693-721) using the existing store.finalize() with verdict='superseded'. This is architecturally fine but the PR description is inaccurate.
7	Executor Methods Exist	PASS	WorkflowExecutor._execute_subgraph_fork and WorkflowExecutor._execute_selection both exist.
8	Skill Export for Parallel Workflow	PASS	workflow_to_skill_md() produces 6534 chars of markdown with parallel-specific content.
9	Checkpoint Parallel Fields	PASS	CheckpointState accepts active_experiment_ids, parallel_branch_status fields. format_checkpoint() renders "Parallel exps: 1, 2, 3" and "Branch status: branch-1=running, branch-2=done" correctly.
10	PR Test Suite (63 tests)	PASS	All 63 tests in tests/test_parallel_improve.py pass in 0.37s.

Additional Adversarial Checks

#	Check	Result	Evidence
A1	SubgraphForkNode parallelism=0	WARNING	Accepted without validation error. The executor guards against this at runtime (if branch_count < 1: branch_count = 1 at executor.py:505-506), but the Pydantic model lacks ge=1 constraint.
A2	SubgraphForkNode parallelism=-1	WARNING	Also accepted. Same runtime guard applies.
A3	SelectionNode invalid strategy	PASS	Correctly rejects strategy='random' and strategy=''.
A4	Workflow Registry	PASS	parallel-improve is registered among 22 workflows and discoverable via WorkflowRegistry.
A5	_parse_parallel edge cases	PASS	Handles empty list, empty string, float, out-of-range (100), unknown keys -- all return None gracefully.
A6	Core unit tests (190 tests)	PASS	test_models.py, test_parallel_improve.py, test_store.py, test_checkpoint.py -- all 190 tests pass in 0.93s.

Issues Found

1. Missing input validation on SubgraphForkNode.parallelism (factory/workflow/primitives.py:179): The field parallelism: int = 3 has no ge=1 constraint. Values of 0 and -1 are accepted by the model. While the executor has a runtime guard (branch_count = max(branch_count, 1)), best practice is to validate at the model level, consistent with how ParallelConfig.parallel_hypotheses uses ge=1, le=8. Severity: low (runtime guard prevents crash, but inconsistent with other model patterns).

2. PR description inaccuracy: The PR summary mentions finalize_parallel_losers() as a new store method, but this method does not exist. Loser finalization is handled inline in _execute_selection. Not a code bug, but misleading documentation.

Verdict

• Tests passed: 9/9 (Test 6 N/A due to method not existing as described)
• Issues found: 2 (1 low-severity code issue, 1 documentation inaccuracy)
• Overall: PASS (with minor issues)
• Summary: The parallel-improve workflow is functionally sound -- all 63 PR tests and 190 core unit tests pass, models validate correctly at boundaries, the workflow graph validates clean, and the registry integration works. The only code issue is missing ge=1 validation on SubgraphForkNode.parallelism, mitigated by a runtime guard in the executor.

---

Posted by Factory CEO

[osilkin98]

@lambdabaa it looks like this needs to be rebased, once that's done we can merge this