ARBuilder LLM Benchmark

A standalone benchmarking framework that evaluates LLMs on Stylus smart contract development tasks across three layers: tool selection (L1), code compilation and test execution (L2), and end-to-end agentic task completion (L3). It wraps any OpenRouter-accessible model in a minimal agentic coding harness, exposes the full ARBuilder MCP tool surface, and runs structured tasks inside per-task Docker containers for fully reproducible, verifiable outcomes.

---

Stats

Metric	Value
Unit tests	136
Total tasks	60
L1 tasks	46
L2 tasks	6
L3 tasks	8
Models benchmarked	3

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    ARBuilder LLM Benchmark                      │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                     Benchmark Harness                    │   │
│  │  run_benchmark.py ──► AgentHarness ──► OpenRouter LLM   │   │
│  │                            │                             │   │
│  │                    ┌───────┴───────┐                     │   │
│  │                    ▼               ▼                     │   │
│  │             Tool Registry    DockerEnvironment           │   │
│  │           ┌──────┴──────┐         │                     │   │
│  │           ▼             ▼         ▼                     │   │
│  │      ARBuilder       Base      bash / file ops          │   │
│  │      MCP Tools       Tools    (per-task container)      │   │
│  │     (19 tools)                                          │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌────────────┐  ┌────────────────────┐  ┌───────────────────┐  │
│  │     L1     │  │         L2         │  │        L3         │  │
│  │  Tool      │  │  Single-Tool       │  │  Agentic Task     │  │
│  │ Selection  │  │  Execution         │  │  Completion       │  │
│  │            │  │                    │  │                   │  │
│  │ 46 tasks   │  │  6 tasks           │  │  8 tasks          │  │
│  │ single-turn│  │  up to 3 turns     │  │  up to 15 turns   │  │
│  │ tool match │  │  compile + test    │  │  pass^k scoring   │  │
│  └────────────┘  └────────────────────┘  └───────────────────┘  │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                     Evaluation                           │   │
│  │  tool_match.py  ──  execution.py  ──  report.py         │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Three Evaluation Layers

Layer	What Is Evaluated	Mode	Tasks
L1	Tool selection — did the model call the right tool with the right arguments?	Single-turn	46
L2	Code quality — does generated Stylus code compile and pass unit tests?	Up to 3 turns	6
L3	End-to-end task completion — full agentic loop with pass^k consistency	Agentic (up to 15 turns)	8

---

Results Summary

L1 Tool Selection (46 tasks)

Model	Overall	Bridging	Creation	dApp	Debugging	Feature	Multi-tool	No-tool	Orbit	Testing
Qwen 3.6+ (free)	80.4%	100%	37.5%	100%	75%	100%	75%	100%	100%	75%
GLM-5	78.3%	100%	100%	100%	75%	67%	100%	75%	50%	25%
MiniMax M2.7	78.3%	100%	75%	100%	75%	33%	75%	100%	100%	75%

L2 Single-Tool Execution (6 tasks)

Model	Overall	Creation	Debugging	Feature	Testing
Qwen 3.6+ (free)	33.3%	0%	100%	0%	0%
GLM-5	66.7%	50%	100%	0%	100%
MiniMax M2.7	66.7%	50%	100%	0%	100%

Qwen L2 affected by 2 rate-limit failures on free tier.

L3 Agentic Task Completion (8 tasks)

Model	Overall	Creation	Debugging	Feature	Avg Turns
Qwen 3.6+ (free)	37.5%	33%	67%	0%	5.7
GLM-5	50.0%	33%	100%	0%	5.2
MiniMax M2.7	37.5%	0%	100%	0%	5.7

Qwen L3 affected by 2 rate-limit errors (1 debugging, 1 creation — tasks never started). Feature addition (0% all models) is the hardest open problem.

Full analysis and methodology: docs/benchmark-report.md

---

Quick Start

Setup

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env   # add your OPENROUTER_API_KEY

Build the Docker Base Image

docker build -t arbuilder-bench/stylus-base:latest docker/base/

Run Benchmarks

L1 — Tool Selection:

python scripts/run_benchmark.py \
  --layer l1 \
  --model anthropic/claude-sonnet-4 \
  --tasks-dir tasks \
  --output-dir results

L2 — Single-Tool Execution:

python scripts/run_benchmark.py \
  --layer l2 \
  --model anthropic/claude-sonnet-4 \
  --tasks-dir tasks \
  --output-dir results

L3 — Agentic Task Completion:

python scripts/run_benchmark.py \
  --layer l3 \
  --model anthropic/claude-sonnet-4 \
  --tasks-dir tasks \
  --output-dir results

# With pass^k (run each task 3 times for reliability scoring)
python scripts/run_benchmark.py \
  --layer l3 \
  --model anthropic/claude-sonnet-4 \
  --repeat 3

Results are written to results/<layer>_<model>.json. A human-readable summary is printed to stdout.

---

Usage Examples

Run L1 with an explicit API key

python scripts/run_benchmark.py \
  --layer l1 \
  --model openai/gpt-4o \
  --api-key sk-or-... \
  --tasks-dir tasks \
  --output-dir results

Run all three layers in sequence

for layer in l1 l2 l3; do
  python scripts/run_benchmark.py --layer $layer --model qwen/qwen3-235b-a22b:free
done

Run pass^k with 5 repeats (tau-bench style)

python scripts/run_benchmark.py \
  --layer l3 \
  --model anthropic/claude-sonnet-4 \
  --repeat 5

Load and inspect a saved result

import json
from pathlib import Path

data = json.loads(Path("results/l1_anthropic_claude-sonnet-4.json").read_text())
print(f"Overall accuracy: {data['overall_accuracy']:.1%}")
print(f"By category: {data['by_category']}")

Run the test suite

pytest tests/ -v

---

Project Structure

ARBuilder-benchmarks/
├── pyproject.toml
├── harness/
│   ├── agent.py               # Core agent loop (AgentHarness)
│   ├── docker_env.py          # Per-task Docker container manager (DockerEnvironment)
│   ├── providers.py           # LLM provider (OpenRouter, OpenAI-compatible)
│   ├── task_loader.py         # YAML task loader (BenchmarkTask dataclass)
│   ├── trajectory.py          # Conversation/tool-call logging
│   └── tools/
│       ├── registry.py        # Unified tool registry
│       ├── arbuilder_defs.py  # All 19 ARBuilder tool definitions (OpenAI format)
│       ├── base_tools.py      # Low-level Docker tools (bash, file_read, file_write, file_search)
│       └── mcp_client.py      # MCP client (stdio transport, proxies tool calls from ARBuilder server)
├── evaluation/
│   ├── tool_match.py          # L1 tool selection evaluator (L1Result, evaluate_tool_selection)
│   ├── execution.py           # L2 execution evaluator (L2Result, evaluate_execution)
│   ├── task_completion.py     # L3 task completion evaluator (L3Result, compute_pass_k)
│   └── report.py              # Report generation (L1Report, L2Report, L3Report)
├── tasks/
│   ├── l1_tool_selection/     # 46 L1 task YAML files
│   ├── l2_single_tool/        # 6 L2 task YAML files (with workspace/)
│   └── l3_agentic/            # 8 L3 task YAML files (with workspace/ and ground_truth/)
├── docker/                    # Base Stylus dev image (Rust 1.91.0, cargo-stylus, wasm32)
├── scripts/
│   └── run_benchmark.py       # CLI entry point
├── docs/
│   └── benchmark-report.md    # Initial results report (April 2026)
└── tests/                     # pytest test suite

---

Task Format (YAML)

Each task lives in its own directory as task.yaml:

id: "l3-debug-storage-002"
layer: "l3_agentic"
category: "debugging"
difficulty: "easy"
prompt: |
  This Stylus contract has a bug. The deposit function doesn't persist changes.
  Read the code, find the bug, fix it, and verify it compiles.
expected_tools:
  - name: "ask_stylus"
    required_args: ["question"]
also_accept:
  - name: "get_stylus_context"
    required_args: ["query"]
docker_image: "arbuilder-bench/stylus-base:latest"
validation:
  compilation: true
  tests:
    fail_to_pass: []
    pass_to_pass: []

If a workspace/ directory exists alongside task.yaml, its contents are automatically copied into the container before the agent starts.

---

ARBuilder Tool Coverage (19 tools)

Module	Tools
M1 Stylus	get_stylus_context, generate_stylus_code, ask_stylus, generate_tests, get_workflow, validate_stylus_code
M2 SDK / Bridge	generate_bridge_code, generate_messaging_code, ask_bridging
M3 dApp	generate_backend, generate_frontend, generate_indexer, generate_oracle, orchestrate_dapp
M4 Orbit	generate_orbit_config, generate_orbit_deployment, generate_validator_setup, ask_orbit, orchestrate_orbit

---

Design Principles

Principle	Source	Implementation
Docker-per-task isolation	SWE-bench	Each task runs in a fresh container from arbuilder-bench/stylus-base; container destroyed after task
fail-to-pass / pass-to-pass test splits	SWE-bench	Evaluator supports both test sets; current tasks use compilation-only (test data planned)
Tool name exact match + arg fuzzy match	BFCL	evaluation/tool_match.py scores tool selection with argument completeness and quality
pass^k reliability metric	tau-bench	--repeat N runs each L3 task N times; pass^k = fraction of runs where all k pass
Streaming for reliability	—	All provider calls use streaming to eliminate mid-generation timeout failures

---

Design Spec

See docs/superpowers/specs/2026-04-05-arbuilder-llm-benchmark-design.md for the full design specification including task authoring guide, evaluation layer definitions, and delivery timeline.