feat(adapters): WebVoyager task loader via the answer-submit interception path - #356
Open
vaibhavdabas16 wants to merge 5 commits into
Open
feat(adapters): WebVoyager task loader via the answer-submit interception path#356vaibhavdabas16 wants to merge 5 commits into
vaibhavdabas16 wants to merge 5 commits into
Conversation
First step of the incremental plan in TIGER-AI-Lab#72: the shared target type, the registry, and an identity adapter, with no behaviour change for bundled tasks. `ClawBenchTask` is a superset of test-cases/task.schema.json plus the provenance an imported task needs — which source it came from and that source's own id — so a leaderboard can partition rows by origin. Time limits stay in minutes, matching task.json and the container watchdog, because most upstream schemas use seconds and the conversion has to live somewhere explicit. Adapters declare which scoring layers they can honour. A layer a source cannot support is meant to score null rather than 0, so aggregation never confuses "the agent failed" with "never scored on that axis"; to_task_json() therefore refuses to render a task with no interception contract instead of inventing one. Fields with no 1:1 mapping surface as an AdapterWarning naming the field, the fallback, and the pinned upstream revision the mapping was written against — a task is never dropped silently. Validation is hand-rolled rather than Pydantic: the package has no Pydantic dependency and its other registries (harness_registry) validate frozen dataclasses exactly this way.
`clawbench-sources` lists the registered adapters with their upstream, pin, declared scoring layers, and whether a checkout is present; `show` prints one source's status and its module field-mapping table; `cases` lists the tasks a source exposes, with any load warnings. TIGER-AI-Lab#72 sketches this as `clawbench sources`, but `clawbench` is the TUI and every non-interactive command in this repo is its own `clawbench-<verb>` entrypoint, so it follows that convention instead. A source is addressable as `<name>:<path>` to pin it to an explicit clone; otherwise it resolves under CLAWBENCH_SOURCES_DIR, XDG_CACHE_HOME, or ~/.cache. CLAWBENCH_OFFLINE is honoured up front so a later fetching adapter fails loudly instead of cloning.
Adds docs/task-sources.md covering the CLI, the shared task type, the scoring-layer contract, field-mapping warnings, and how to write an adapter, plus the CHANGELOG entry and a docs/cli.md row. Tests cover the registry, source-spec parsing and cache resolution, the schema's validation and task.json rendering, the native loader against both a suite directory and a corpus root, and each CLI subcommand.
WebVoyager scores by a screenshot-and-LLM judge over the trajectory; there is no final write request for Stage 1 to intercept. ClawBench already solved that shape for the claw-eval port: the instruction tells the agent to submit its final answer at the runtime server's /submit form, POST /api/task-submit is the interception target, and the LLM judge scores the submitted answer against the reference. This adapter reuses that path exactly, so a WebVoyager task runs through the standard two-stage pipeline with no runner change. Maps id/ques/web/web_name from data/WebVoyager_data.jsonl and attaches the matching data/reference_answer.json entries — labelled [golden] or [possible] so the judge can weigh them — as judge_context. A missing start URL or reference becomes a warning on the task, never a silent drop; a task with no id or question fails the load with its line number.
eval/adapters/webvoyager.md covers getting the tasks, how a WebVoyager task scores under ClawBench, the field mapping, and what is deliberately not done here (the upstream screenshot-judge passthrough, the reproduction check, pinning). The test fixture is a synthetic sample in the upstream file shape, not upstream content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances #190 (and #72, step 4). Stacked on #345 — the adapter foundation — so the diff shows both until that merges; the WebVoyager-specific commits are
43f5c76anda06083b.What this is
A task loader for MinorJerry/WebVoyager: reads
data/WebVoyager_data.jsonlanddata/reference_answer.jsonfrom a checkout and producesClawBenchTasks.clawbench-sources cases webvoyager:/path/to/WebVoyager clawbench-sources show webvoyager # status + field-mapping tableThe design decision worth reviewing
WebVoyager scores by a screenshot-and-LLM judge over the trajectory. There is no final write request for Stage 1 to intercept, which is the one thing ClawBench's scoring keys on. The easy answer was to mark these tasks
llm_judge_onlywith noeval_schemaand leave them unrunnable until a runner change.But ClawBench already solved exactly this shape for the claw-eval port, and it's in
main: the instruction tells the agent to submit its final answer at the runtime server'shttp://127.0.0.1:7878/submitform, that formPOSTs to/api/task-submit, the task'seval_schematargets that endpoint, and the LLM judge scores the submitted answer againstjudge_context. The adapter reuses that path verbatim — same endpoint, same footer wording — so a WebVoyager task runs through the ordinary Stage-1 + Stage-2 pipeline with zero runner changes and produces the standard trace bundle. A test pins the contract against a real bundled claw-eval task so the two can't drift.Mapping
task_id/source_idid(lower-cased / verbatim)instructionques+ submit footerurlwebcategoryweb_nametime_limitjudge_context.reference_solutionreference_answer.jsonentries, labelled[golden]/[possible]so the judge can weigh themMissing start URL or missing reference →
AdapterWarningon the task, never a silent drop. Noidor noques→AdapterErrorwith the line number. The reference-file parser skips anything that doesn't match the upstream shape rather than guessing at it.Testing
tests/test_webvoyager_adapter.py— 15 cases against a synthetic fixture in the upstream file shape (deliberately not upstream content): field mapping, the interception contract matching claw-eval, labelled references, both missing-reference cases (no entry vs. no file), missing URL, direct.jsonlpath, four malformed-checkout failures with locations,to_task_json()validated againsttask.schema.json, and the CLI. Full suite:317 passed, 10 skipped;ruffandpyrightclean.Not done here, and said so in the doc
eval/adapters/webvoyager.mdis the walkthrough #190 asks for. It's explicit that the upstream screenshot-judge passthrough — both scores side by side inrun-meta.json, which is the comparison #190 is really after — is not attempted: that needs their judge prompt wired as a second scorer plus a runner change to record a second verdict. This loader is the prerequisite for it. The ±3pp reproduction check follows from that, andpinned_shawaits on the shared_pins.yaml(#72 step 5).I have not run this against a real WebVoyager checkout — the field names (
web_name/id/ques/web) match #72's mapping table and my understanding of the upstream file, but a maintainer with a clone should runclawbench-sources cases webvoyager:<clone>once before merge.