fix(v1): refuse a served run whose taskset loads per-case Task subclasses

[willccbb]

Fixes the silent half of #2570.

A served run (the default) rebuilds each task inside the env-server worker as the taskset's declared Task type from its wire data. A taskset whose load() builds some cases as a Task subclass loses that subclass's rewards, toolsets and hooks on the served path and scores them silently wrong, while --no-serve scores them as authored.

This change makes the runner refuse before dispatch when any loaded task's type is not the taskset's task_type(), naming the classes and the two ways out: dispatch on a TaskData field inside the declared class, or run in-process with --no-serve (which keeps the class; remote tunnel placement still sizes off the base class, see #2570). Nothing is written or spawned before the check. Whether per-case subclasses should become a supported authoring pattern (carrying the class over the wire) is left to #2570.

• verifiers/v1/cli/eval/runner.py: the refusal, after task selection.
• tests/v1/fixtures/subclass_task_v1.py: a two-case taskset whose second case is a subclass with its own reward and toolset.
• tests/v1/test_e2e.py: a deterministic test (no model, no e2e mark) asserting the ValueError and that no run directory is created.

Validation, model-free: the new test passes; uv run pytest tests/ -m "not e2e" 81 passed; ruff check, ruff format and the pre-commit hooks pass; the in-process loader still yields both classes for the fixture. Reproduction of the underlying bug against EnvServer._build_task is in #2570.

🤖 Generated with Claude Code

---

Note

Low Risk
Adds a pre-dispatch validation on the served eval path only; in-process runs are unchanged and the failure mode is an explicit error instead of silent mis-scoring.

Overview
Served eval runs now fail fast if the taskset’s load() returns tasks whose runtime type is not the taskset’s declared Task class. On the default served path, workers rebuild tasks from wire data using only that declared type, so per-case subclasses would lose extra rewards, toolsets, and hooks and score incorrectly without an obvious error.

The check runs in run_eval after task selection and before any output directory or worker pool is created. The raised ValueError names the unexpected class(es) and points authors to branch on TaskData inside the declared task type or use --no-serve for in-process runs that preserve the subclass instances.

A subclass-task-v1 fixture (base task + SpecialTask with its own reward and toolset) and a model-free test assert the refusal message and that no run directory is written.

^{Reviewed by Cursor Bugbot for commit a1514f6. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Refuse served runs when taskset loads per-case Task subclasses in run_eval

• Adds validation in run_eval (runner.py) that checks whether any selected task's exact type differs from the taskset's declared Task type, raising ValueError before output setup or dispatch for served runs.
• In-process runs skip this check and retain their existing path; the error message recommends using a TaskData field or --no-serve.
• Adds a subclass-task-v1 fixture (subclass_task_v1.py) and an end-to-end test (test_e2e.py) that confirms the ValueError mentions SpecialTask and --no-serve and that no served output directory is created.
• Risk: served evaluations that previously relied on per-case task subclasses will now fail with ValueError instead of proceeding; verify tasksets in verifiers/v1 for SubclassTaskset-style loaders.

^{Macroscope summarized a1514f6.}

[willccbb]

On the risk note in the summary: no bundled taskset trips the refusal. Every module under environments/ and verifiers/v1/tasksets/ was imported at a1514f66 and its Task classes checked for subclass chains; 60 of 65 modules import in isolation and none defines a Task that subclasses another Task in the same module. The five that do not import alone are two harness/verify scripts (compact/program.py, gsm8k/verify.py) and the nemo_gym taskset and weather example (package-relative imports), and by inspection they define one Task class each (NeMoGymTask). The only taskset that builds a subclass per case is the new test fixture.