feat(rollout): add built-in AgentArk environment support

[P90-RushB]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Summary

This PR adds built-in AgentArk environment support to ms-swift GRPO.

AgentArk is an Apache-2.0 open-source, extensible general-purpose environment framework for VLM/LLM agents. Coding agents are used to automatically develop and expand its task library. It currently includes more than 200 tasks spanning general 2D and 3D tasks, physics-related tasks, charts and diagrams, multi-view tasks, ARC-AGI-3-like tasks, color-related tasks, and more; the collection is continuously expanding.

The Swift trainer communicates with a standalone AgentArk Environment Server over HTTP, while the server manages the task runtime pool.

This integration was discussed with ms-swift maintainers in the official community group, where the maintainers welcomed an upstream PR.

swift rlhf
  -> AgentArkScheduler / AgentArkEnv
  -> AgentArk HTTP protocol v2
  -> AgentArk Environment Server
  -> task runtime pool

The trainer environment does not install agent-ark and does not import task-runtime code. The HTTP implementation reuses ms-swift's existing aiohttp and requests dependencies, so this PR adds no new package requirement.

Why built-in support

AgentArk was previously integrated through an external Swift plugin. Moving the trainer-side adapter into Swift provides a documented and tested environment implementation while preserving Swift's rollout contracts:

• complete multimodal messages, including inline image_url content;
• exact sampled token IDs and per-token loss masks across multiple turns;
• all_turns and last_round assistant loss scopes;
• deterministic lease acquisition, heartbeat, release, and stale-lease fencing;
• cleanup after success, initialization failure, generation failure, or cancellation;
• both colocate and server-side multi-turn rollout paths.

AgentArk Environment Server and task runtimes remain independently deployed environment components.

Commit structure

feat(rollout): finalize multi-turn trajectories

Adds a general on_trajectory_end(requests, error) lifecycle hook for multi-turn schedulers and invokes it from both colocate and server-side drivers.

• finalization runs after success and failure, including initialization and first-generation failures;
• cleanup failures do not hide an already active rollout exception;
• the default hook is a no-op for backward compatibility;
• GYMScheduler uses the hook to close any environments still owned by the completed batch;
• lifecycle behavior is covered by focused unit tests.

This generic lifecycle boundary is required so environment-backed schedulers can reliably release resources that survive beyond an individual turn.

feat(rollout): add built-in AgentArk integration

Adds the complete trainer-side integration:

• AgentArkHttpClient for protocol-v2 acquire, step, release, retry, idempotency, and structured error handling;
• lease identity and a process-safe heartbeat supervisor;
• AgentArkEnv and AgentArkScheduler registration;
• multimodal message delta handling, action extraction, exact token IDs, and loss-mask propagation;
• stale-lease containment and terminal cleanup;
• ticket generation and a minimal one-step LoRA example;
• Chinese and English documentation;
• fake HTTP server, environment, heartbeat, message, scheduler, and end-to-end driver tests.

The final implementation uses reusable aiohttp.ClientSession and requests.Session clients and does not add httpx.

Experiment results

Automated tests

Tested after rebasing onto modelscope/ms-swift@7d001448a:

python -m unittest \
  tests.rollout.agentark.test_env \
  tests.rollout.agentark.test_heartbeat \
  tests.rollout.agentark.test_http_client \
  tests.rollout.agentark.test_messages \
  tests.rollout.agentark.test_scheduler \
  tests.rollout.test_trajectory_lifecycle \
  tests.rollout.test_exact_token_io \
  tests.rollout.test_server_exact_token_io

Ran 61 tests in 1.572s
OK

All configured pre-commit hooks pass on the 25 files changed by this PR: flake8, isort, YAPF, trailing whitespace, YAML, EOF, requirements, quote, merge-conflict, and mixed-line-ending checks.

The upstream tree currently contains pre-existing CRLF files outside this diff, so running the mixed-line-ending fixer over the entire checkout rewrites unrelated files. No such line-ending-only changes are included here; every non-empty file in this PR is stored as LF in Git.

Real AgentArk smoke test

The rebased built-in integration was tested end to end with:

• model: Qwen3.5-4B;
• hardware: 6 x RTX 3090;
• training: LoRA, AdamW, GRPO/DAPO;
• rollout: colocated vLLM, tensor parallel size 2;
• environment: Snake 8x8, six warmed task runtimes, HTTP protocol v2;
• sequence settings: max_turns=6, max_length=12288, max_completion_length=4096 per round;
• thinking mode enabled.

Result:

global_step/max_steps: 1/1
loss: -0.2237
grad_norm: 0.4919
reward: 0.3333
reward_std: 0.8165
num_turns: 2.333
completions/mean_length: 493.5
checkpoint-1: written
active leases after training: 0
idle task runtimes after training: 6/6

Longer convergence run

The built-in AgentArk path was also validated with a complete 600-step, full-parameter Qwen3.5-9B Snake run on 8 x H800. The run completed successfully and showed a clear upward reward trend. RL sampling is stochastic, so the curve is evidence of end-to-end convergence rather than an exact reproduction target.

The full recipe and training curve are documented here:

• AgentArk Snake training tutorial
• Training curve

Compatibility and CI scope

• Existing schedulers retain a no-op finalization hook unless they override it.
• AgentArk protocol v2 is the recommended path; the client retains the existing protocol-v1 compatibility path.
• Unit tests use in-memory fakes and a local fake HTTP server, so official CI does not require AgentArk Environment Server or task runtimes.
• The external AgentArk plugin remains available in the AgentArk repository for released Swift versions that do not yet contain this integration.