Skip to content

feat(v1): progress hook for live traces - #2573

Draft
faresobeid wants to merge 1 commit into
mainfrom
feat/trace-progress-hook
Draft

feat(v1): progress hook for live traces#2573
faresobeid wants to merge 1 commit into
mainfrom
feat/trace-progress-hook

Conversation

@faresobeid

@faresobeid faresobeid commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

What

Agent.run(...), Agent.interaction(...) and Rollout(...) take on_progress: Callable[[Trace], None] | None. The interception server invokes it from record_call — the one seam every recorded exchange passes through, after the turn it committed — so the hook sees the live trace each time it gains a ModelCall (and the nodes of that turn). A hook that raises is logged at WARNING and swallowed; a consumer bug never fails a rollout. None (the default) changes nothing. Callers debounce.

Why

on_trace hands out the trace object at mint, but nothing signals when it grows: a live consumer has to poll. In the data-flywheel pipeline (a task factory on verifiers.v1 + the rho harness, ~90 concurrent rollouts), a 2-hour rollout is invisible until it closes — its model calls and turns can only be inspected after the fact, so stuck or runaway rollouts are noticed late.

Validation

  • tests/v1/test_trace.py::test_on_progress_observes_live_trace — offline: a loopback chat-completions upstream, an in-process harness that makes 3 model calls through the real interception server, agent.run(task, on_progress=...). Asserts the hook fires once per call with the live (is) trace, len(trace.calls) grows 1, 2, 3, and that a hook raising every time still leaves trace.ok with no errors.
  • uv run pytest tests/v1 -m "not e2e": 87 passed. uv run ruff check, uv run pre-commit run --all-files: clean.

Note

Add on_progress live-trace callback to Agent.run, Rollout, and RolloutSession

  • Adds an optional on_progress parameter threaded through Agent.run, _run_once, interaction, _EpisodeAgent, Rollout.__init__, and RolloutSession so callers can observe the live Trace after each recorded model call.
  • InterceptionServer call-recording handler now invokes RolloutSession.progress after trace content is committed, delivering the current trace to the callback.
  • RolloutSession.progress returns immediately when no callback is set and logs ordinary Exception subclasses without re-raising, so a faulty consumer cannot fail the rollout.
  • Adds an integration test in test_trace.py using a local loopback provider and ProbeHarness to assert three calls, a shared live trace, callback counts of one through three, and rollout success despite a RuntimeError raised from the consumer.
  • Behavioral Change: callbacks that raise non-BaseException errors are swallowed and logged; callers relying on exceptions propagating out of Agent.run will no longer see them.

Macroscope summarized e19af4d.

Agent.run / Agent.interaction (and Rollout) take on_progress; the interception
server notifies it from record_call, once per recorded exchange, after the turn
it committed. A raising hook is logged and swallowed, never the rollout's failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant