feat(evals): add cancel_event to Eval and EvalAsync

[Abhijeet Prasad (AbhiPrasad)]

Cancel an evaluation

EvalAsync and Eval accept an optional cancel_event: threading.Event. Keep a reference to the event and call set() when the evaluation should stop. You can also set timeout in seconds.

EvalAsync

import asyncio
import threading

from braintrust import EvalAsync

cancel_event = threading.Event()

async def task(prompt):
    return await generate_answer(prompt)

evaluation = asyncio.create_task(
    EvalAsync(
        "my-project",
        data=[{"input": "What is 2+2?"}],
        task=task,
        scores=[],
        cancel_event=cancel_event,
        timeout=30,
    )
)

# Later, when the user chooses Stop:
cancel_event.set()

try:
    await evaluation
except asyncio.CancelledError:
    pass  # Stopped by cancel_event.set()
except asyncio.TimeoutError:
    pass  # Stopped by timeout

Tasks do not need to inspect the event to receive async cancellation. Cancelling the asyncio.Task running EvalAsync also stops the evaluation and sets its event.

Eval

import threading

from braintrust import Eval

cancel_event = threading.Event()

def on_stop():
    cancel_event.set()

# Have another thread call on_stop() while Eval blocks.
Eval(
    "my-project",
    data=cases,
    task=sync_task,
    scores=[],
    cancel_event=cancel_event,
    timeout=30,
)

A synchronous task already running in a thread cannot be forcibly stopped. It can check hooks.cancel_event.is_set() to stop sooner; otherwise the evaluation waits for it to finish. This can make cancellation take longer than timeout.

For a long-running synchronous loop, checking the hook is optional:

def sync_task(items, hooks):
    results = []
    for item in items:
        if hooks.cancel_event.is_set():
            break
        results.append(process(item))
    return results

flowchart TD
    A["EvalAsync or Eval"] --> B{"Stop requested?"}
    B -- "cancel_event.set()" --> C["Stop starting queued trials"]
    B -- "timeout" --> C
    C --> D["Running async tasks receive cancellation"]
    C --> E["Running sync tasks can check hooks.cancel_event"]
    D --> F["Wait for running work to finish cleanup"]
    E --> F
    F --> G["Raise asyncio.CancelledError or asyncio.TimeoutError"]

Loading

An event that is already set prevents trials from starting. A timeout also sets the supplied event.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-05T22:25:14.482238Z	3c3f2be	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "Codex (@codex) review" or "Codex (@codex) security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c3f2be41e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".