feat(cli): add token usage block to --json run report - #821
Open
RONAK-AI647 wants to merge 2 commits into
Open
Conversation
Sum inputTokens/outputTokens/totalTokens from each turn's ApiUsage response and surface as an optional 'usage' field on PlainConversationOutcome. When no turn reports usage (provider doesn't support it), the field is omitted rather than reported as zero, so downstream consumers can distinguish 'no data' from 'zero tokens'.
Include the accumulated usage object from the conversation outcome in the JSON report emitted to stdout, when present. Enables external tooling consuming --json output (e.g. NanoBench's failure taxonomy) to detect context/token exhaustion, which was previously unobservable outside interactive mode.
RONAK-AI647
requested review from
Avtrkrb,
akramcodez and
will-lamerton
as code owners
August 7, 2026 15:46
Contributor
No changeset foundThis PR does not add a changeset, so it will not appear in the changelog or trigger a release. If the change is user-facing, add one: pnpm changesetPick a bump (patch / minor / major) and write the changelog entry in our usual voice ("Added X... Thanks to @you. Closes #123."), then commit the generated If this PR is docs-only or a chore that needs no release note, you can ignore this - or run |
1 task
Author
|
@will-lamerton , review it please ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
While finalizing the NanoBench architecture, I identified a critical missing dependency: downstream evaluation harnesses have no way to programmatically track token consumption or diagnose API quota limits.
Following up on the discussion in
Nano-Collective/docs#60, I investigated issues #756 and #796. Both issues highlight a clear community need for token telemetry. After reviewing them, I found neither of those addresses the--jsonrun report. NanoBench's orchestrator runs Nanocoder in--plain --jsonmode and reads structured stdout — that's a completely different code path from the interactive CLI footer or the VS Code ACP layer.So #796, as written, would not actually put a usage block into the payload NanoBench needs. It's adjacent (same underlying
ResponseUsageconcept, same provider-reported token data), but it doesn't close the gap mentioned in NanoBench: Nano-Collective/docs#60 (comment)What this PR does
This PR extends the headless
stdoutpayload to expose total token consumption metrics when Nanocoder is run with the--jsonflag.Key Changes:
runPlainConversation.usageblock into the final--jsonoutput.usageproperty for local/unsupported models that don't report token metrics, preventing parser breaks.conversation.spec.tsandshell.spec.tsto validate state tracking and payload generation.It captures the usage data from the underlying AI SDK provider and appends a
usageblock to the final JSON report. The shape looks like this:{ "kind": "run_completed", "exitCode": 0, "toolCalls": [...], "filesChanged": [...], "temperature": 0, "usage": { "inputTokens": 4520, "outputTokens": 850, "totalTokens": 5370 } }Type of Change