Skip to content

feat(cli): add token usage block to --json run report - #821

Open
RONAK-AI647 wants to merge 2 commits into
Nano-Collective:mainfrom
RONAK-AI647:feat/json-report-usage-block
Open

feat(cli): add token usage block to --json run report#821
RONAK-AI647 wants to merge 2 commits into
Nano-Collective:mainfrom
RONAK-AI647:feat/json-report-usage-block

Conversation

@RONAK-AI647

@RONAK-AI647 RONAK-AI647 commented Aug 7, 2026

Copy link
Copy Markdown

Description

While finalizing the NanoBench architecture, I identified a critical missing dependency: downstream evaluation harnesses have no way to programmatically track token consumption or diagnose API quota limits.

Following up on the discussion in Nano-Collective/docs#60, I investigated issues #756 and #796. Both issues highlight a clear community need for token telemetry. After reviewing them, I found neither of those addresses the --json run report. NanoBench's orchestrator runs Nanocoder in --plain --json mode and reads structured stdout — that's a completely different code path from the interactive CLI footer or the VS Code ACP layer.

So #796, as written, would not actually put a usage block into the payload NanoBench needs. It's adjacent (same underlying ResponseUsage concept, same provider-reported token data), but it doesn't close the gap mentioned in NanoBench: Nano-Collective/docs#60 (comment)

What this PR does

This PR extends the headless stdout payload to expose total token consumption metrics when Nanocoder is run with the --json flag.

Key Changes:

  • Multi-Turn Accumulation: Tracks and sums input, output, and total tokens across the entire conversation loop in runPlainConversation.
  • Schema Extension: Injects a structured usage block into the final --json output.
  • Graceful Degradation: Safely omits the usage property for local/unsupported models that don't report token metrics, preventing parser breaks.
  • Test Coverage: Adds unit tests in conversation.spec.ts and shell.spec.ts to validate state tracking and payload generation.

It captures the usage data from the underlying AI SDK provider and appends a usage block to the final JSON report. The shape looks like this:

{
  "kind": "run_completed",
  "exitCode": 0,
  "toolCalls": [...],
  "filesChanged": [...],
  "temperature": 0,
  "usage": {
    "inputTokens": 4520,
    "outputTokens": 850,
    "totalTokens": 5370
  }
}

Type of Change

  • New feature

Sum inputTokens/outputTokens/totalTokens from each turn's ApiUsage
response and surface as an optional 'usage' field on
PlainConversationOutcome. When no turn reports usage (provider doesn't
support it), the field is omitted rather than reported as zero, so
downstream consumers can distinguish 'no data' from 'zero tokens'.
Include the accumulated usage object from the conversation outcome
in the JSON report emitted to stdout, when present. Enables external
tooling consuming --json output (e.g. NanoBench's failure taxonomy)
to detect context/token exhaustion, which was previously unobservable
outside interactive mode.
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

No changeset found

This PR does not add a changeset, so it will not appear in the changelog or trigger a release.

If the change is user-facing, add one:

pnpm changeset

Pick a bump (patch / minor / major) and write the changelog entry in our usual voice ("Added X... Thanks to @you. Closes #123."), then commit the generated .changeset/*.md file.

If this PR is docs-only or a chore that needs no release note, you can ignore this - or run pnpm changeset --empty to record that intentionally.

@RONAK-AI647

Copy link
Copy Markdown
Author

@will-lamerton , review it please !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant