Skip to content

docs(streaming): add section on token usage tracking #1282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 29, 2025

Conversation

Pouyanpi
Copy link
Collaborator

@Pouyanpi Pouyanpi commented Jul 10, 2025

Add documentation about token usage tracking when streaming is enabled in NeMo Guardrails.

PR #1264

TODO:

  • text completion models like gpt-3.5-turbo-instruct do not support token tracking in streaming.

Add documentation about token usage tracking when streaming is enabled in
NeMo Guardrails.
@Pouyanpi Pouyanpi added this to the v0.15.0 milestone Jul 10, 2025
@Pouyanpi Pouyanpi added the documentation Improvements or additions to documentation label Jul 10, 2025
@Pouyanpi Pouyanpi requested a review from Copilot July 10, 2025 11:38
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new section on token usage tracking when streaming is enabled in NeMo Guardrails documentation, covering usage details, version requirements, and methods to access token usage data.

  • Introduces a “Token Usage Tracking” section with key features
  • Specifies version requirements for LangChain packages
  • Provides code examples to retrieve token usage information
Comments suppressed due to low confidence (3)

docs/user-guides/advanced/streaming.md:82

  • [nitpick] Add a period at the end of this bullet for consistent punctuation in the list.
- Can be safely passed to all LLM providers since those that don't support it will simply ignore the parameter

docs/user-guides/advanced/streaming.md:80

  • [nitpick] Add a period at the end of this bullet for consistent punctuation in the list.
- Provides token usage statistics even when streaming responses

docs/user-guides/advanced/streaming.md:120

  • The link to generation-options.md should likely be prefixed with ../generation-options.md so it resolves correctly from this directory.
For more information about streaming token usage support across different providers, refer to the [LangChain documentation on token usage tracking](https://python.langchain.com/docs/how_to/chat_token_usage_tracking/#streaming). For detailed information about accessing generation logs and token usage, see the [Generation Options](generation-options.md#detailed-logging-information) and [Detailed Logging](../detailed-logging/README.md) documentation.

Copy link

Documentation preview

https://nvidia.github.io/NeMo-Guardrails/review/pr-1282

@Pouyanpi Pouyanpi requested review from trebedea and miyoungc July 10, 2025 11:39
@codecov-commenter
Copy link

codecov-commenter commented Jul 10, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.45%. Comparing base (0d3ddfc) to head (90acdfa).
Report is 8 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1282      +/-   ##
===========================================
+ Coverage    69.64%   70.45%   +0.80%     
===========================================
  Files          161      161              
  Lines        16055    16214     +159     
===========================================
+ Hits         11181    11423     +242     
+ Misses        4874     4791      -83     
Flag Coverage Δ
python 70.45% <ø> (+0.80%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 13 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Collaborator

@trebedea trebedea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, after applying the Copilot changes.

@Pouyanpi Pouyanpi removed the request for review from miyoungc July 10, 2025 15:19
Pouyanpi and others added 6 commits July 25, 2025 15:08
Co-authored-by: Miyoung Choi <[email protected]>
Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]>
Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]>
Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]>
Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]>
Signed-off-by: Pouyan <[email protected]>
@Pouyanpi Pouyanpi marked this pull request as ready for review July 25, 2025 13:12
@Pouyanpi Pouyanpi merged commit 7f0f46e into develop Jul 29, 2025
19 checks passed
@Pouyanpi Pouyanpi deleted the docs/token-usage-streaming branch July 29, 2025 08:40
christinaexyou pushed a commit to christinaexyou/NeMo-Guardrails that referenced this pull request Jul 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants