-
Notifications
You must be signed in to change notification settings - Fork 512
docs(streaming): add section on token usage tracking #1282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Add documentation about token usage tracking when streaming is enabled in NeMo Guardrails.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds a new section on token usage tracking when streaming is enabled in NeMo Guardrails documentation, covering usage details, version requirements, and methods to access token usage data.
- Introduces a “Token Usage Tracking” section with key features
- Specifies version requirements for LangChain packages
- Provides code examples to retrieve token usage information
Comments suppressed due to low confidence (3)
docs/user-guides/advanced/streaming.md:82
- [nitpick] Add a period at the end of this bullet for consistent punctuation in the list.
- Can be safely passed to all LLM providers since those that don't support it will simply ignore the parameter
docs/user-guides/advanced/streaming.md:80
- [nitpick] Add a period at the end of this bullet for consistent punctuation in the list.
- Provides token usage statistics even when streaming responses
docs/user-guides/advanced/streaming.md:120
- The link to
generation-options.md
should likely be prefixed with../generation-options.md
so it resolves correctly from this directory.
For more information about streaming token usage support across different providers, refer to the [LangChain documentation on token usage tracking](https://python.langchain.com/docs/how_to/chat_token_usage_tracking/#streaming). For detailed information about accessing generation logs and token usage, see the [Generation Options](generation-options.md#detailed-logging-information) and [Detailed Logging](../detailed-logging/README.md) documentation.
Documentation preview |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #1282 +/- ##
===========================================
+ Coverage 69.64% 70.45% +0.80%
===========================================
Files 161 161
Lines 16055 16214 +159
===========================================
+ Hits 11181 11423 +242
+ Misses 4874 4791 -83
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, after applying the Copilot changes.
Co-authored-by: Copilot <[email protected]> Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]> Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]> Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]> Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]> Signed-off-by: Pouyan <[email protected]>
Co-authored-by: Miyoung Choi <[email protected]> Signed-off-by: Pouyan <[email protected]>
Add documentation about token usage tracking when streaming is enabled in NeMo Guardrails.
PR #1264
TODO:
gpt-3.5-turbo-instruct
do not support token tracking in streaming.