Skip to content

fix the introduction of an additional space and split words in subtokens #1198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: develop
Choose a base branch
from
Draft

fix the introduction of an additional space and split words in subtokens #1198

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2acf809
fix the introduction of an additional space and split words in subtokens
May 15, 2025
959742c
yield words as they are
Jun 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions nemoguardrails/rails/llm/llmrails.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1297,7 +1297,7 @@ def _prepare_params(
)

async for chunk_list, chunk_str_rep in buffer_strategy(streaming_handler):
chunk_str = " ".join(chunk_list)
chunk_str = "".join(chunk_list)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I assume all tokenizers are adding spaces and any other whitespace and special chars in some of the chunks, so this should work for any LLM provider.


# Check if chunk_str_rep is a JSON string
# we yield a json error payload in generate_async when
Expand All @@ -1311,8 +1311,8 @@ def _prepare_params(
if stream_first:
words = chunk_str_rep.split()
if words:
yield words[0]
for word in words[1:]:
# yield words[0]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds an extra space before the first word, no?
I think the previous logic was good here.

for word in words:
yield f" {word}"

for flow_id in output_rails_flows_id:
Expand Down