Skip to content

bug: Guardrail server failing when streaming is enabled #1325

@swayam-singhal

Description

@swayam-singhal

Did you check docs and existing issues?

  • I have read all the NeMo-Guardrails docs
  • I have updated the package to the latest version before submitting this issue
  • (optional) I have used the develop branch
  • I have searched the existing issues of NeMo-Guardrails

Python version (python --version)

Python 3.12.5

Operating system/version

MacOS 15.3.2

NeMo-Guardrails version (if you must use a specific version and not the latest

No response

Describe the bug

models:

  • type: main
    engine: openai
    model: gpt-4.1-2025-04-14
    parameters:
    temperature: 0.2
    stream: true

for this config trying to work with the nemoguardrails server with stream: true it is failing with the issue below:

File "/Users/Swayam/Documents/Projects/pandemos/client/.venv/lib/python3.12/site-packages/nemoguardrails/library/self_check/input_check/actions.py", line 72, in self_check_input
response = await llm_call(llm, prompt, stop=stop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/Swayam/Documents/Projects/pandemos/client/.venv/lib/python3.12/site-packages/nemoguardrails/actions/llm/utils.py", line 122, in llm_call
raise LLMCallException(e)
nemoguardrails.actions.llm.utils.LLMCallException: LLM Call Exception: 'AsyncStream' object has no attribute 'model_dump'

Steps To Reproduce

  1. Create an OpenAI config with stream true
  2. Start the nemoguardrails server
  3. Try to ask any query or use a custom openai client with local base url to ask AsyncOpenAI streaming queries.

Expected Behavior

  1. Streaming response should be returned neatly.

Actual Behavior

  1. Internal server error occurs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstatus: needs triageNew issues that have not yet been reviewed or categorized.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions