Getting error 'The endpoint ... failed to stream the response. Retrying without streaming'.

[mnkumar8]

Hi,

Getting this error in new versions of RAG app. Please let me know how to fix this error.

Background:
I have created a RAG app using Databricks Mosaic AI Agent and was able to ask questions and got responses. I wanted to modify the prompts and create new version. But, every time I create a new version, I get the same error message in the new chatbot application (not able to ask questions in new versions).

I create new version means, I deployed the model to create a new API endpoint using the below code:

from databricks import agents
#Register the chain to UC
uc_registered_model_info = mlflow.register_model(model_uri=logged_chain_info.model_uri, name=MODEL_NAME_FQN)

#Deploy to enable the Review APP and create an API endpoint
deployment_info = agents.deploy(model_name=MODEL_NAME_FQN, model_version=uc_registered_model_info.version, scale_to_zero=True)

instructions_to_reviewer = f"""### Instructions for Testing ...
"""

#Add the user-facing instructions to the Review App
agents.set_review_instructions(MODEL_NAME_FQN, instructions_to_reviewer)
wait_for_model_serving_endpoint_to_be_ready(deployment_info.endpoint_name)

Thanks,
Naveen

[QuentinAmbard]

hey,
if you try your model without deploying it is it working ?

[mnkumar8]

Thanks for the response. Yes, it is working without deploying. Getting error only in chatbot application, but not while trying using code.