Use ollama's structured outputs feature

[daniel-j-h]

I'm looking into pydantic-ai with small and locally running ollama models as backbones.

I'm noticing that sometimes even for simple models it's possible to run into unexpected ValidationErrors.

Here's what I mean: With a pydantic model as simple as

class Answer(BaseModel):
    value: str = ""

I can see pytandic-ai sometimes retrying and failing in validation.

Having experience with llama.cpp's grammars this was unexpected to me. I was under the assumption that pydantic-ai would transform the pydantic model into a grammar or json schema to hard-restrict the llm's output accordingly. Then validation could never fail by design since the llm's output is restricted to the specific grammar.

Instead when I debug the request pydantic-ai sends to the locally running ollama with

nc -l -p 11434

I can see pydantic-ai turning the pydantic model into a tool use invocation.

With ollama v0.5.0 structured output via json schema is now supported:

https://github.com/ollama/ollama/releases/tag/v0.5.0

I was wondering if that would solve the issue of small locally running models sometimes running into validation errors, since we hard-restrict the output to the shape of our pydantic model.

Any thoughts on this, or ideas why validation can fail with tool usage as implemented right now? Any pointers in terms of for which model providers validation might fail and for what reason? Thanks!

[daniel-j-h]

I have looked into this a bit more and here is what is happening

1. The user specifies a return type for a pydantic-ai agent in form of a pydantic model
2. The pydantic model's jsonschema gets passed to the llm in form of a tool and its arguments
3. Because tool usage is optional the ollama models on the smaller side often never use the tool
4. pydantic-ai then fails validation and retries, and more often than not the 2nd try also doesn't use the tool

The underlying issue here is that tool usage is optional. This makes the pydantic-ai approach of validating and parsing types unreliable and non-deterministic.

There exists a tool_choice=required parameter in the OpenAI API but it's not supported in ollama as of today. From the ollama tool blog post:

> Future improvements [..] Tool choice: force a model to use a tool

I see the following ways forward

1. Change the documentation and make it clear that validation and parsing may or may not happen and that especially with smaller models it appears like it's often failing due to lack of tool usage. This undermines the main selling point of pydantic-ai
2. Look into and implement the structured output approach which, I believe, should make validation and parsing deterministic

Note: I have only looked at the ollama model as implemented in pydantic-ai. I do not know if other models are affected by this, too, or if their output is deterministically constrained.

[samuelcolvin]

Look into and implement the structured output approach which, I believe, should make validation and parsing deterministic

Is this part of the OpenAI API, or would we need a dedicated model for it?

Happy to consumer consider both, just trying to understand.

[samuelcolvin]

Looks like we should switch from the OpenAI compatibility API to using the Ollama python library.

[renkehohl]

As I was also facing several validation errors when working with Ollama, I have made an own implementation of the OllamaModel utilizing Ollama's new Structured Output feature. If you are interested in it, I can contribute it here.

[gt732]

@renkehohl do you mind sharing? I'm running into the same issue running local models it sometimes works but its random. I'm mostly testing with qwen models.

[renkehohl]

@gt732 here is the gist: https://gist.github.com/renkehohl/407cdcdd3bfc8d0baee3783782b31e3d

Usage is same as PydanticAI's OllamaModel:

from pydantic_ai import Agent
from typing import List
from ollama_model import OllamaModel

agent = Agent(
     model=OllamaModel(model_name="llama3.1"),
     result_type=List[str]
)

Please keep in mind, that I've not implemented stream responses yet.

[christopherfowers]

I don't see Message,ModelAnyResponse, ModelStructuredResponse, ToolCall, or ModelTextResponse on pydantic_ai.messages at all. Nor do I see any of these types noted in the messages documentation. What am I missing to be able to replicate your gist @renkehohl?

[renkehohl]

@christopherfowers the message format has been changed in the commit from December 15th. I was using PydanticAI at version 0.0.12, so my implementation needs to get updated for later versions.

[gabrielgrant]

@samuelcolvin to answer your question from earlier about whether this is something OpenAI also supports: yes, they have structured outputs (as of end of Aug 2024 iirc?): https://platform.openai.com/docs/guides/structured-outputs

This avoids the whole roundabout method of having to ask for a function call when you know that you just want a response to always conform to a specific schema. Would recommend looking into switching to this for OpenAI calls too (this is their official recommendation)

Their docs on it are a bit lacking. That page shows using the client.beta.chat.completions.parse() call, which accepts a pydantic BaseModel directly in the response_format arg, and returns a populated model instance (iirc from looking at the code a while ago it does a bit more work behind the scenes).

Not documented on that page, but you can also just use openai.chat.completions.create() and pass a JSON schema into response_format directly:

response_format: {
        // See /docs/guides/structured-outputs
        type: "json_schema",
        json_schema: {
            name: "email_schema",
            schema: {
                type: "object",
                properties: {
                    email: {
                        description: "The email address that appears in the input",
                        type: "string"
                    }
                },
                additionalProperties: false
            }
        }
    }