Let model settings be passed to model classes (#2136)

Co-authored-by: J S <[email protected]>
Co-authored-by: svilupp <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>

Author: 3 people

docs/agents.md

@@ -466,26 +466,43 @@ PydanticAI offers a [`settings.ModelSettings`][pydantic_ai.settings.ModelSetting
 This structure allows you to configure common parameters that influence the model's behavior, such as `temperature`, `max_tokens`,
 `timeout`, and more.
 
-There are two ways to apply these settings:
+There are three ways to apply these settings, with a clear precedence order:
 
-1. Passing to `run{_sync,_stream}` functions via the `model_settings` argument. This allows for fine-tuning on a per-request basis.
-2. Setting during [`Agent`][pydantic_ai.agent.Agent] initialization via the `model_settings` argument. These settings will be applied by default to all subsequent run calls using said agent. However, `model_settings` provided during a specific run call will override the agent's default settings.
+1. **Model-level defaults** - Set when creating a model instance via the `settings` parameter. These serve as the base defaults for that model.
+2. **Agent-level defaults** - Set during [`Agent`][pydantic_ai.agent.Agent] initialization via the `model_settings` argument. These are merged with model defaults, with agent settings taking precedence.
+3. **Run-time overrides** - Passed to `run{_sync,_stream}` functions via the `model_settings` argument. These have the highest priority and are merged with the combined agent and model defaults.
 
 For example, if you'd like to set the `temperature` setting to `0.0` to ensure less random behavior,
 you can do the following:
 
 ```py
 from pydantic_ai import Agent
+from pydantic_ai.models.openai import OpenAIModel
+from pydantic_ai.settings import ModelSettings
 
-agent = Agent('openai:gpt-4o')
+# 1. Model-level defaults
+model = OpenAIModel(
+    'gpt-4o',
+    settings=ModelSettings(temperature=0.8, max_tokens=500)  # Base defaults
+)
+
+# 2. Agent-level defaults (overrides model defaults by merging)
+agent = Agent(model, model_settings=ModelSettings(temperature=0.5))
 
+# 3. Run-time overrides (highest priority)
 result_sync = agent.run_sync(
-    'What is the capital of Italy?', model_settings={'temperature': 0.0}
+    'What is the capital of Italy?',
+    model_settings=ModelSettings(temperature=0.0)  # Final temperature: 0.0
 )
 print(result_sync.output)
 #> Rome
 ```
 
+The final request uses `temperature=0.0` (run-time), `max_tokens=500` (from model), demonstrating how settings merge with run-time taking precedence.
+
+!!! note "Model Settings Support"
+    Model-level settings are supported by all concrete model implementations (OpenAI, Anthropic, Google, etc.). Wrapper models like `FallbackModel`, `WrapperModel`, and `InstrumentedModel` don't have their own settings - they use the settings of their underlying models.
+
 ### Model specific settings
 
 If you wish to further customize model behavior, you can use a subclass of [`ModelSettings`][pydantic_ai.settings.ModelSettings], like [`GeminiModelSettings`][pydantic_ai.models.gemini.GeminiModelSettings], associated with your model of choice.

docs/models/index.md

@@ -124,6 +124,39 @@ The `ModelResponse` message above indicates in the `model_name` field that the o
 !!! note
     Each model's options should be configured individually. For example, `base_url`, `api_key`, and custom clients should be set on each model itself, not on the `FallbackModel`.
 
+### Per-Model Settings
+
+You can configure different [`ModelSettings`][pydantic_ai.settings.ModelSettings] for each model in a fallback chain by passing the `settings` parameter when creating each model. This is particularly useful when different providers have different optimal configurations:
+
+```python {title="fallback_model_per_settings.py"}
+from pydantic_ai import Agent
+from pydantic_ai.models.anthropic import AnthropicModel
+from pydantic_ai.models.fallback import FallbackModel
+from pydantic_ai.models.openai import OpenAIModel
+from pydantic_ai.settings import ModelSettings
+
+# Configure each model with provider-specific optimal settings
+openai_model = OpenAIModel(
+    'gpt-4o',
+    settings=ModelSettings(temperature=0.7, max_tokens=1000)  # Higher creativity for OpenAI
+)
+anthropic_model = AnthropicModel(
+    'claude-3-5-sonnet-latest',
+    settings=ModelSettings(temperature=0.2, max_tokens=1000)  # Lower temperature for consistency
+)
+
+fallback_model = FallbackModel(openai_model, anthropic_model)
+agent = Agent(fallback_model)
+
+result = agent.run_sync('Write a creative story about space exploration')
+print(result.output)
+"""
+In the year 2157, Captain Maya Chen piloted her spacecraft through the vast expanse of the Andromeda Galaxy. As she discovered a planet with crystalline mountains that sang in harmony with the cosmic winds, she realized that space exploration was not just about finding new worlds, but about finding new ways to understand the universe and our place within it.
+"""
+```
+
+In this example, if the OpenAI model fails, the agent will automatically fall back to the Anthropic model with its own configured settings. The `FallbackModel` itself doesn't have settings - it uses the individual settings of whichever model successfully handles the request.
+
 In this next example, we demonstrate the exception-handling capabilities of `FallbackModel`.
 If all models fail, a [`FallbackExceptionGroup`][pydantic_ai.exceptions.FallbackExceptionGroup] is raised, which
 contains all the exceptions encountered during the `run` execution.

pydantic_ai_slim/pydantic_ai/agent.py

@@ -674,12 +674,14 @@ async def main():
         # typecast reasonable, even though it is possible to violate it with otherwise-type-checked code.
         output_validators = cast(list[_output.OutputValidator[AgentDepsT, RunOutputDataT]], self._output_validators)
 
-        model_settings = merge_model_settings(self.model_settings, model_settings)
+        # Merge model settings in order of precedence: run > agent > model
+        merged_settings = merge_model_settings(model_used.settings, self.model_settings)
+        model_settings = merge_model_settings(merged_settings, model_settings)
         usage_limits = usage_limits or _usage.UsageLimits()
 
         if isinstance(model_used, InstrumentedModel):
-            instrumentation_settings = model_used.settings
-            tracer = model_used.settings.tracer
+            instrumentation_settings = model_used.instrumentation_settings
+            tracer = model_used.instrumentation_settings.tracer
         else:
             instrumentation_settings = None
             tracer = NoOpTracer()

pydantic_ai_slim/pydantic_ai/models/__init__.py

@@ -321,6 +321,27 @@ class Model(ABC):
     """Abstract class for a model."""
 
     _profile: ModelProfileSpec | None = None
+    _settings: ModelSettings | None = None
+
+    def __init__(
+        self,
+        *,
+        settings: ModelSettings | None = None,
+        profile: ModelProfileSpec | None = None,
+    ) -> None:
+        """Initialize the model with optional settings and profile.
+
+        Args:
+            settings: Model-specific settings that will be used as defaults for this model.
+            profile: The model profile to use.
+        """
+        self._settings = settings
+        self._profile = profile
+
+    @property
+    def settings(self) -> ModelSettings | None:
+        """Get the model settings."""
+        return self._settings
 
     @abstractmethod
     async def request(

pydantic_ai_slim/pydantic_ai/models/anthropic.py

@@ -127,6 +127,7 @@ def __init__(
         *,
         provider: Literal['anthropic'] | Provider[AsyncAnthropic] = 'anthropic',
         profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ):
         """Initialize an Anthropic model.
 
@@ -136,13 +137,15 @@ def __init__(
             provider: The provider to use for the Anthropic API. Can be either the string 'anthropic' or an
                 instance of `Provider[AsyncAnthropic]`. If not provided, the other parameters will be used.
             profile: The model profile to use. Defaults to a profile picked by the provider based on the model name.
+            settings: Default model settings for this model instance.
         """
         self._model_name = model_name
 
         if isinstance(provider, str):
             provider = infer_provider(provider)
         self.client = provider.client
-        self._profile = profile or provider.model_profile
+
+        super().__init__(settings=settings, profile=profile or provider.model_profile)
 
     @property
     def base_url(self) -> str:

pydantic_ai_slim/pydantic_ai/models/bedrock.py

@@ -202,6 +202,7 @@ def __init__(
         *,
         provider: Literal['bedrock'] | Provider[BaseClient] = 'bedrock',
         profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ):
         """Initialize a Bedrock model.
 
@@ -213,13 +214,15 @@ def __init__(
                 'bedrock' or an instance of `Provider[BaseClient]`. If not provided, a new provider will be
                 created using the other parameters.
             profile: The model profile to use. Defaults to a profile picked by the provider based on the model name.
+            settings: Model-specific settings that will be used as defaults for this model.
         """
         self._model_name = model_name
 
         if isinstance(provider, str):
             provider = infer_provider(provider)
         self.client = cast('BedrockRuntimeClient', provider.client)
-        self._profile = profile or provider.model_profile
+
+        super().__init__(settings=settings, profile=profile or provider.model_profile)
 
     def _get_tools(self, model_request_parameters: ModelRequestParameters) -> list[ToolTypeDef]:
         tools = [self._map_tool_definition(r) for r in model_request_parameters.function_tools]

pydantic_ai_slim/pydantic_ai/models/cohere.py

@@ -111,6 +111,7 @@ def __init__(
         *,
         provider: Literal['cohere'] | Provider[AsyncClientV2] = 'cohere',
         profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ):
         """Initialize an Cohere model.
 
@@ -121,13 +122,15 @@ def __init__(
                 'cohere' or an instance of `Provider[AsyncClientV2]`. If not provided, a new provider will be
                 created using the other parameters.
             profile: The model profile to use. Defaults to a profile picked by the provider based on the model name.
+            settings: Model-specific settings that will be used as defaults for this model.
         """
         self._model_name = model_name
 
         if isinstance(provider, str):
             provider = infer_provider(provider)
         self.client = provider.client
-        self._profile = profile or provider.model_profile
+
+        super().__init__(settings=settings, profile=profile or provider.model_profile)
 
     @property
     def base_url(self) -> str:

pydantic_ai_slim/pydantic_ai/models/fallback.py

@@ -42,6 +42,7 @@ def __init__(
             fallback_models: The names or instances of the fallback models to use upon failure.
             fallback_on: A callable or tuple of exceptions that should trigger a fallback.
         """
+        super().__init__()
         self.models = [infer_model(default_model), *[infer_model(m) for m in fallback_models]]
 
         if isinstance(fallback_on, tuple):

pydantic_ai_slim/pydantic_ai/models/function.py

@@ -52,7 +52,12 @@ class FunctionModel(Model):
 
     @overload
     def __init__(
-        self, function: FunctionDef, *, model_name: str | None = None, profile: ModelProfileSpec | None = None
+        self,
+        function: FunctionDef,
+        *,
+        model_name: str | None = None,
+        profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ) -> None: ...
 
     @overload
@@ -62,6 +67,7 @@ def __init__(
         stream_function: StreamFunctionDef,
         model_name: str | None = None,
         profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ) -> None: ...
 
     @overload
@@ -72,6 +78,7 @@ def __init__(
         stream_function: StreamFunctionDef,
         model_name: str | None = None,
         profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ) -> None: ...
 
     def __init__(
@@ -81,6 +88,7 @@ def __init__(
         stream_function: StreamFunctionDef | None = None,
         model_name: str | None = None,
         profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ):
         """Initialize a `FunctionModel`.
 
@@ -91,16 +99,19 @@ def __init__(
             stream_function: The function to call for streamed requests.
             model_name: The name of the model. If not provided, a name is generated from the function names.
             profile: The model profile to use.
+            settings: Model-specific settings that will be used as defaults for this model.
         """
         if function is None and stream_function is None:
             raise TypeError('Either `function` or `stream_function` must be provided')
+
         self.function = function
         self.stream_function = stream_function
 
         function_name = self.function.__name__ if self.function is not None else ''
         stream_function_name = self.stream_function.__name__ if self.stream_function is not None else ''
         self._model_name = model_name or f'function:{function_name}:{stream_function_name}'
-        self._profile = profile
+
+        super().__init__(settings=settings, profile=profile)
 
     async def request(
         self,

pydantic_ai_slim/pydantic_ai/models/gemini.py

@@ -133,6 +133,7 @@ def __init__(
         *,
         provider: Literal['google-gla', 'google-vertex'] | Provider[httpx.AsyncClient] = 'google-gla',
         profile: ModelProfileSpec | None = None,
+        settings: ModelSettings | None = None,
     ):
         """Initialize a Gemini model.
 
@@ -142,6 +143,7 @@ def __init__(
                 'google-gla' or 'google-vertex' or an instance of `Provider[httpx.AsyncClient]`.
                 If not provided, a new provider will be created using the other parameters.
             profile: The model profile to use. Defaults to a profile picked by the provider based on the model name.
+            settings: Default model settings for this model instance.
         """
         self._model_name = model_name
         self._provider = provider
@@ -151,7 +153,8 @@ def __init__(
         self._system = provider.name
         self.client = provider.client
         self._url = str(self.client.base_url)
-        self._profile = profile or provider.model_profile
+
+        super().__init__(settings=settings, profile=profile or provider.model_profile)
 
     @property
     def base_url(self) -> str: