[Feature] Improve clarity about LLM configs in the documentation

[Devy99]

What feature would you like to see?

Currently, it is possible to setup configurations such as the temperature of the LLM both via dspy.LM(...) (documentation link) and dspy.Predict() (and for the other modules too, documentation link). However, from the documentation it is unclear what happens when we set different temperatures using both approaches.

For example:

import dspy

# Setting the temperature to 0.9 at LLM initialization
lm = dspy.LM('openai/gpt-4o-mini', temperature=0.9)
dspy.configure(lm=lm)

sentence = "it's a charming and often affecting journey."  # example from the SST-2 dataset.

# Setting the temperature to 0.2 at Module initialization
classify = dspy.Predict('sentence -> sentiment', temperature=0.2)

In the previous example, I would say that setting the temperature in the dspy.Predict module will override the initial configuration of the LLM. However, looking at the Predict class (source code) in the source code it seems the opposite (I also ask for confirmation)

def forward(self, **kwargs):
       import dspy

       # Extract the three privileged keyword arguments.
       assert "new_signature" not in kwargs, "new_signature is no longer a valid keyword argument."
       signature = ensure_signature(kwargs.pop("signature", self.signature))
       demos = kwargs.pop("demos", self.demos)

       # Here, the configurations (self.config) provided in the Predict constructor are merged 
       # with those in kwargs["config"] and overrided with the latter, if already specified.
       config = dict(**self.config, **kwargs.pop("config", {}))

Is it possible to better clarify this scenario?

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[Devy99]

@okhat , sorry to bother you, can I ask (for confirmation) whether in the provided example, the temperature set in the LLM configuration overrides that provided in the module?

[xaviermehaut]

have you tried :
with dspy.context(lm=dspy.LM('openai/gpt-4o-mini', temperature=0.9)) :

[Devy99]

@xaviermehaut not yet. My current use case consists of creating a custom module with several Predict / ChainOfThought modules, using the same LLM but with different temperatures.
Imagine something like this:

class CustomModule(dspy.Module):
    def __init__(self):
         self.classify = dspy.Predict('sentence -> sentiment', temperature=0)
         self.answer = dspy.ChainOfThought('question -> answer', temperature=0.2)
         ...

However, (from the documentation) it is not clear what happens when I set the temperature from the LLM configuration, like:
lm = dspy.LM('openai/gpt-4o-mini', temperature=0.2)

In this case, when I use self.classify, does it use a temperature of 0 or 0.2?

Right now, I am setting the temperature only in the single modules (i.e., self.answer = dspy.ChainOfThought('question -> answer', temperature=0.2)), but I am also curious to know which is the configuration priority.

[xaviermehaut]

have yout tested : class CustomModule(dspy.Module): def __init__(self):                    with dspy.context(lm=dspy.LM('gemini/gemini-1.5-pro', temperature=0.1, max_tokens=3800): self.classify = dspy.Predict('sentence -> sentiment', temperature=0)          with dspy.context(lm=dspy.LM('gemini/gemini-1.5-pro', temperature=0.7, max_tokens=500): self.answer = dspy.ChainOfThought('question -> answer', temperature=0.2)

…

________________________________ De : Alessandro Giagnorio ***@***.***> Envoyé : lundi 17 février 2025 14:17 À : stanfordnlp/dspy ***@***.***> Cc : Xavier MÉHAUT ***@***.***>; Mention ***@***.***> Objet : Re: [stanfordnlp/dspy] [Feature] Improve clarity about LLM configs in the documentation (Issue #7808) @xaviermehaut<https://github.com/xaviermehaut> not yet. My current use case consists of creating a custom module with several Predict / ChainOfThought modules, using the same LLM but with different temperatures. Imagine something like this: class CustomModule(dspy.Module): def __init__(self): self.classify = dspy.Predict('sentence -> sentiment', temperature=0) self.answer = dspy.ChainOfThought('question -> answer', temperature=0.2) ... However, (from the documentation) it is not clear what happens when I set the temperature from the LLM configuration, like: python lm = dspy.LM('openai/gpt-4o-mini', temperature=0.2) In this case, when I use self.classify, does it use a temperature of 0 or 0.2? Right now, I am setting the temperature only in the single modules (i.e., self.answer = dspy.ChainOfThought('question -> answer', temperature=0.2)), but I am also curious to know which is the configuration priority. — Reply to this email directly, view it on GitHub<#7808 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BJINXEVB4S5BFASQE7UKVDT2QHOPRAVCNFSM6AAAAABXE2SJJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRTGEYTINRZGI>. You are receiving this because you were mentioned.Message ID: ***@***.***> [Devy99]Devy99 left a comment (stanfordnlp/dspy#7808)<#7808 (comment)> @xaviermehaut<https://github.com/xaviermehaut> not yet. My current use case consists of creating a custom module with several Predict / ChainOfThought modules, using the same LLM but with different temperatures. Imagine something like this: class CustomModule(dspy.Module): def __init__(self): self.classify = dspy.Predict('sentence -> sentiment', temperature=0) self.answer = dspy.ChainOfThought('question -> answer', temperature=0.2) ... However, (from the documentation) it is not clear what happens when I set the temperature from the LLM configuration, like: python lm = dspy.LM('openai/gpt-4o-mini', temperature=0.2) In this case, when I use self.classify, does it use a temperature of 0 or 0.2? Right now, I am setting the temperature only in the single modules (i.e., self.answer = dspy.ChainOfThought('question -> answer', temperature=0.2)), but I am also curious to know which is the configuration priority. — Reply to this email directly, view it on GitHub<#7808 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BJINXEVB4S5BFASQE7UKVDT2QHOPRAVCNFSM6AAAAABXE2SJJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRTGEYTINRZGI>. You are receiving this because you were mentioned.Message ID: ***@***.***> [https://www.sibylone.com/wp-content/uploads/2025/01/Signatures-e-Mail-Meilleurs-Voeux-2025-4-1.png] Ce courrier électronique et, le cas échéant, les pièces jointes sont confidentiels et établis à l'attention exclusive de ses destinataires. Sa reproduction, totale ou partielle, ou toute autre utilisation sont interdites sans autorisation préalable de Sibylone. L’intégrité de ce courrier n’étant pas assurée sur Internet, Sibylone décline toute responsabilité au titre de son contenu, s'il a été altéré, déformé ou falsifié. Si vous le recevez par erreur, merci de bien vouloir le détruire et en informer son expéditeur. Pensez à l’environnement. N’imprimez ce courriel que si vous en avez vraiment besoin.

[chenmoneygithub]

@Devy99 Thanks for reporting the issue! It's truly confusing that we allow settings at different layers, and both at construction and call time.

To your original question, your pasted code config = dict(**self.config, **kwargs.pop("config", {})) is not related to the LM you set, it means your call time args in dspy.Predict overrides the value set in constructor. And the final result will go ahead override the one you set in dspy.LM:

dspy/dspy/clients/lm.py

Line 106 in a834c1d

kwargs = {**self.kwargs, **kwargs}

[Devy99]

@chenmoneygithub thanks!