You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
.Net: Improve auto-recovery for Azure{OpenAI} models (#10275)
### Motivation and Context
Every time an AI model calls a function, and that call fails for
whatever reason - hallucinated name, invalid arguments, etc. SK sends
the failure details back to the model to allow it to correct
(auto-recovery) itself and call the function one more time with the
valid name, arguments, etc.
Having tested this auto-recovery mechanism with a few modern
{Azure}OpenAI models, it appeared that it does not work for the
`Gpt-4(0314, 0613)` and `Gpt-4o(2024-08-06)` models. To make it work for
the `Gpt-4o(2024-08-06)` model, the default error message needs to be
extended either with the `Correct yourself` instruction, function name,
or both. This solution is not enough to make the `Gpt-4(0314, 0613)`
model's auto-recovery work, and as shown in the table below, the only
way found so far to make it work is to use prompt engineering:
| Error message/Model | Gpt-4(0314, 0613) | Gpt-4-Turbo(0125-Preview,
1106-Preview) | Gpt-4o mini(2024-07-18) | Gpt-4o(2024-08-06) |
|----------------------------------------|--------------------|-------------------------------------------|---------------------------|---------------------|
| `Error: Function call request for a function that wasn't defined`
default message. | X | ✔️ | ✔️ | X |
| Add `Correct yourself` instruction to the default error message. | X |
✔️ | ✔️ | ✔️ |
| Add function name to the default error message. | X | ✔️ | ✔️ | ✔️ |
| Add function name & `Correct yourself` instruction to the default
error message.| X | ✔️ | ✔️ | ✔️ |
| Use `You can call tools. If a tool call failed, correct yourself.`
system message. | ✔️ | N/A | N/A | X |
The `N/A` in the last row means that there is no need for prompt
engineering because the AI model can auto-recover without it.
### Description
This PR adds the `Correct yourself` option to the default error message
to enable auto-recovery for the `Gpt-4o(2024-08-06)` model.
Additional context: [Function Calling
Reliability](https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0063-function-calling-reliability.md)
Closes: #8472
0 commit comments