Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
.Net: Improve auto-recovery for Azure{OpenAI} models (#10275)
### Motivation and Context Every time an AI model calls a function, and that call fails for whatever reason - hallucinated name, invalid arguments, etc. SK sends the failure details back to the model to allow it to correct (auto-recovery) itself and call the function one more time with the valid name, arguments, etc. Having tested this auto-recovery mechanism with a few modern {Azure}OpenAI models, it appeared that it does not work for the `Gpt-4(0314, 0613)` and `Gpt-4o(2024-08-06)` models. To make it work for the `Gpt-4o(2024-08-06)` model, the default error message needs to be extended either with the `Correct yourself` instruction, function name, or both. This solution is not enough to make the `Gpt-4(0314, 0613)` model's auto-recovery work, and as shown in the table below, the only way found so far to make it work is to use prompt engineering: | Error message/Model | Gpt-4(0314, 0613) | Gpt-4-Turbo(0125-Preview, 1106-Preview) | Gpt-4o mini(2024-07-18) | Gpt-4o(2024-08-06) | |----------------------------------------|--------------------|-------------------------------------------|---------------------------|---------------------| | `Error: Function call request for a function that wasn't defined` default message. | X | ✔️ | ✔️ | X | | Add `Correct yourself` instruction to the default error message. | X | ✔️ | ✔️ | ✔️ | | Add function name to the default error message. | X | ✔️ | ✔️ | ✔️ | | Add function name & `Correct yourself` instruction to the default error message.| X | ✔️ | ✔️ | ✔️ | | Use `You can call tools. If a tool call failed, correct yourself.` system message. | ✔️ | N/A | N/A | X | The `N/A` in the last row means that there is no need for prompt engineering because the AI model can auto-recover without it. ### Description This PR adds the `Correct yourself` option to the default error message to enable auto-recovery for the `Gpt-4o(2024-08-06)` model. Additional context: [Function Calling Reliability](https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0063-function-calling-reliability.md) Closes: #8472
- Loading branch information