Skip to content

Commit 2f0ecac

Browse files
.Net: Improve auto-recovery for Azure{OpenAI} models (#10275)
### Motivation and Context Every time an AI model calls a function, and that call fails for whatever reason - hallucinated name, invalid arguments, etc. SK sends the failure details back to the model to allow it to correct (auto-recovery) itself and call the function one more time with the valid name, arguments, etc. Having tested this auto-recovery mechanism with a few modern {Azure}OpenAI models, it appeared that it does not work for the `Gpt-4(0314, 0613)` and `Gpt-4o(2024-08-06)` models. To make it work for the `Gpt-4o(2024-08-06)` model, the default error message needs to be extended either with the `Correct yourself` instruction, function name, or both. This solution is not enough to make the `Gpt-4(0314, 0613)` model's auto-recovery work, and as shown in the table below, the only way found so far to make it work is to use prompt engineering: | Error message/Model | Gpt-4(0314, 0613) | Gpt-4-Turbo(0125-Preview, 1106-Preview) | Gpt-4o mini(2024-07-18) | Gpt-4o(2024-08-06) | |----------------------------------------|--------------------|-------------------------------------------|---------------------------|---------------------| | `Error: Function call request for a function that wasn't defined` default message. | X | ✔️ | ✔️ | X | | Add `Correct yourself` instruction to the default error message. | X | ✔️ | ✔️ | ✔️ | | Add function name to the default error message. | X | ✔️ | ✔️ | ✔️ | | Add function name & `Correct yourself` instruction to the default error message.| X | ✔️ | ✔️ | ✔️ | | Use `You can call tools. If a tool call failed, correct yourself.` system message. | ✔️ | N/A | N/A | X | The `N/A` in the last row means that there is no need for prompt engineering because the AI model can auto-recover without it. ### Description This PR adds the `Correct yourself` option to the default error message to enable auto-recovery for the `Gpt-4o(2024-08-06)` model. Additional context: [Function Calling Reliability](https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0063-function-calling-reliability.md) Closes: #8472
1 parent 391d1b4 commit 2f0ecac

File tree

2 files changed

+6
-6
lines changed

2 files changed

+6
-6
lines changed

dotnet/src/InternalUtilities/connectors/AI/FunctionCalling/FunctionCallsProcessor.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -312,14 +312,14 @@ private static bool TryValidateFunctionCall(
312312
// Check if the function call has an exception.
313313
if (functionCall.Exception is not null)
314314
{
315-
errorMessage = $"Error: Function call processing failed. {functionCall.Exception.Message}";
315+
errorMessage = $"Error: Function call processing failed. Correct yourself. {functionCall.Exception.Message}";
316316
return false;
317317
}
318318

319319
// Make sure the requested function is one of the functions that was advertised to the AI model.
320320
if (!checkIfFunctionAdvertised(functionCall))
321321
{
322-
errorMessage = "Error: Function call request for a function that wasn't defined.";
322+
errorMessage = "Error: Function call request for a function that wasn't defined. Correct yourself.";
323323
return false;
324324
}
325325

@@ -330,7 +330,7 @@ private static bool TryValidateFunctionCall(
330330
return true;
331331
}
332332

333-
errorMessage = "Error: Requested function could not be found.";
333+
errorMessage = "Error: Requested function could not be found. Correct yourself.";
334334
return false;
335335
}
336336

dotnet/src/SemanticKernel.UnitTests/Utilities/AIConnectors/FunctionCallsProcessorTests.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,7 @@ await this._sut.ProcessFunctionCallsAsync(
164164

165165
Assert.Equal("MyPlugin", functionResult.PluginName);
166166
Assert.Equal("Function1", functionResult.FunctionName);
167-
Assert.Equal("Error: Function call processing failed. Deserialization failed.", functionResult.Result);
167+
Assert.Equal("Error: Function call processing failed. Correct yourself. Deserialization failed.", functionResult.Result);
168168
}
169169

170170
[Fact]
@@ -225,7 +225,7 @@ await this._sut.ProcessFunctionCallsAsync(
225225

226226
Assert.Equal("MyPlugin", functionResult.PluginName);
227227
Assert.Equal("Function1", functionResult.FunctionName);
228-
Assert.Equal("Error: Function call request for a function that wasn't defined.", functionResult.Result);
228+
Assert.Equal("Error: Function call request for a function that wasn't defined. Correct yourself.", functionResult.Result);
229229
}
230230

231231
[Fact]
@@ -253,7 +253,7 @@ await this._sut.ProcessFunctionCallsAsync(
253253

254254
Assert.Equal("MyPlugin", functionResult.PluginName);
255255
Assert.Equal("Function1", functionResult.FunctionName);
256-
Assert.Equal("Error: Requested function could not be found.", functionResult.Result);
256+
Assert.Equal("Error: Requested function could not be found. Correct yourself.", functionResult.Result);
257257
}
258258

259259
[Theory]

0 commit comments

Comments
 (0)