.Net: Bug: .NET Bedrock streaming completion is keeping the request thread busy

[tlecomte]

Describe the bug
The current implementation of the Bedrock connector is using response.Stream.AsEnumerable() in BedrockChatCompletionClient.cs#L193, which is keeping the current .NET thread busy while it reads the response stream. This could cause thread exhaustion issues.

To Reproduce
Steps to reproduce the behavior:

• See code at

  semantic-kernel/dotnet/src/Connectors/Connectors.Amazon/Bedrock/Core/Clients/BedrockChatCompletionClient.cs

  Line 193 in f88cf5f

  foreach (var chunk in response.Stream.AsEnumerable())

Expected behavior
Replace .AsEnumerable() with something that would read the response stream in an async manner.

Screenshots
N/A

Platform

• OS: all
• IDE: N/A
• Language: C#
• Source: NuGet package version 1.26.0

Additional context

[tlecomte]

I realize that, ideally, the AWS .NET SDK would provide a solution out of the box. So I've also created aws/aws-sdk-net#3542 and will suggest a solution there first.

[evchaki]

@RogerBarreto - can you take a look at this? @tlecomte - thanks for submitting this, we will take a look at updating this.

[shethaadit]

Hi @markwallace-microsoft / @RogerBarreto, since the fix is available in new SDK, I am raising below PR to fix this issue.

References:

• Bedrock ConverseStreamResponse provides no easy way to read the streamed response without blocking the calling .NET thread aws/aws-sdk-net#3542
• There is no way to get bedrock streaming api responses using async io. aws/aws-sdk-net#3360 (comment)
• feat(Event streams): add support for cancellation and IAsyncEnumerable aws/aws-sdk-net#3543