Kestrel memory pool does not reliably decrease with load

[GREsau]

Describe the bug

While load testing a new service using SignalR, we noticed that memory usage does not always decrease after connections drop off. Sometimes it decreases a minute or two after the connections end, but sometimes it just never recovers. This causes us two problems:

1. When an instance nears its memory limit, it will reject new connections (achieved using custom middleware that responds with a 503) in order to protect its existing connections. But since memory usage does not decrease, the instance will never accept new connections, even after most of its existing connections end
2. Our service autoscales on memory, but because memory usage stays high our service will rarely scale-down when under low load. This is less of a problem, it just costs a bit to run with extra capacity

This shows memory usage increasing with number of connections (tracked using the OnConnectedAsync/OnDisconnectedAsync methods of our Hub), but not reliably decreasing afterwards:

The service is running in an Alpine 3.12 docker container, but we notice similar behaviour when running locally in Windows 10. We initially assumed this was a simple memory leak, but when making a large number of connections over time while keeping the number of concurrent connections low, memory usage does not increase.

When taking a heap dump, we see memory fairly dominated by Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets!Buffers.MemoryPoolBlock:

To Reproduce

1. Make a large number of concurrent connections to a SignalR Core service
2. End connections
3. Monitor memory - it should decrease, but usually does not

Further technical details

• ASP.NET Core version 3.1.6, using docker image based on mcr.microsoft.com/dotnet/core/aspnet:3.1.6-alpine3.12
• Built with SDK 3.1.302, using docker image based on mcr.microsoft.com/dotnet/core/sdk:3.1.302-alpine3.12
• Reproduced locally on Windows 10 using ASP.NET Core 3.1.6, SDK 3.1.202

[BrennanConroy]

Kestrel will pool internal buffer up to a limit and will not let them go. We should consider having a settable limit or release strategy for the pool.

cc @halter73

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[halter73]

Related: aspnet/KestrelHttpServer#609, aspnet/KestrelHttpServer#2673

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.