Slow but constant memory increase in .net web services on Linux

[fstugren]

Description

We deploy an Asp.Net web service on AKS Linux nodes (.Net version 8.0). The service behavior is pretty standard: it accepts incoming https requests and makes outbound https requests.
All outbound requests use the HttpClient class and the IHttpClientFactory pattern. The factory creates a SocketsHttpHandler periodically and eventually replaces the old one. Http clients are created by the factory on the fly and always disposed. As far as I can tell the code always disposes the HttpResponseMessage objects it gets from HttpClient.SendAsync() calls. We don't see any SNAT port exhaustion.

Yet over time, the total memory consumed by the application pods steadily increases to the maximum allowed by the AKS deployment configuration; then eventually the pods are killed. It takes a few days to get there, but the pods invariably reach that limit. Memory goes up, garbage collector occasionally brings it down, but over time there's a steady increase. The service caches some data, but all caches eventually expire.

I captured full dumps with tools like dotnet-dump and dotnet-gcdump, but they have not been useful. All the stats reported by dotnet-dump show that the heap and the total memory consumed by the CLR (under 250MB) is much smaller than the total process memory (inching closer to 4GB)

We have not been able to identify the cause of this leak and it's affecting production services. Any ideas what we should look at? Are there any environment variables to enable so we can see more details in those dumps? It almost looks like there's an unmanaged memory leak, but I have no data yet to back this up.

Reproduction Steps

The repro consists of typically accepting inbound calls and making outbound http calls. I know it's a but vague, but I don't have something to pinpoint it now.

Expected behavior

The total memory is expected to go up and down, but it should not increase steadily over time.

Actual behavior

Total memory constantly increasing, while .net managed heap stays small.

Regression?

Unclear.

Known Workarounds

No response

Configuration

The service runs

• Standard asp.net application on .net 8.0, current sdk version 8.0.415
• Docker image mcr.microsoft.com/dotnet/aspnet:8.0-azurelinux3.0

Other information

No response

[am11]

There is a suggestion to use non-Server GC k8s in this article https://blog.markvincze.com/troubleshooting-high-memory-usage-with-asp-net-core-on-kubernetes/. You could also take memory dump of running app using:

$ kubectl exec -it <pod-name> -- dotnet-dump collect --process-id <pid> --output /tmp/dump.dmp

or attach profiler via VS or Rider etc. to see large object retention (and dispose them as appropriate).

[fstugren]

@am11 - thanks for following up. The puzzling part is that the dumps taken with dotnet-dump or dotnet-gcdump show a CLR memory footprint much smaller than the total process memory (200 of MB vs 4GB). There are a handful of objects on the LOH, not much to worry about. We are investigating what other tools can be used to get more diagnostic information on where that memory is used. Hopefully this group has some hints.

[mangod9]

it could be related to some native memory leaks in the networking layer like this one: #108557

[fstugren]

Can anyone advise what approaches are generally best used to investigate this kind of issues? There's a plethora of memory investigation tools available for Linux; it's not clear to me which ones are most likely to produce results.

I found similar issues reported in this older thread #89776 that talks about /memfd:doublemapper (deleted) memory sections. While I definitely see a lot of those (1600+ after a day or so), the vast majority of memory seems to be associated to segments reported as [ anon ] by pmap, approximately 3.5GB out of a total of 3.8GB.

[fstugren]

Things we have tried over the last few days, with help from @janvorli from the .net runtime team:

• The memory dumps we investigated did not show any obvious sign of a memory leak. The managed heap is running fine at merely a couple of hundred MB, while the total memory grows continuously, quickly hitting 2GB. If pods are left to run longer, eventually they hit 4GB, the limit configured in AKS for the pod, and are restarted.
• Initially the usage of StackExchange.Redis raised some red flags, but after fully disabling the Redis cache the behavior did not change, so that was ruled out
• Experimented with different values for MALLOC_MMAP_THRESHOLD_ and MALLOC_ARENA_MAX - while some improvement is noticed, the large unmanaged memory leak is still there
• Added unmanaged memory counters retrieved via the mallinfo2() API in Pinvoke - these have shown that there's a large amount of managed space that is technically free, but likely very fragmented, and can't be released
• Reverted back from the mcr.microsoft.com/dotnet/aspnet:8.0-azurelinux3.0 image to the prior mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner2.0 image. The slow memory growth is present in the old image, however, much less prominent. The azurelinux3.0 image seems much more memory hungry.
• I've seen the suggestion to disable client certificate revocation check via CRL. In the ConfigureKestrel() settings HttpsConnectionAdapterOptions.CheckCertificateRevocation is left to its default false value, so I assume that's already the case, unless CRL checks are turned on by some other option. The same goes for the outbound calls, which use HttpClient. The CRL check is not enabled explicitly on the SocketsHttpHandler that is managed by the IHttpClientFactory.
• heaptrack was mentioned too; I was able to build the tool, as it cannot be installed with the tdnf package manager on azurelinux, but I have not yet been able to run it. Would it provide some insights?