YARP hostname routing + MapStaticAssets cause first request slowness and high memory usage

[luryus]

Describe the bug

When YARP is used with ~200 or more hostname routes, and the ASP.NET Core service also uses MapStaticAssets(), the first request is very slow to execute, uses lots of cpu, and the memory usage of the service rises very high. This does not happen if either MapReverseProxy() or MapStaticAssets() is excluded, or if only path routes are used in YARP.

Profiling shows the culprit might be DfaMatcher initialization in the framework. Something about my usage of YARP causes it to blow up. This is an issue when running in resource-limited container environments: with limited CPU and memory the first request will take a long time and the very high memory usage can cause OOMKills.

My service uses YARP to proxy traffic to a couple hundred of dynamically configured services, each routed based on an unique hostname. It also exposes it's own MVC UI on one hostname, and that UI needs static files. I'm using MapStaticAssets().RequireHost(...) to achieve this, because I couldn't figure out how to make the older UseStaticFiles() only respond on the single hostname and not on any of the YARP proxied hostnames.

Is there something that I can alter in the YARP configuration to avoid this slowness?

To Reproduce

Full reproducer project here: https://github.com/luryus/yarp-host-explosion

This Program.cs, combined with ~20-40 static files under wwwroot/ reproduces this issue.

using Yarp.ReverseProxy.Configuration;

var builder = WebApplication.CreateBuilder(args);
const int routeCount = 300;
var routes = new List<RouteConfig>(routeCount);
var cluster = new ClusterConfig
{
    ClusterId = "cluster",
    Destinations = new Dictionary<string, DestinationConfig>
    {
        { "server", new DestinationConfig { Address = "http://127.0.0.1" } }
    }
};

for (var i = 1; i <= routeCount; i++)
{
    var hostName = $"route{i:D5}.example.com";
    var routeId = $"route_{i}";
    routes.Add(new RouteConfig
    {
        Order = -1000, RouteId = routeId, ClusterId = cluster.ClusterId,
        Match = new RouteMatch { Hosts = [hostName] },
    });
}

builder.Services.AddReverseProxy().LoadFromMemory(routes, [cluster]);
builder.Services.AddControllersWithViews();
builder.Services.AddHealthChecks();

var app = builder.Build();
app.UseRouting();
app.MapReverseProxy();
app.MapStaticAssets().RequireHost("localhost");
app.Run();

After the service has started up, make a request to http://localhost:<port>/css/1.css. The first request takes over 5 seconds when running on my laptop, and the service memory usage rises to over 200 MB.

Further technical details

• Linux & Windows
• YARP 2.3.0
• ASP.NET Core 9.0.10

[MihaZupan]

When you define YARP routes we create corresponding ASP.NET Core routes. Routing optimizes for request throughput instead of startup, so the startup hit can be quite high as you use lots of routes.

This does not happen if either MapReverseProxy() or MapStaticAssets() is excluded, or if only path routes are used in YARP.

I'm not too familiar with routing internals, but it's possible that the combination triggers some condition that makes it more expensive to precompute the DFA for.

If you have lots of such routes where the only thing you're changing is the host, you could consider not defining separate routes at all. You could define a single catch-all lroute and validate the host in a custom middleware after the fact (e.g. with a dictionary lookup).
See also https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/yarp/direct-forwarding

I'll transfer this over to ASP.NET Core in case they're interested in this case for any perf investigations.

[MackinnonBuck]

because I couldn't figure out how to make the older UseStaticFiles() only respond on the single hostname and not on any of the YARP proxied hostnames.

As a workaround, you could probably use MapWhen() to achieve this.

[MackinnonBuck]

cc @javiercn in case you have any thoughts about this.