SignalR connections are shutdown directly during graceful shutdowns

[WJRovers]

Context:
Hosting an application in a kubernetes environment can sometimes cause the service (with active connections) to be shutdown as autoscaling happens. We have implemented graceful shutdown behaviour for various parts of the application but I see no such feature/possibility for SignalR servers. This causes problems when using the streaming features in SignalR. The Hub interface does offer you a OnDisconnectedAsync but as it turns out this is fired after the connection has been closed (by the server itself).

Things Tried
I've stepped into the source code with a debugger to find out most is handled in privates/internal classes. Should a complete new implementation be written of a HubConnectionHandler that can deal with IApplicationLifetime? Or am I missing something like an injectable class that can deal with this. Or is there an existing SignalR lifetime available?
Also; Im not sure if the HubFilters that seem to be coming out in a later release would help us with our problem.

Question/Idea:
We would like to await stream completion (with a max timeout) before the SignalR connection is cut off. Is it possible to interact with the IApplicationLifetime or await completion before SignalR connections are actually closed?

[davidfowl]

You might be able to achieve something if you handle the IHostApplicationLifetime notification before SignalR does and closes all connections. This logic actually isn't directly in the signalr layer, it's part of the transport logic.

[WJRovers]

I am not sure this is not possible as I could not find any interaction between the SignalR hosting service and the IHostApplicationLifetime. Ive done a little test:

Startup.cs

public void Configure(IApplicationBuilder app, IHostApplicationLifetime applicationLifetime)
{
     applicationLifetime.ApplicationStopping.Register(() =>
     {
          Console.WriteLine("App is stopping - We should wait before SignalR closes connections?");
     }, true);

    // Some more stuff here

ChatHub

public async Task StreamData(ChannelReader<string> streamOfStrings){
    try
    {
        await foreach (var receivedString in _channelReader.ReadAllAsync())
            Console.WriteLine(receivedString)
    catch (OperationCanceledException e)
    {
        Console.WriteLine(e.Message) // "The underlying connection was closed."
    }
}

When you start a Graceful shutdown during a stream of strings being send; the console will show the followiing:
Console Output

The underlying connection was closed.
App is stopping

To me it looks like the SignalR service does not respect or interact with any of the IHostApplicationLifetime events. Am I missing something obvious or is this something that is not implemented?

Little extra information
When you have a SignalR client with auto reconnect; and you put something like a thread sleep (for testing) in the ApplicationStopping registration. It closes the connection during graceful shutdown; and then allows the agent to reconnect.

[davidfowl]

What server are you running on?

[BrennanConroy]

So one idea would be to add a hook to allow users to run something before SignalR starts closing connections.

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[BrennanConroy]

We're thinking of adding an option to delay connection closes. By default there is no delay so today as soon as shutdown is triggered connections start closing. With the option you can specify how long SignalR delays once shutdown is triggered before trying to gracefully close connections.

The server can still aggressively shutdown connections that are still alive, Kestrel has configuration for this delay, not sure about other servers.

cc @davidfowl

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.