IIS does not restart the application after it is killed

[kevinlo]

Hi,

I have application A.exe that is run under IIS. A.exe will start a B.exe which listen to a websocket URL.

At first, when A.exe is hosted in IIS, it has problem to start B.exe because of the Exception_PrefixAlreadyRegistered. After some debugging, I find out the CreateDefaultBuilder will call UseIISIntegration which makes it ignore the UseUrls and use the ASPNETCORE url and port. I fix it by creating a new WebHostBuilder and do most of the calls in the CreateDefaultBuilder, except the UseIISIntegration.

I also find out I need to call UseSetting(WebHostDefaults.PreventHostingStartupKey, "true") or it will load the hosting assembly to UseIIsIntegration. Here is what I have:

            var builder = new WebHostBuilder()
                        .UseHttpSys()
                        .UseUrls(urls)
                        .UseSetting(WebHostDefaults.PreventHostingStartupKey, "true")
                        .UseContentRoot(Directory.GetCurrentDirectory())
                        .ConfigureAppConfiguration((hostingContext, config) =>
                        {
                            var env = hostingContext.HostingEnvironment;

                            config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                                  .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: true)
                                  .AddEnvironmentVariables();

                            if (env.IsDevelopment())
                            {
                                var appAssembly = Assembly.Load(new AssemblyName(env.ApplicationName));
                                if (appAssembly != null)
                                {
                                    config.AddUserSecrets(appAssembly, optional: true);
                                }
                            }

                            if (args != null)
                            {
                                config.AddCommandLine(args);
                            }
                        })
                        .ConfigureLogging((hostingContext, logging) =>
                        {
                            logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
                            logging.AddConsole();
                            logging.AddDebug();
                        })
                        .UseDefaultServiceProvider((context, options) =>
                        {
                            options.ValidateScopes = context.HostingEnvironment.IsDevelopment();
                        })
                        .UseStartup<Startup>();
            builder.Build().Run();

It works find after that. A.exe can be launched by IIS and A.exe can start B.exe.

However, when I use the Task Manager to kill A.exe and then I make the request to the IIS to run A, IIS does not re-launch A.exe.

After some debugging to the aspnetcore.dll, I find out when the A.exe is killed, the SERVER_PROCESS::HandleProcessExit will be called. It calls the CheckIfServerIsUp which calls the GetExtendedTcpTable and check if there is a TCP endpoint matching the port A.exe is listening.

The problem is, even A.exe is killed, it still has an endpoint listening to that port with A's process id. Therefore, the aspnetcore thinks A.exe still exists and it does not start a new one when the request comes in.

If I kill the B.exe first and then kill A.exe, that TCP endpoint does not exist and A.exe will be restarted by IIS when the request comes in.

I don't know why starting B.exe (which uses a different url and port from A.exe) will create a TCP endpoint that uses A's port and process id. I try to make B not doing anything and just has A to start B. The problem still exists.

I suspect there is something done by default in the hosting or UseHttpSys that make B adding that TCP endpoint using its host (A) process id and port.

Can anyone tell me how the TCP endpoint is created or give me some clues where I should look at?

[kevinlo]

It seems it has nothing to do with how B.exe is doing. I change the A.exe to start mspaint.exe by the following codes:

            ProcessStartInfo info = new ProcessStartInfo();
            info.FileName = "mspaint.exe";
            info.UseShellExecute = false;
            info.WindowStyle = ProcessWindowStyle.Hidden;
            info.Verb = "runas";
            Process proc = Process.Start(info);

I kill A.exe and leave mspaint.exe running, it still has the TCP endpoint of A.exe after it is killed and IIS won't start a A.exe when a new request comes in.

I don't understand why starting a new process will create a new TCP endpoint and it needs to kill all .exe started to clear the TCP endpoints and IIS will start a new A.exe.

Any comment is welcome.

[Tratcher]

This looks like the same issue as aspnet/KestrelHttpServer#2789 (comment), which turns out to be caused by process handle inheritance. Switching back to the Libuv transport avoids the issue, and I have a PR in 2.2 that fixes it for the Socket transport.

[kevinlo]

@Tratcher Thanks for your quick reply. I change the A.exe to call UseLibuv. Then, when the A.exe is killed, all the B.exe are killed too. Why is that where is the code doing that?

On the other hand, as all the processes are killed, sending new request really can restart the processes.

Could you please also point to your PR in 2.2 that fixes it for Socket transport?

[Tratcher]

See aspnet/KestrelHttpServer#2944

[kevinlo]

@Tratcher I think the links you provided really describe what I experience.

Do you know why all started processes are also killed when the starting process is killed after switching to UseLibuv?

Another question is if using the following call in the started process the right way to may those process be able to use a different url from the starting one?

.UseSetting(WebHostDefaults.PreventHostingStartupKey, "true")

[Tratcher]

They're all created as part of a Windows Job Object and when ANCM loses communication it shuts everything down. This prevents resource leaks.

Yes, PreventHostingStartupKey is what you want. You also need to avoid having UseIISIntegration called, which means you can't use CreateDefaultBuilder. Alternatively you could clear some of the environment variables it checks before activating.

IISIntegration/src/Microsoft.AspNetCore.Server.IISIntegration/WebHostBuilderIISExtensions.cs

Lines 43 to 47 in f808bdc

	var port = hostBuilder.GetSetting(ServerPort) ?? Environment.GetEnvironmentVariable($"ASPNETCORE_{ServerPort}");
	var path = hostBuilder.GetSetting(ServerPath) ?? Environment.GetEnvironmentVariable($"ASPNETCORE_{ServerPath}");
	var pairingToken = hostBuilder.GetSetting(PairingToken) ?? Environment.GetEnvironmentVariable($"ASPNETCORE_{PairingToken}");
	var iisAuth = hostBuilder.GetSetting(IISAuth) ?? Environment.GetEnvironmentVariable($"ASPNETCORE_{IISAuth}");
	var websocketsSupported = hostBuilder.GetSetting(IISWebSockets) ?? Environment.GetEnvironmentVariable($"ASPNETCORE_{IISWebSockets}");

[kevinlo]

@Tratcher Thanks for all your explanation. I now understand why when the AppPool is stopped, both A.exe and B.exe are killed. At first I think that is done by the killing codes in A.exe (we do have codes that will kill all B.exe when A.exe is stopped), now I think it is done by IIS.

I now understand why all these happen. You can close this issue.