[Bug]: Server freezes for 5 minutes when a forked ssh crashes in gRPC, killing the server and leaking Nebius instances #4362

Steps to reproduce

1. Run dstack server with a Nebius backend on a host where core dumps go to systemd-coredump, such as the ECS-optimized Amazon Linux 2023 AMI. Enable OpenTelemetry: set DSTACK_OTEL_TRACES_ENABLED, DSTACK_OTEL_LOGS_ENABLED, DSTACK_OTEL_METRICS_ENABLED and OTEL_EXPORTER_OTLP_ENDPOINT.
2. Run a service with several replicas and an HTTP probe, for example replicas: 7 and probes: [{type: http, url: /, interval: 15s}]. For every probe, the server opens an SSH tunnel to the replica and closes it again.
3. Keep the server calling the Nebius API. For example, apply a Nebius fleet and a service with several replicas on it that keeps retrying when there is no capacity.
4. Poll /healthcheck.

We ran these steps with dstack 0.22.3 as released: 2 servers on ECS (t3.xlarge, Amazon Linux 2023) behind a load balancer health check, Postgres, the probed service on an SSH fleet, and a fake Nebius API.

Actual behaviour

Within minutes the server stops logging and stops answering /healthcheck. In our test, every server froze within 4 minutes of starting. At the same moment, the host kernel logs a crash in the copy that the server had just forked to run ssh, and systemd-coredump starts on it:

kernel: event_engine[77937]: segfault at 7f8547ffe370 ... in cygrpc.cpython-311-x86_64-linux-gnu.so
systemd[1]: Started systemd-coredump@1-77939-0.service - Process Core Dump

While the server is frozen:

• its main thread is blocked in read(), waiting for the copy;
• the copy, still a full copy of the server, is in do_exit while its core is written;
• in our production image, py-spy showed the main thread in create_subprocess_exec ← SSHTunnel.aclose ← get_service_replica_tunnel ← _execute_probe, a probe closing its tunnel.

If nothing kills it first, the server resumes after 300 s, when systemd-coredump hits its 5-minute timeout. In our production deployment, resumed servers then logged Failed to close SSH tunnel, exit status: -11 (SIGSEGV) or -6 (SIGABRT). Behind a load balancer health check, the server is killed and replaced before that. Nebius instances it was creating at that moment keep running, but dstack never records them.

In our production deployment, the server froze 45 times between Sep 8 and Oct 9, 2026. 44 of those freezes match a crashed copy's core dump in the host's records. 3 happened while Nebius instances were being created and left 5 H200 instances running untracked, for up to 5 days.

Expected behaviour

The server keeps answering /healthcheck and records every instance it creates.

dstack version

0.22.3 (grpcio 1.84.0, uvloop 0.23.0, nebius 0.6.20, Python 3.11). It also reproduces with grpcio 1.83.0, uvloop 0.22.1 and nebius 0.6.17, the versions in our production image.

Notes

Root cause

1. dstack server doesn't set uvicorn's event loop, so uvicorn uses uvloop. uvloop starts every subprocess with fork() and holds Python's global lock until the copy calls exec, so the whole server waits for each copy.

2. In the copy, two things run before exec:

   • gRPC (used by the Nebius SDK) restarts its threads;
   • OpenTelemetry's after-fork handlers start new worker threads.

   Together they can crash the copy (a segfault or abort in cygrpc) before it reaches exec.

3. With systemd-coredump, the crashed copy stays alive until its core is written or the 5-minute timeout hits. The server waits all that time, the health check kills it, and any instance it was creating leaks.

What each condition does, from runs of the steps above:

• Event loop: with the standard asyncio loop instead of uvloop, nothing crashes.
• OpenTelemetry: with it off, there were 0 freezes in about 4,500 forks made while gRPC was in use. Enabling only Sentry doesn't trigger it.
• Library versions: both version sets listed under "dstack version" freeze.

In our deployment, OpenTelemetry has been on since Jul 20, and services are probed all the time. Freezes were rare, 2 between Jul 20 and Sep 27, until Sep 28, when a user's services began creating H200 instances on Nebius. Forks made while gRPC was in use went from 2–330 a day to about 1,600. On Oct 1, an upgrade of the Nebius SDK from 0.3.101 to 0.6.17 kept gRPC active between calls, and that rose to 3,400–5,200 per server per hour. A replacement server often froze again within minutes of starting.

Suggested fix

Run the server on the standard asyncio event loop. It starts subprocesses with vfork(), which runs no after-fork handlers in the child, so neither gRPC nor OpenTelemetry runs in the copy.

• In server.py, pass loop="asyncio" to uvicorn.run.
• Wherever the server is started with the uvicorn CLI, add --loop asyncio.

Tested

On the setup above, with OpenTelemetry on: