Repository navigation
Run dstack server on the asyncio event loop - #4363
Open
peterschmidt85 wants to merge 1 commit into
Open
peterschmidt85 wants to merge 1 commit into
peterschmidt85 wants to merge 1 commit into
Conversation
uvloop spawns subprocesses with fork(). With gRPC and OpenTelemetry loaded, the forked ssh child could crash before exec and hang the server for up to 5 minutes (#4362). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4362.
With a Nebius backend and OpenTelemetry enabled, the server could freeze for up to 5 minutes while starting
sshfor a service probe. Behind a load balancer health check, the frozen server was killed, and Nebius instances it was creating at that moment kept running untracked.dstack servernow runs uvicorn on the asyncio event loop instead of uvloop. asyncio starts subprocesses withvfork(), so gRPC and OpenTelemetry fork handlers no longer run in thesshchild, which could crash there beforeexecwhile the server waited on it.If you start the server with the
uvicornCLI rather thandstack server, pass--loop asyncio.Gateways keep uvloop; they don't load gRPC or OpenTelemetry.
Verified with the #4362 steps on ECS (Amazon Linux 2023,
systemd-coredump, OpenTelemetry on): 0.22.3 froze twice within 4 minutes; with this change, 0 crashes and 0 freezes in 58 minutes.🤖 Generated with Claude Code