-
Notifications
You must be signed in to change notification settings - Fork 2.6k
feat(core): resilient daemon restart with exponential backoff reconnection #33432
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
✅ Deploy Preview for nx-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
93da5af to
649a466
Compare
|
View your CI Pipeline Execution ↗ for commit 9feb468
☁️ Nx Cloud last updated this comment at |
649a466 to
3ac9089
Compare
fbd6d42 to
5a960f4
Compare
5a960f4 to
248eca6
Compare
248eca6 to
07375e9
Compare
07375e9 to
58a07c2
Compare
58a07c2 to
5e0b6d3
Compare
5e0b6d3 to
58f0db9
Compare
ace03dc to
e7ef248
Compare
e7ef248 to
cf5980d
Compare
9b6e5ec to
d2bf2f0
Compare
d2bf2f0 to
3483131
Compare
3483131 to
ad7ad18
Compare
ad7ad18 to
8ba84e8
Compare
8ba84e8 to
065d366
Compare
065d366 to
bf35169
Compare
bf35169 to
9d8d5e7
Compare
…ndling - Add exponential backoff reconnection for daemon client - Implement version check handshake in socket messenger - Improve socket error handling and prevent hanging on errors - Keep watch process alive during file change listening - Simplify socket messenger by removing socketPath parameter - Move socket creation into listen() to prevent unhandled errors - Add graceful daemon reconnection with user feedback - Restart daemon on lock file and version changes - Prevent watch connections from exiting during reconnection - Improve in-flight request handling during daemon reconnection - Add periodic check for daemon being outdated - Extract shared connections for file watchers and graph listeners - Simplify pending message handling in reconnection - Silence noisy reconnection logs during normal restarts - Fix file-server executor to use output.log instead of logger
9d8d5e7 to
9feb468
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nx Cloud is proposing a fix for your failed CI:
We removed an unguarded debug console.log statement from the daemon client's handleMessage method that was outputting "[Client] handleMessage called, currentMessage type: REQUEST_PROJECT_GRAPH" to stdout. This debug output was being captured by the e2e watch tests and contaminating their assertions, causing 6 tests to fail with unexpected output in the results.
We could not verify this fix.
diff --git a/packages/nx/src/daemon/client/client.ts b/packages/nx/src/daemon/client/client.ts
index ed370c9f69..75cb28f643 100644
--- a/packages/nx/src/daemon/client/client.ts
+++ b/packages/nx/src/daemon/client/client.ts
@@ -1030,10 +1030,6 @@ export class DaemonClient {
}
private handleMessage(serializedResult: string) {
- console.log(
- '[Client] handleMessage called, currentMessage type:',
- this.currentMessage?.type
- );
try {
performance.mark('result-parse-start-' + this.currentMessage.type);
const parsedResult = isJsonMessage(serializedResult)
Or Apply changes locally with:
npx nx-cloud apply-locally IkmJ-yY6q
Apply fix locally with your editor ↗ View interactive diff ↗
🎓 Learn more about Self-Healing CI on nx.dev
Current Behavior
The daemon currently depends on client requests failing with a
LOCK_FILES_CHANGEDerror to trigger a restart. This creates several issues:Expected Behavior
The daemon should proactively restart itself when lock files change, and the client should gracefully reconnect with exponential backoff for any server shutdown scenario.
Changes
Client-Side: Exponential Backoff Reconnection
handleConnectionError()method that retries with exponential backoff (10ms → 5000ms, 30 attempts max)LOCK_FILES_CHANGEDandNX_VERSION_CHANGEDerrorsretryMessageAfterNewDaemonStarts()method as it's no longer neededServer-Side: Self-Restart on Lock File Changes
startNewDaemonInBackground()to spawn a replacement daemon before shutdownhandleServerProcessTerminationWithRestart()for restartable shutdown scenariosBenefits
Related Issue(s)
Part of daemon rework to make restarts more reliable and resilient.