Skip to content

Fix SSH session resume with bounded backpressure - #6650

Open
anton-107 wants to merge 2 commits into
mainfrom
deco-28433-reconnect-backpressure
Open

Fix SSH session resume with bounded backpressure#6650
anton-107 wants to merge 2 commits into
mainfrom
deco-28433-reconnect-backpressure

Conversation

@anton-107

@anton-107 anton-107 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Changes

Restore SSH session resume with bounded backpressure: a full replay window pauses the sender until acknowledgments free space. Separate acknowledgment handling and concurrent replay keep handover from deadlocking. Negotiate protocol version 2, preserve final bytes before EOF, and bound reconnect retries and abandoned-session retention.

Why

Transient websocket resets terminate customers' active terminal and IDE sessions. #6558 truncated healthy transfers when its 1 MiB replay window filled; #6609 would disable resume during large transfers and mishandle subsequent acknowledgments. This fix keeps resume enabled while applying backpressure.

Tests

  • Reproduced the prior 1 MiB truncation and handover/replay deadlocks. Regression tests now verify byte-exact 8 MiB transfers each way through handover and repeated resets, plus negotiation, cancellation, handshake drops, and session cleanup.
  • SSH unit/local acceptance tests, full unit tests, targeted race tests, Windows compilation, full lint, and repository checks passed. The full acceptance run failed outside SSH (79 failures among 5,130 tests), including user-agent snapshot differences from meta-harness/omnigent, completion SIGPIPE, and bundle test timeouts; the full suite is not green.
  • Changed Go files are formatted and passed targeted lint. Python/YAML formatting passed; the full Go formatter was stopped without completion. A broader race run reproduced an existing VS Code settings backup test race on the original checkout.
  • Live-workspace validation remains required before release. Local acceptance covers existing SSH behavior; new proxy tests exercise resume directly because the local acceptance server does not negotiate resume.

This PR was written by Codex.

DECO-28433: restore dropped-session reattachment without the full-window
failure introduced by #6558 or the protocol degradation attempted in #6609.

Wait for acknowledgments outside the write lock, send acknowledgments on a
separate loop, and resume reads before replaying full windows. Negotiate a
new protocol version to avoid enabling resume against the faulty server.
Preserve final payload before EOF, distinguish session completion from a
dropped socket, retry dropped handshakes, and stop retrying expired sessions.

Regression tests reproduce the original 1 MiB truncation, an acknowledgment
deadlock during handover, and a simultaneous replay deadlock. They verify
8 MiB transfers in both directions through handover and repeated resets.

Validation: SSH unit and local acceptance suites; race tests for changed
SSH and telemetry packages; targeted lint; Windows test compilation.
Live driver-proxy validation remains required before release.
@github-actions

Copy link
Copy Markdown
Contributor

Approval status: pending

/libs/telemetry/ - needs approval

Files: libs/telemetry/protos/ssh_tunnel.go
Suggested: @renaudhartert-db
Also eligible: @simonfaltum, @Divyansh-db, @hectorcast-db, @parthban-db, @tanmay-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

General files (require maintainer)

18 files changed
Based on git history:

  • @janniklasrose -- recent work in libs/testserver/, .nextchanges/cli/, experimental/ssh/internal/proxy/

Any maintainer (@andrewnester, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 445ec66

Run: 34621007770

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux 1 275 15 6:10
💚​ aws windows 1 277 13 4:20
💚​ azure linux 1 274 15 5:54
💚​ azure windows 1 276 13 5:28
💚​ gcp linux 1 275 15 7:10
💚​ gcp windows 1 277 13 6:02
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
5:05 gcp windows TestAccept
3:51 azure windows TestAccept
3:38 aws windows TestAccept

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants