Fix race in `PeerManager` read pausing. #4168

TheBlueMatt · 2025-10-22T21:15:39Z

We recently ran into a race condition on macOS where `read_event`
would return `Ok(true)` (implying reads should be paused) but calls
to `send_data` which flushed the buffer completed before the
`read_event` caller was able to set the read-pause flag.

This should be fairly rare, but not unheard of - the `pause_read`
flag in `read_event` is calculated before handling the last
message, so there's some time between when its calculated and when
its returned. However, that has to race with multiple calls to
`send_data` to send all the pending messages, which all have to
complete before the `read_event` return happens. We've (as far as I
can tell) never hit this on Linux, but a benchmark HTLC-flood test
managed to hit it somewhat reliably within a few minutes on macOS.

Ultimately we can't fix this with the current API (though we could
make it more rare). Thus, here, we stick to a single "stream" of
pause-read events from `PeerManager` to user code via `send_data`
calls, dropping the read-pause flag return from `read_event`
entirely.

Technically this adds risk that someone can flood us with enough
messages fast enough to bloat our outbound buffer for a peer before
`PeerManager::process_events` gets called and can flush the pause
flag via `read_event` calls to all descriptors. This isn't ideal
but it should still be relatively hard to do as `process_events`
calls are pretty quick and should be triggered immediately after
each `read_event` call completes.

ldk-reviews-bot · 2025-10-22T21:15:42Z

👋 Thanks for assigning @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

codecov · 2025-10-22T21:34:52Z

Codecov Report

❌ Patch coverage is 81.25000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.63%. Comparing base (05f2848) to head (bd4356a).
⚠️ Report is 29 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/ln/peer_handler.rs	82.92%	4 Missing and 3 partials ⚠️
lightning-background-processor/src/lib.rs	0.00%	1 Missing ⚠️
lightning-net-tokio/src/lib.rs	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4168      +/-   ##
==========================================
- Coverage   88.78%   88.63%   -0.16%     
==========================================
  Files         180      179       -1     
  Lines      137004   136979      -25     
  Branches   137004   136979      -25     
==========================================
- Hits       121642   121409     -233     
- Misses      12538    12838     +300     
+ Partials     2824     2732      -92

Flag	Coverage Δ
fuzzing	`?`
tests	`88.63% <81.25%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joostjager · 2025-10-24T10:56:29Z

lightning-net-tokio/src/lib.rs

-										us_lock.read_paused = true;
-									}
-								},
+								Ok(()) => {},


Would it be easy to reproduce the problem on linux by reducing OUTBOUND_BUFFER_LIMIT_READ_PAUSE and adding a delay? Just to verify that the bug really is what we think it is.

No need, just by adding an extra few-ms sleep after the handle_message call (after setting pause_read) easily reproduces (and this PR fixes it).

joostjager · 2025-10-24T11:28:23Z

lightning-net-tokio/src/lib.rs

 			us.read_paused = false;
 			let _ = us.read_waker.try_send(());
+		} else if !resume_read {
+			us.read_paused = true;


Would a comment here be beneficial, or a fn-level doc explaining the resume_read semantics?

Better yet, I simplified the code to be a bit clearer so that its hopefully not needed.

joostjager · 2025-10-24T11:35:56Z

lightning/src/ln/peer_handler.rs

 	/// Note that these messages are *not* encrypted/MAC'd, and are only serialized.
 	gossip_broadcast_buffer: VecDeque<MessageBuf>,
 	awaiting_write_event: bool,
+	sent_pause_read: bool,


Is this necessary to avoid always calling into send_data with no data, and obtaining the conn lock unnecessarily?

Yea, basically. We don't want to just slam each SocketDescriptor with a call every time we go through the process_events loop (which is very often).

lightning/src/ln/peer_handler.rs

ldk-reviews-bot · 2025-10-24T11:40:45Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

joostjager · 2025-10-24T11:41:26Z

Ultimately we can't fix this with the current API (though we could
make it more rare).

What API change would be required to fix it completely? And is a reason not to do it to avoid breaking external usage of this code?

TheBlueMatt · 2025-10-24T20:44:45Z

The API change in this PR should fix it completely. My comment was about backporting to 0.1, where we aren't allowed to remove the returned-bool from read_event, and probably shouldn't silently change the semantics of the resume_read/continue_read bool (which now implies we should stop reading if its false).

We recently ran into a race condition on macOS where `read_event` would return `Ok(true)` (implying reads should be paused) but calls to `send_data` which flushed the buffer completed before the `read_event` caller was able to set the read-pause flag. This should be fairly rare, but not unheard of - the `pause_read` flag in `read_event` is calculated before handling the last message, so there's some time between when its calculated and when its returned. However, that has to race with multiple calls to `send_data` to send all the pending messages, which all have to complete before the `read_event` return happens. We've (as far as I recall) never hit this in prod, but a benchmark HTLC-flood test managed to hit it somewhat reliably within a few minutes on macOS and when a synthetic few-ms sleep was added to each message handling call. Ultimately we can't fix this with the current API (though we could make it more rare). Thus, here, we stick to a single "stream" of pause-read events from `PeerManager` to user code via `send_data` calls, dropping the read-pause flag return from `read_event` entirely. Technically this adds risk that someone can flood us with enough messages fast enough to bloat our outbound buffer for a peer before `PeerManager::process_events` gets called and can flush the pause flag via `read_event` calls to all descriptors. This isn't ideal but it should still be relatively hard to do as `process_events` calls are pretty quick and should be triggered immediately after each `read_event` call completes.

In the previous commit, we moved the `send_data` `resume_read` flag to also indicate that we should pause if its unset. This should work as we mostly only set the flag when we're sending but may cause us to fail to pause if we are blocked on gossip validation but `awaiting_write_event` wasn't set as we had previously failed to fully flush a buffer (which no longer implies read-pause). Here we make this logic much more robust by ensuring we always make at least one `send_data` call in `do_attempt_write_data` if we need to pause read (or unpause read).

TheBlueMatt · 2025-10-25T14:17:59Z

Dropped the first commit as it makes it more annoying to remove the spurious Box::pins now that our MSRV is higher.

TheBlueMatt added this to the 0.2 milestone Oct 22, 2025

TheBlueMatt requested a review from joostjager October 22, 2025 21:15

TheBlueMatt added the backport 0.2 label Oct 22, 2025

TheBlueMatt force-pushed the 2025-10-net-race-fixes branch from aa5f64e to ad1e948 Compare October 22, 2025 21:18

joostjager reviewed Oct 24, 2025

View reviewed changes

TheBlueMatt force-pushed the 2025-10-net-race-fixes branch from ad1e948 to e4a70b9 Compare October 24, 2025 20:44

TheBlueMatt added 4 commits October 25, 2025 14:15

f update net-tokio to new semantics and simplify a bit

18f44b4

f update all impl arg names

6ab5924

TheBlueMatt force-pushed the 2025-10-net-race-fixes branch from e4a70b9 to bd4356a Compare October 25, 2025 14:15

Fix race in PeerManager read pausing. #4168

Are you sure you want to change the base?

Fix race in PeerManager read pausing. #4168

Conversation

TheBlueMatt commented Oct 22, 2025

Uh oh!

ldk-reviews-bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joostjager Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

joostjager Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

joostjager Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ldk-reviews-bot commented Oct 24, 2025

Uh oh!

joostjager commented Oct 24, 2025

Uh oh!

TheBlueMatt commented Oct 24, 2025

Uh oh!

TheBlueMatt commented Oct 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fix race in `PeerManager` read pausing. #4168

Fix race in `PeerManager` read pausing. #4168

ldk-reviews-bot commented Oct 22, 2025 •

edited

Loading

codecov bot commented Oct 22, 2025 •

edited

Loading