Skip to content

Fix flaky pex reactor tests that could miss an evicted peer reconnecting - #4221

Merged
masih merged 2 commits into
mainfrom
masih/1789622341-flaky-pex-peer-down
Sep 18, 2026
Merged

masih merged 2 commits into
mainfrom
masih/1789622341-flaky-pex-peer-down

Conversation

@masih

@masih masih commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

TestReactorSendsRequestsTooOften hung for the full 30 minute test timeout in listenForPeerDown. The CI log shows the eviction did happen (evicting ... pex: peer rate limit exceeded at 32.842) but the mock peer was dialled and connected again three milliseconds later, because both routers still know each other's address from connectAll. WaitForConn(target, false) waits on the peer manager's connection set for a state in which no connection to that node ID exists, and that state only existed inside that three millisecond window, so a waiter that was not scheduled in between never sees it and waits forever. The same tests were only ever passing because the redial usually lost that race.

The fix uses the existing WaitForDisconnect, which waits for one specific *ConnV2 to leave the connection set rather than for the node ID to be absent; that is a monotone condition, so a reconnect cannot mask it. The pex suite captures the connection with peerConn before triggering the eviction and hands it to listenForPeerDown, and TestNode.Disconnect waits on the connections it closed the same way instead of on WaitForConn(target, false), since it has the same exposure to a redial.

Flaked in: https://github.com/sei-protocol/sei-chain/actions/runs/34871012761/job/104066604846

@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedSep 17, 2026, 10:29 AM

@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.83%. Comparing base (4c97b0f) to head (a17fdeb).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
sei-tendermint/internal/p2p/testonly.go 0.00% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4221      +/-   ##
==========================================
- Coverage   66.97%   65.83%   -1.14%     
==========================================
  Files        2182     2061     -121     
  Lines      167488   155541   -11947     
==========================================
- Hits       112171   102402    -9769     
+ Misses      55176    52998    -2178     
  Partials      141      141              
Flag Coverage Δ
sei-chain-pr 83.74% <0.00%> (?)
sei-db 74.50% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-tendermint/internal/p2p/testonly.go 79.08% <0.00%> (-0.82%) ⬇️

... and 121 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@masih
masih marked this pull request as ready for review September 17, 2026 10:22
@masih
masih requested a review from wen-coding September 17, 2026 10:22
@cursor

cursor Bot commented Sep 17, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Test-only changes to P2P helpers and PEX reactor tests; no production networking behavior is modified.

Overview
Fixes flaky PEX reactor tests that could hang until timeout when an evicted peer redialed immediately after disconnect.

Test harness: listenForPeerDown no longer waits for “no connection to node ID” (WaitForConn(..., false)). Tests capture the live *ConnV2 via new peerConn before triggering eviction, then assert teardown with WaitForDisconnect on that handle so a quick reconnect cannot satisfy the waiter.

Shared test helper: TestNode.Disconnect closes connections to a peer and waits on each closed *ConnV2 the same way, instead of waiting for the peer ID to be absent from the connection set.

Reviewed by Cursor Bugbot for commit a17fdeb. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6eb19cd. Configure here.

Comment thread sei-tendermint/internal/p2p/testonly.go Outdated

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flake diagnosis is correct and the fix is sound — capturing the specific *ConnV2 makes the wait condition monotone, and peerManager.Evict closes every conn for the node ID so either direction's captured conn is a valid target. One reuse issue: the new WaitForConnClosed duplicates the existing WaitForDisconnect helper in the same file.

Findings: 0 blocking | 1 non-blocking | 1 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • None at the file/PR level.
  • 1 suggestion(s)/nit(s) flagged inline on specific lines.

Comment thread sei-tendermint/internal/p2p/testonly.go Outdated
@masih
masih requested a review from codchen September 17, 2026 12:37
@masih
masih enabled auto-merge September 17, 2026 12:52
@masih
masih added this pull request to the merge queue Sep 18, 2026
Merged via the queue into main with commit 5c3040b Sep 18, 2026
78 checks passed
@masih
masih deleted the masih/1789622341-flaky-pex-peer-down branch September 18, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants