fix(node): Set backfill max job duration to a safe amount of time#90
Merged
fix(node): Set backfill max job duration to a safe amount of time#90
Conversation
Our timeout crash is caused by a footgun on Reth's backfill threshold configuration API. Here's how it roughly looks:
- reth allows the mdbx tx timeout sentinel thread to have a configurable max timeout, hidden behind a flag (--db.read-transaction-timeout). We currently, AFAIK, do not configure this. The default value is 5 minutes. Any transactions that live beyond that are killed.
- Reth, for the exex backfill, sets the max_duration for its backfill processes to 30 seconds. This does not exceed the default max timeout for the mdbx sentinel thread.
- The `ExecutionStageThresholds` default for max duration for execution jobs is 10 minutes. This, by far, exceeds the mdbx timeout sentinel thread max duration. On the execution stage for the reth node this is safe as it calls `disable_long_read_transaction_safety()`, which bypasses the timeout. This is not done on backfill, so this default is unsafe. This is therefore a massive footgun for any exex that configures backfill thresholds.
- We were only tweaking the `max_blocks` config value, and using the unsafe default for max duration, when setting the backfill thresholds. This leads to us avoiding an OOM crash, but then running into an mdbx crash due to the sentinel thread forcibly killing the transaction.
- Before `set_backfill_thresholds` was added, what caused the OOM was the massive amount of blocks being processed. `max_duration` had a sane default (30s). Once it was introduced, this new bug was added due to the `::default()` usage.
- The fix, is therefore, to limit the max duration to a reasonably low time. 30s should be fine.
- Note: 30s is a reasonably margin due to a few reasons:
- We avoid the hard 5m timeout.
- The intended default for exex backfill was already 30s.
- There's a 60s timeout warning already, this avoids it entirely. We've seen this error spuriously over the entirety of the signet node's lifecycle.
dylanlott
reviewed
Feb 24, 2026
crates/node/src/node.rs
Outdated
| max_blocks: Some(max_blocks), | ||
| // Keep the backfill mdbx read transaction open for no longer than 30 seconds. | ||
| // This prevents MDBX from killing the read transaction, leading to a crash if reth's default mdbx read transaction timeout is exceeded. | ||
| max_duration: Some(Duration::from_secs(30)), |
Contributor
There was a problem hiding this comment.
Do we want to expose this to configs with 30 seconds as the default?
Member
Author
There was a problem hiding this comment.
No. The correct fix for this is for reth to make this footgun impossible
Contributor
There was a problem hiding this comment.
Sure but do we actually think that'll happen? I just can easily see a future where we're back to tweaking this and the above max_blocks values because of another change.
Member
Author
There was a problem hiding this comment.
Right I see what you mean now. Yeah, we can expose this
dylanlott
approved these changes
Feb 24, 2026
Contributor
dylanlott
left a comment
dylanlott
left a comment
There was a problem hiding this comment.
LGTM. Let's get this loaded up and syncing ASAP. Good detective work! 🔍
crates/node/src/node.rs
Outdated
| max_blocks: Some(max_blocks), | ||
| // Keep the backfill mdbx read transaction open for no longer than 30 seconds. | ||
| // This prevents MDBX from killing the read transaction, leading to a crash if reth's default mdbx read transaction timeout is exceeded. | ||
| max_duration: Some(Duration::from_secs(30)), |
Contributor
There was a problem hiding this comment.
Sure but do we actually think that'll happen? I just can easily see a future where we're back to tweaking this and the above max_blocks values because of another change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Our timeout crash is caused by a footgun on Reth's backfill threshold configuration API. Here's how it roughly looks:
ExecutionStageThresholdsdefault for max duration for execution jobs is 10 minutes. This, by far, exceeds the mdbx timeout sentinel thread max duration. On the execution stage for the reth node this is safe as it callsdisable_long_read_transaction_safety(), which bypasses the timeout. This is not done on backfill, so this default is unsafe. This is therefore a massive footgun for any exex that configures backfill thresholds.max_blocksconfig value, and using the unsafe default for max duration, when setting the backfill thresholds. This leads to us avoiding an OOM crash, but then running into an mdbx crash due to the sentinel thread forcibly killing the transaction.set_backfill_thresholdswas added, what caused the OOM was the massive amount of blocks being processed.max_durationhad a sane default (30s). Once it was introduced, this new bug was added due to the::default()usage.This now exposes a new
BACKFILL_MAX_DURATION, which tweaks the max duration time, in seconds.Related Issue
Testing