Skip to content

Conversation

michaelsproul
Copy link
Member

@michaelsproul michaelsproul commented Jul 17, 2025

Issue Addressed

A different (and complementary) approach for:

Proposed Changes

This PR adds a flag to set the DA boundary to the Deneb fork. The effect of this change is that Lighthouse will try to backfill all blobs.

Most peers do not have this data, but I'm thinking that combined with trusted-peers this could be quite effective.

Additional Info

  • Should definitely turn on --prune-blobs false when this flag is enabled, and report a conflict with --prune-blobs true.
  • Should we turn on genesis-backfill whenever complete-blob-backfill is turned on? I'm inclined to say no, because users may want to backfill blobs only to e.g. 5 months (standard block backfill distance).

@chong-he
Copy link
Member

chong-he commented Sep 3, 2025

I have done some testing on this together with @michaelsproul

To backfill the blobs beyond the 18-day window, we supply to the node that we want to backfill the following flags:
--libp2p-addresses
--trusted-peer

Example: lighthouse bn --libp2p-addresses /ip4/ip_address/tcp/9000 --trusted-peer 16Ulongstring --some-other-flags

It is also recommended to supply the same flag to the source node (the node with the blobs data).

Some caveats/known issues:

  1. From time to time the following logs will appear:
ERROR Backfill sync failed                          error: BatchDownloadFailed(Epoch(365247))
WARN  Backfill batch failed to download. Penalizing peers  score_adjustment: Low Tolerance Error, batch_epoch: 365329 

I believe this is expected as most of the peers do not have the blobs data outside the 18-day window. As the peers are penalized, this results in a fluctuating peer count, as evident from the connected peer below (note that the default peer is now 200 in the unstable branch):

Screenshot from 2025-09-03 20-47-18
  1. The backfill of blocks+blobs further from the head could be slow, in the range of 2-4 slots/sec (this could also be partly due to hardware limitations). So a complete backfill of blobs until the Deneb fork is estimated to take some time, could be as long as a few weeks.

    This could be improve if there are more nodes with blobs data are supplied - though this is not yet tested. In the testing, we supply only 1 node

  2. The backfill could stuck - I have not investigated this. When it stalls, a restart of the beacon node will "resume" the backfill

Additional info: the node blobs data has backfilled to slot 10.2M from a fresh checkpoint sync:

"anchor": {
    "anchor_slot": "12452832",
    "oldest_block_slot": "10202176",
    "oldest_block_parent": "0xc91bf2f51b1ffdefe6dd791ba9c0465bf486cedea6f28d231b8f5742012be6d9",
    "state_upper_limit": "18446744073709551615",
    "state_lower_limit": "0"
  },
  "blob_info": {
    "oldest_blob_slot": "10202176",
    "blobs_db": true
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants