Crash during sync #3526
Description
DBG 2025-07-31 12:29:27.459+00:00 Returning JSON-RPC response id=138 name=engine_forkchoiceUpdatedV3 len=101
DBG 2025-07-31 12:29:27.459+00:00 Validating block topics="forked chain" blkHash=0x0fb7998cd5f3c33bb76b6f3a3e5081a36092b6c3afdcd9fad36d81d982718494 blk="(parentHash: 0x199184be94dfe5bf5dbd0825aa70244b5b86a1743a5f2f5e85baf750ea6a0b4c, coinbase: 0x91e32efb8139cd88cae0df30d2bf471294c6ed27, stateRoot: 0xfc4bd9f68d82684b3c1a30cc946dabad39a8a088f7ad2bb49b658959e54b772b, transactionsRoot: 0x0f5b1e5e5982f1ad8936bb0a18995ec2998f3603b04d12c2c9a4db3d06ef2f14, receiptsRoot: 0x88a4353f12691de668a6b74214ac7e5cd0cbe4ce2ba69e89b1b6126171336323, number: 914353, gasLimit: 59941408, gasUsed: 59094609, nonce: 0x0000000000000000, baseFeePerGas: ok(981943811), withdrawalsRoot: ok(0xad69400a3fe74ddf5f3794c7ffb7154b47c2a05dfcf2bbe73e7771c2c128514c), blobGasUsed: ok(524288), excessBlobGas: ok(0), parentBeaconBlockRoot: ok(0x395132cf76c4fbd1bfae380ab1d648ae187d386dd52c852286dab37b462bad9c), requestsHash: ok(0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855))" parentTxFrame=124406200714768 txFrame=124406195292336
DBG 2025-07-31 12:29:27.941+00:00 Discovery receive error topics="p2p" discv4="disc: msg missing hash" discv5="Invalid protocol id"
DBG 2025-07-31 12:29:27.941+00:00 Parent block not found topics="forked chain" blockHash=199184be94df parentHash=8928ce99d20d
DBG 2025-07-31 12:29:27.941+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[135.125.98.216:12007]
DBG 2025-07-31 12:29:27.941+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[77.239.109.40:30303]
DBG 2025-07-31 12:29:27.942+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[149.106.232.13:30303]
DBG 2025-07-31 12:29:27.942+00:00 Accept timeout topics="p2p rlpx" remoteAddress=65.109.117.125:4001
INF 2025-07-31 12:29:27.942+00:00 Import error (skip remaining) topics="beacon sync" n=17 iv=#914335..#914355 nBlocks=21 nthBn=#914352 nthHash=199184be94df base=#914161 head=#914251 blkFailCount=1 error="(excp: ENoException, name: , msg: Block is not part of valid chain, elapsed: 483ms409us752ns)"
INF 2025-07-31 12:29:27.942+00:00 Imported blocks topics="beacon sync" iv=#914335..#914351 nBlocks=17 nFailed=4 base=#914161 head=#914251 target=#914355 targetHash=309a41e26f02
INF 2025-07-31 12:29:27.942+00:00 Queued/staged or imported blocks topics="beacon sync" topImported=#914351 unprocBottom=#914352 nQueued=0 nImported=57 nStagedQ=0 nSyncPeers=1
DBG 2025-07-31 12:29:27.942+00:00 RunStop: release peer topics="beacon sync" peer=Node[134.199.184.23:30303] nSyncPeers=0 syncState=ZombieRun:blocks.locked
DBG 2025-07-31 12:29:27.942+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[3.1.76.101:9222]
DBG 2025-07-31 12:29:27.942+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[34.159.49.180:9004]
DBG 2025-07-31 12:29:27.942+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[18.185.199.51:9003]
DBG 2025-07-31 12:29:27.942+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[84.244.95.233:12001]
DBG 2025-07-31 12:29:27.942+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[38.80.81.183:9003]
DBG 2025-07-31 12:29:28.342+00:00 Core DB persisted kvtDur=14ms593us8ns mptDur=106ms878us352ns endDur=278ms455us16ns stateRoot=ok(0xd08fcc312d7277bb145db41f4cfed308560028b67f4d8e9cb10c5579a536863c)
NTC 2025-07-31 12:29:28.353+00:00 Finalized blocks persisted topics="forked chain" nBlocks=64 base=914225 baseHash=9834ee894240 pendingFCU=b87cf1970496 resolvedFin=914303
DBG 2025-07-31 12:29:28.353+00:00 Useless peer finishing hello topics="p2p rlpx" err="Eth handshake for different network" remote=Node[150.136.175.212:30303] clientId=Geth/v1.14.13-stable-eb00f169/linux-amd64/go1.23.5
DBG 2025-07-31 12:29:28.359+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[101.100.184.226:9000]
DBG 2025-07-31 12:29:28.359+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[47.74.40.220:30303]
DBG 2025-07-31 12:29:28.359+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[51.68.248.27:12081]
DBG 2025-07-31 12:29:28.359+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[84.46.247.196:30303]
DBG 2025-07-31 12:29:28.360+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[57.128.33.201:30303]
DBG 2025-07-31 12:29:28.360+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[169.155.169.27:9000]
DBG 2025-07-31 12:29:28.360+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[157.90.2.145:12000]
DBG 2025-07-31 12:29:28.360+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[5.9.142.245:9000]
DBG 2025-07-31 12:29:28.360+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[193.30.123.158:9000]
DBG 2025-07-31 12:29:28.361+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[213.199.48.105:30303]
DBG 2025-07-31 12:29:28.361+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[135.181.48.77:9000]
DBG 2025-07-31 12:29:28.361+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[135.181.77.41:9000]
DBG 2025-07-31 12:29:28.361+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[34.116.203.26:30303]
DBG 2025-07-31 12:29:28.361+00:00 Connect handshake disconnection topics="p2p rlpx" err="Peer disconnecting during hello: TooManyPeers" reason=TooManyPeers remote=Node[65.21.78.105:30303]
DBG 2025-07-31 12:29:28.361+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[198.96.95.214:9001]
DBG 2025-07-31 12:29:28.361+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[148.113.163.141:9000]
DBG 2025-07-31 12:29:28.361+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[15.204.208.183:9222]
DBG 2025-07-31 12:29:28.361+00:00 Connect transport error topics="p2p rlpx" err="Data incomplete!" remote=Node[207.244.241.121:9000]
Traceback (most recent call last, using override)
/home/advaita/nimbus-eth1/vendor/nim-json-rpc/json_rpc/client.nim(863) main
/home/advaita/nimbus-eth1/vendor/nim-json-rpc/json_rpc/client.nim(851) NimMain
/home/advaita/nimbus-eth1/execution_chain/nimbus_execution_client.nim(292) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
/home/advaita/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
/home/advaita/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
/home/advaita/nimbus-eth1/execution_chain/core/chain/forked_chain.nim(536) _ZN12processQueue12processQueueE3refIN7futures26FutureBasecolonObjectType_EE
/home/advaita/nimbus-eth1/execution_chain/core/chain/forked_chain.nim(518) _ZN13processOrphan12asyncHandlerE
/home/advaita/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
/home/advaita/nimbus-eth1/execution_chain/core/chain/forked_chain.nim(522) _ZN12asyncHandler45asyncHandler
/home/advaita/nimbus-eth1/execution_chain/core/chain/forked_chain.nim(509) _ZN12forked_chain13processOrphanE3refIN10chain_desc30ForkedChainRefcolonObjectType_EE3refIN12chain_branch24BlockRefcolonObjectType_EE4bool
/home/advaita/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
/home/advaita/nimbus-eth1/execution_chain/core/chain/forked_chain.nim(513) _ZN13processOrphan13processOrphanE3refIN7futures26FutureBasecolonObjectType_EE
/home/advaita/nimbus-eth1/execution_chain/core/chain/forked_chain.nim(426) _ZN12forked_chain13validateBlockE3refIN10chain_desc30ForkedChainRefcolonObjectType_EE3refIN12chain_branch24BlockRefcolonObjectType_EEN6blocks5BlockE4bool
/home/advaita/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
/home/advaita/nimbus-eth1/execution_chain/core/chain/forked_chain.nim(438) _ZN13validateBlock13validateBlockE3refIN7futures26FutureBasecolonObjectType_EE
/home/advaita/nimbus-eth1/execution_chain/db/core_db/base.nim(449) _ZN4base12txFrameBeginE3refIN9base_desc27CoreDbTxRefcolonObjectType_EE
/home/advaita/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(652) signalHandler
/home/advaita/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(334) _ZN6system18rawWriteStackTraceE3varI6stringE
/home/advaita/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(59) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI6stringE
SIGSEGV: Illegal storage access. (Attempt to read from nil?)