Fix reorg - Create savepoint when close to tip and at max reorg depth

[emilcondrea]

Details

Create savepoint when close to tip and at max reorg depth

• this is useful as previously some times the first savepoint would have been the tip/really close to tip sometimes and if a reorg happened it could not restore at all
• if above happened it tried to infinitely restore the snapshot and it increased the index size by 1GB every few minutes (would be good to understand why did that happen)

Context

This happened while trying to index testnet4(ord 22.1), and it indexes all the way to the tip (in my case 63038), then this happens:

• it commits to the tip and creates the savepoint at the tip.
• Then a reorg of deep 2 happens, ord tries to rollback to oldest savepoint which is still 63038, thinks its recoverable (but its not, as the savepoint is not that old),
• reverts to the oldest savepoint which basically does nothing,
• and then does this in an infinite loop and increases the used disk space exponentially its like 1 more GB each 2-3 mins. (ended up filling all disk space on my mac over night)

Here are some additional logs I captured while debugging this

2025-01-07T14:23:40Z INFO  ord::index::reorg] creating savepoint at height 63038
fetched block 00000000566eb6fe93c9c72553e97aa64d37667df74b9f2de423789ae4a420fe

index hash(00000000f399a5a201c25a0fcff0d4331c0fc819da001ac199f8c787a321f6df) vs bitcoind_hash(000000009cfa3d0e34a663389d28cf721674211ffeda3b5b601b2f6ababb9c1f)
detect_reorg: 00000000566eb6fe93c9c72553e97aa64d37667df74b9f2de423789ae4a420fe
bitcoind height63037
index hash(00000000f399a5a201c25a0fcff0d4331c0fc819da001ac199f8c787a321f6df) vs bitcoind_hash(000000009cfa3d0e34a663389d28cf721674211ffeda3b5b601b2f6ababb9c1f)
detect_reorg1: 00000000566eb6fe93c9c72553e97aa64d37667df74b9f2de423789ae4a420fe
bitcoind height63036
index hash(0000000028db011b8cab6a5e58ebb2105015f065a37ae0cf32885c94ac5a4bf3) vs bitcoind_hash(0000000028db011b8cab6a5e58ebb2105015f065a37ae0cf32885c94ac5a4bf3)
[2025-01-07T14:23:45Z INFO  ord::index] 2 block deep reorg detected at height 63038
[2025-01-07T14:23:45Z INFO  ord::index::reorg] rolling back database after reorg of depth 2 at height 63038
existing savepoints: [1]
[2025-01-07T14:23:46Z INFO  ord::index::reorg] successfully rolled back database to height 63038
...

Tested

Indexed testnet 4 and it created the savepoint 21 blocks away from the tip

...
[2025-01-07T15:48:04Z INFO  ord::index::reorg] is_savepoint_required=true: height=63024, last_savepoint_height=57205, blocks=63045
[2025-01-07T15:48:04Z INFO  ord::index::updater] Committing at block height 63024, 0 outputs traversed, 599370 in map, 300205 cached
[2025-01-07T15:48:28Z INFO  ord::index::reorg] is_savepoint_required=true: height=63024, last_savepoint_height=57205, blocks=63045
[2025-01-07T15:48:28Z INFO  ord::index::reorg] creating savepoint at height 63024
[2025-01-07T15:48:28Z INFO  ord::index::updater] Block 63024 at 2025-01-07 15:23:21 UTC with 60 transactions…
[2025-01-07T15:48:28Z INFO  ord::index::updater] Wrote 0 sat ranges from 0 outputs in 5 ms
[2025-01-07T15:48:28Z INFO  ord::index::reorg] is_savepoint_required=false: height=63025, last_savepoint_height=63024, blocks=63045
[2025-01-07T15:48:28Z INFO  ord::index::updater] Block 63025 at 2025-01-07 14:23:19 UTC with 34 transactions…
[2025-01-07T15:48:28Z INFO  ord::index::updater] Wrote 0 sat ranges from 0 outputs in 1 ms
[2025-01-07T15:48:28Z INFO  ord::index::reorg] is_savepoint_required=false: height=63026, last_savepoint_height=63024, blocks=63045
[2025-01-07T15:48:28Z INFO  ord::index::updater] Block 63026 at 2025-01-07 14:43:20 UTC with 3 transactions…
[2025-01-07T15:48:28Z INFO  ord::index::updater] Wrote 0 sat ranges from 0 outputs in 0 ms
[2025-01-07T15:48:28Z INFO  ord::index::reorg] is_savepoint_required=false: height=63027, last_savepoint_height=63024, blocks=63045
[2025-01-07T15:48:28Z INFO  ord::index::updater] Block 63027 at 2025-01-07 15:03:21 UTC with 1 transactions…
[2025-01-07T15:48:28Z INFO  ord::index::updater] Wrote 0 sat ranges from 0 outputs in 0 ms
[2025-01-07T15:48:28Z INFO  ord::index::reorg] is_savepoint_required=false: height=63028, last_savepoint_height=63024, blocks=63045
[2025-01-07T15:48:28Z INFO  ord::index::updater] Block 63028 at 2025-01-07 15:23:22 UTC with 1 transactions…
[2025-01-07T15:48:28Z INFO  ord::index::updater] Wrote 0 sat ranges from 0 outputs in 0 ms
[

[raphjaph]

I'm going to have a look at this over the weekend. Could you check out if all the tests pass locally for you? CI seems to keep failing.

[emilcondrea]

Fixed tests. This should skip commiting for integ tests as it would require too long sleep duration for it to work. It passes all test slocally on my mac, I see on CI fails on ubuntu but I cannot reproduce this locally.

[emilcondrea]

Run the tests locally on an unbuntu env and it succeeds, probably something else happening in CI?

[emilcondrea]

Seems testnet4 has deep reorgs, https://testnet4.ordinals.com/status is unrecoverably reorged. If I am reading correctly the logs there was a reorg of 13 blocks deep, we should probably make configurable the savepoint interval?

[raphjaph]

LGTM

[raphjaph]

I don't know why the CI doesn't pass