Replacing Badger with Pebble DB

[j1010001]

Why

1. Badger DB is a project that is not maintained anymore. At some point it will not be compatible with a new version of Go and will block the upgrade of flow-go to newer version of Go language.
2. Badger has caused memory spikes and performance issues in the past (so far mitigated with more aggressive GC GoMemLimit setting) and as the load on the DB grows these issues are likely to reoccur in the future.
3. Badger DB does not support data pruning, so on long-running network removing unused data requires node downtime.

We have identified following project milestones that complete the work of removing flow-go dependency on Badger DB.

✅ Milestone 1 - Refactor data access & prune chunk data packs

• Badger -> Pebble DB M1 - Chunk Data pack Pruner #6516

Milestone 2 - DB access refactoring for low-risk data on EN, VN and AN

• Badger -> Pebble DB M2 - DB access refactoring for low-risk data (AN, EN, VN) #6527

Milestone 3 - unblock pruning of Execution, Access and Verification data

Note: ideally we use the locking framework as proposed in: https://github.com/jordanschalm/lockctx/blob/main/README.md
Completing this milestone enables removal of dependency on Badger DB from Verification node.

Execution

• Follower

Verification

• Follower

Access

• Follower

Collection

• Follower

Common to all nodes - needed for completing pruning

• Protocol Data (Consensus Follower)
• Libp2p Blocklist

Milestone 4 - DB access refactoring - remove dependency on Badger DB completely form ENs and ANs

Execution

• Execution Data (Bitswap)

Access

• OKR draft: Replace Badger Transaction with Batch updates - Access Node-related data #6518
• Execution Data (Bitswap)
• Tx/Block Status RPC API
• Switching from Badger to Pebble

Milestone 5 - DB access refactoring - remove dependency on Badger DB completely form CN and SNs

OKR placeholder: #6528

Consensus

• Protocol Data (Consensus Builder)
  • BeaconKey, DKG State
  • Consensus Builder, Finalizer, Persister
  • ExecForkSuppressor
• Deployment - Switching from Badger to Pebble (Dynamic bootstrap (preferred), spork, or Migration)

Collection

• Collection Consensus

  • Cluster State
  • [Guarantee creation](https://github.com/onflow/flow-go/blob/master/storage/badger/procedure/cluster.go#L162
  • Collections Provider

• Deployment - Switching from Badger to Pebble (Dynamic bootstrap (preferred), spork, or Migration)

• Tools
  TBD

Milestone 6 - pruning of Execution, access and verification data

Task breakdown TDB

Milestone 7 - upgrade to Pebble 2.x

Task breakdown TDB

• [Storage] Evaluate migrating Badger->Pebble v2 (instead of Badger->Pebble v1) #7126

[fxamacker]

Badger DB is a project that is not maintained anymore.

@j1010001 did you mean that BadgerDB v2 and v3 are not maintained anymore (instead of entire BadgerDB project)?

BadgerDB released v4.0 in Feb 2023 and v4.3 in Aug 2024.

• v4.3.0 (Aug 28, 2024)
• v3.2103.5 (Dec 15, 2022) is the last v3.
• v2.2007.4 (Aug 25, 2021) is the last v2 and version currently used by flow-go (go.mod).

More details and other releases at BadgerDB releases.

[zhangchiqing]

Issues for this epic:

• Badger -> Pebble DB M1 - Chunk Data pack Pruner #6516
• Replace Badger Transaction with Batch updates - Access Node-related data #6518
• Replace Badger Transaction with Batch updates - Verification Node-related data  #6519
• Replace Badger Transaction with Batch updates - Execution Node-related data  #6520
• Replace Badger Transaction with Batch updates - Collection Node-related data #6521
• Replace Badger Transaction with Batch updates - Protocol-related data  #6522
• Database Operation Transition: From Badger to Pebble #6523

[j1010001]

Issues for this epic:

• Chunk Data pack Pruner #6516
• Replace Badger Transaction with Batch updates - Access Node-related data #6518
• Replace Badger Transaction with Batch updates - Verification Node-related data  #6519
• Replace Badger Transaction with Batch updates - Execution Node-related data  #6520
• Replace Badger Transaction with Batch updates - Collection Node-related data #6521
• Replace Badger Transaction with Batch updates - Protocol-related data  #6522
• Database Operation Transition: From Badger to Pebble #6523

All the issues above have been linked to the relevant phases

[bluesign]

did you have any preliminary results with running nodes with pebble instead of badger? I am using pebble and pretty happy so far, but was thinking to check badger for faster writes, though I am scared on memory usage etc a bit.

[zhangchiqing]

Unfortunately , we didn’t gather metrics for the proof-of-concept benchmark, as our focus was on ensuring execution correctness. Once this issue is completed, I will collect metrics for comparison.

[zhangchiqing]

Trying to list all the data and their location that need to be refactored.

• Execution

  • Execution Result [Storage Refactor] Refactor saving execution results #6906
  • Collections [Execution] Refactor collections storage #7059
  • StopControl (VersionBeacon) [Storage] Refactor storage version beacon #7085
  • Execution Data (Bitswap)
  • Result GCP Uploader [Storage] Refactor computation result status storage #7084
  • Switching from Badger to Pebble
    • Migrate the last executed result to pebble in order to make next block executable [Execution] Migrate last executed block from badger to pebble #7117
  • Follower

• Verification

  • Approvals [Storage Refactor] Refactor Approvals  #6868
  • ChunkQueue [Storage Refactor] Refactor Chunk Locators to badger updates #6947
  • Switching from Badger to Pebble [Storage Refactor] Using flags to roll out database operation changes for Verification nodes. #6948
  • Follower

• Access

  • Execution Data (Bitswap)
  • Execution Result [Storage Refactor] Refactor saving execution results #6906
  • Collections [Access] Refactor storage collections for access node #7093
  • StopControl (VersionBeacon) [Storage] Refactor storage version beacon #7085
  • Tx/Block Status RPC API
  • Switching from Badger to Pebble
  • Follower

• Follower

  • Protocol Data (Consensus Follower)
  • Libp2p Blocklist

• Consensus

  • Protocol Data (Consensus Builder)
    • BeaconKey, DKG State
    • Consensus Builder, Finalizer, Persister
    • ExecForkSuppressor
  • Switching from Badger to Pebble

• Collection

  • Collection Consensus
    • Cluster State
    • Guarantee creation
  • Collections Provider
  • Switching from Badger to Pebble
  • Follower

• Tools

  • Utility
    • add flags to read from pebble [Util] update util to read results from pebble #7092
    • Test if pebble can read data without stopping the process by creating a checkpoint of db on the fly. [Util] update util to read results from pebble #7092
  • AdminTool
    • If Utility can read pebble without stopping the process, we can get rid of the admin tool that reads data from pebble. [Util] update util to read results from pebble #7092

[vishalchangrani]

hi @j1010001, @zhangchiqing - is my understanding correct w.r.t outcomes of the milestones above:

1. Milestone 1 - chunkdata pack pruning on EN ✅

2. Milestone 2 - low risk data on EN, AN, VN moved to Pebble DB (in progress)

3. Milestone 3 -

   • Follower engine migrated to PebbleDB.
   • Verification node can run exclusively on PebbleDB.

4. Milestone 4 - EN and AN can run exclusively on PebbleDB

5. Milestone 5- SN and LN can run exclusively on PebbleDB

6. Milestone 6 - Data can be automatically pruned on EN, AN, VN.

7. Milestone 7 - Upgrade to Pebble 2.x