Skip to content

feat: external block provider for genesis sync #26

@tac0turtle

Description

@tac0turtle

Summary

Nodes running in pruning mode (default) discard old blocks to bound disk growth. To support syncing new nodes from genesis without requiring every node to be a full archive node, we need an external block serving service.

Motivation

  • Block archive storage now prunes by default (retention window of ~100k blocks)
  • Chain-index (SQLite) serves RPC/explorer queries independently of the block archive
  • Only seed/archive nodes need full block history
  • New nodes joining the network need a way to sync from genesis without requiring every peer to store all blocks

Design

An external block provider service that:

  1. Stores all blocks from genesis (potentially using commonware's immutable::Archive with freezer for minimal memory + compression)
  2. Serves block data over a well-defined API (HTTP/gRPC) to syncing nodes
  3. Can be operated independently of the P2P network -- e.g., a centralized service, cloud storage, or a dedicated archive node

Sync flow

New Node                    Block Provider
  |                              |
  |-- request blocks [0, N) --->|
  |<--- stream ArchivedBlock ---|
  |                              |
  |-- apply blocks to STF ---   |
  |-- build chain-index ---     |
  |                              |
  |-- join P2P, continue from  |
  |   latest block              |

Considerations

  • The provider does not need to be trusted for correctness -- the node validates state roots against the chain's state transition function
  • Provider could be backed by object storage (S3/R2) serving pre-compressed block segments
  • Protocol should support range requests and resumable sync
  • Consider using commonware's immutable::Archive (freezer) as the backing store for the provider -- it's designed for exactly this workload (write-once, read-many, minimal memory, compressed)

Acceptance Criteria

  • Define block provider API (gRPC service definition or HTTP endpoints)
  • Implement block provider server backed by full block archive
  • Implement client-side sync that fetches blocks from provider and applies them
  • Support resumable sync (track last synced block)
  • Node config option to specify block provider URL for initial sync

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions