feat: certstore snapshot export #1032

hanabi1224 · 2025-06-26T14:09:00Z

~~This is a draft implementation of F3 snapshot export functionality in the format proposed in filecoin-project/FIPs#1169 to collect feedbacks~~

This PR implements

exporting and importing a certstore snapshot in FRC-0108 format
unit test

Tried to export snapshots on mainnet(@instance 176187) and calibnet(@instance 483660), below are some basic stats

➜  tmp ls -ahl f3*
-rw-r--r-- 1 me me 299M Jul  8 17:27 f3_snap_calibnet.bin
-rw-r--r-- 1 me me 113M Jul  8 17:27 f3_snap_calibnet.bin.zst
-rw-r--r-- 1 me me 173M Jul  8 17:26 f3_snap_mainnet.bin
-rw-r--r-- 1 me me  88M Jul  8 17:26 f3_snap_mainnet.bin.zst

codecov · 2025-06-26T14:24:28Z

Codecov Report

Attention: Patch coverage is 44.91525% with 65 lines in your changes missing coverage. Please review.

Project coverage is 64.56%. Comparing base (7f2287c) to head (246fdf9).

Files with missing lines	Patch %	Lines
certstore/snapshot.go	46.90%	40 Missing and 20 partials ⚠️
f3.go	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1032      +/-   ##
==========================================
- Coverage   64.96%   64.56%   -0.41%     
==========================================
  Files          80       81       +1     
  Lines        9751     9869     +118     
==========================================
+ Hits         6335     6372      +37     
- Misses       2918     2975      +57     
- Partials      498      522      +24

Files with missing lines	Coverage Δ
f3.go	`54.34% <0.00%> (+0.57%)`	⬆️
certstore/snapshot.go	`46.90% <46.90%> (ø)`

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

masih

I think we need to pin down high level abstractions more concretely. Left a bunch of suggestions and thank you for pushing this work forward 🍻

masih · 2025-06-26T14:19:28Z

certstore/snapshot.go

+
+	"github.com/filecoin-project/go-f3/gpbft"
+	"github.com/ipfs/go-datastore"
+	xerrors "golang.org/x/xerrors"


This repo avoids using xerrors in favour of standard Go SDK packages.

certstore/snapshot.go

masih · 2025-06-27T09:42:25Z

certstore/snapshot.go

+
+var ErrlatestCertificateNil = errors.New("latest certificate is not available")
+
+// Exports an F3 snapshot that includes the finality certificate chain until the current `latestCertificate`.


By convention start godoc with the function name. Ditto in other places.

masih · 2025-06-27T09:43:03Z

certstore/snapshot.go

+	return cs.ExportSnapshot(ctx, cs.latestCertificate.GPBFTInstance, writer)
+}
+
+// Exports an F3 snapshot that includes the finality certificate chain until the specified `lastInstance`.


Clarify what the from instance is.

masih · 2025-06-27T09:43:40Z

certstore/snapshot.go

+}
+
+// Exports an F3 snapshot that includes the finality certificate chain until the specified `lastInstance`.
+func (cs *Store) ExportSnapshot(ctx context.Context, lastInstance uint64, writer io.Writer) error {


Simply call latestInstance, to? There is nothing in the logic that mandates "latest instance".

masih · 2025-06-27T09:48:06Z

certstore/snapshot.go

+	InitialPowerTable gpbft.PowerEntries
+}
+
+func (h *SnapshotHeader) WriteToSnapshot(writer io.Writer) error {


I recommend adhering to the existing SDK interfaces like io.WriterTo.

masih · 2025-06-27T09:48:41Z

certstore/snapshot.go

+}
+
+// Writes CBOR-encoded header or data block with a varint-encoded length prefix
+func writeSnapshotCborEncodedBlock(writer io.Writer, block MarshalCBOR) error {


This seems over refactored, i.e. called only from one place.

masih · 2025-06-27T09:52:46Z

certstore/snapshot.go

+	return nil
+}
+
+type MarshalCBOR interface {


Why is this type defined? Consider using existing types from cborgen?

Fixed. (Only found Marshaler in github.com/filecoin-project/go-state-types/cbor)

masih · 2025-06-27T09:53:58Z

certstore/snapshot.go

+}
+
+// Exports an F3 snapshot that includes the finality certificate chain until the specified `lastInstance`.
+func (cs *Store) ExportSnapshot(ctx context.Context, lastInstance uint64, writer io.Writer) error {


Maybe makes sense to define an Exporter type that implements io.WriterTo instead, which takes Store and any other to-from instance params? also see other comment.

io.WriterTo does not take context.Context for cancellation given the export/import could take some time

masih · 2025-06-27T10:01:11Z

certstore/snapshot.go

+var ErrlatestCertificateNil = errors.New("latest certificate is not available")
+
+// Exports an F3 snapshot that includes the finality certificate chain until the current `latestCertificate`.
+func (cs *Store) ExportLatestSnapshot(ctx context.Context, writer io.Writer) error {


The term Snapshot doesn't carry enough of a weight within the context of go-f3. What we really mean by it at this stage is simply: export certificates.

You could make it mean something: define a type Snapshot that is either produced by the Store, or takes the Store, and implements io.WriterTo, io.ReaderFrom etc.

Or avoid the term entirely and instead make the store simply an Iterator of certs and separate IO ops elsewhere.

I would probably go with the first approach but there is not enough implementation in this PR for me to fully wrap my head around how you are thinking of approaching the problem.

So please feel free to ignore the recommendations if they happen to not make sense as the work progresses forward.

Added FRC link to the comment to define Snapshot. As I understand, a Snapshot struct is an in-memory representation of the snapshot which is not ideal for implementing a streaming export with a low RAM usage. Implementing io.WriterTo would not allow cancellation with context.Context.

hanabi1224 · 2025-07-14T13:13:11Z

Hey @BigLep @Kubuxu this one is ready for review, please let me know your feedback, especially if the export / import APIs can be easily integrated into Lotus. Thanks!

Kubuxu · 2025-07-17T15:00:10Z

certstore/snapshot.go

+// ExportSnapshot exports an F3 snapshot that includes the finality certificate chain from the `Store.firstInstance` to the specified `lastInstance`.
+//
+// Checkout the snapshot format specification at <https://github.com/filecoin-project/FIPs/blob/master/FRCs/frc-0108.md>
+func (cs *Store) ExportSnapshot(ctx context.Context, latestInstance uint64, writer io.Writer) (cid.Cid, *SnapshotHeader, error) {


I have questions about how to use it, especially with go-car, because IDK if we can provide the header with the CID afterwards.

Here is a draft Forest implementation of v2 snapshot export. The flow is

exporting F3 snapshot to a tmp file

exporting v2 Filecoin snapshot with F3 snapshot CID and tmp file

cleanup the tmp file

go-car has ReplaceRootsInFile which will update the header after write as long as you start off with a dummy root CID that's the same length: https://pkg.go.dev/github.com/ipld/go-car/v2#ReplaceRootsInFile

BigLep · 2025-07-22T02:40:46Z

What are the next steps? Is this needing full review comments from @Kubuxu ?

BigLep · 2025-07-22T16:57:05Z

Per comment in 2025-07-22 implementer working group, it looks like this is ready for maintainer review.

@hanabi1224 : will you look at the failing checks?

feat: certstore snapshot export

740fe1a

github-project-automation bot added this to F3 Jun 26, 2025

github-project-automation bot moved this to Todo in F3 Jun 26, 2025

BigLep moved this from Todo to In progress in F3 Jun 26, 2025

BigLep requested a review from Kubuxu June 26, 2025 15:23

masih reviewed Jun 27, 2025

View reviewed changes

hanabi1224 mentioned this pull request Jul 1, 2025

frc-0108: f3-augmented-snapshot filecoin-project/FIPs#1169

Merged

hanabi1224 added 3 commits July 8, 2025 16:58

Merge remote-tracking branch 'origin/main' into hm/snapshot-export

896dde4

resolve comments

9ac1e74

Merge remote-tracking branch 'origin/main' into hm/snapshot-export

6ac2aca

hanabi1224 force-pushed the hm/snapshot-export branch from fe81652 to e1507fa Compare July 14, 2025 11:42

Merge branch 'main' into hm/snapshot-export

0ebf227

hanabi1224 force-pushed the hm/snapshot-export branch from e1507fa to 0ebf227 Compare July 14, 2025 11:50

roundtrip tests

415f470

hanabi1224 marked this pull request as ready for review July 14, 2025 13:11

export snapshot CID

3cde113

BigLep moved this from In progress to In review in F3 Jul 15, 2025

BigLep added this to FilOz Jul 15, 2025

github-project-automation bot moved this to 📌 Triage in FilOz Jul 15, 2025

BigLep moved this from 📌 Triage to 🔎 Awaiting Review in FilOz Jul 15, 2025

return snap header in export func

7cc7b77

Kubuxu reviewed Jul 17, 2025

View reviewed changes

BigLep requested a review from Kubuxu July 22, 2025 16:55

Merge branch 'main' into hm/snapshot-export

246fdf9


		var ErrlatestCertificateNil = errors.New("latest certificate is not available")

		// Exports an F3 snapshot that includes the finality certificate chain until the current `latestCertificate`.

feat: certstore snapshot export #1032

Are you sure you want to change the base?

feat: certstore snapshot export #1032

Conversation

hanabi1224 commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

masih left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hanabi1224 Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masih Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masih Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hanabi1224 Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hanabi1224 commented Jul 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BigLep commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BigLep commented Jul 22, 2025

Uh oh!

Uh oh!

hanabi1224 commented Jun 26, 2025 •

edited

Loading

codecov bot commented Jun 26, 2025 •

edited

Loading

hanabi1224 Jul 14, 2025 •

edited

Loading

masih Jun 27, 2025 •

edited

Loading

masih Jun 27, 2025 •

edited

Loading

hanabi1224 Jul 14, 2025 •

edited

Loading

BigLep commented Jul 22, 2025 •

edited

Loading