[Access] Refactor storage collections for access node #7093

zhangchiqing · 2025-02-25T18:39:32Z

Working towards #6515

Review #7059 first.

This PR refactors the transactions and collection storage in access node to use the generic storage module.

codecov-commenter · 2025-03-05T19:46:04Z

Codecov Report

Attention: Patch coverage is 6.38298% with 352 lines in your changes missing coverage. Please review.

Project coverage is 41.16%. Comparing base (89dfb91) to head (a9c3140).

Files with missing lines	Patch %	Lines
storage/mock/events_reader.go	0.00%	94 Missing ⚠️
storage/mock/transaction_results_reader.go	0.00%	72 Missing ⚠️
storage/mock/execution_results_reader.go	0.00%	63 Missing ⚠️
consensus/hotstuff/mocks/persister_reader.go	0.00%	50 Missing ⚠️
storage/mock/commits_reader.go	0.00%	28 Missing ⚠️
cmd/access/node_builder/access_node_builder.go	0.00%	23 Missing ⚠️
storage/store/collections.go	55.88%	11 Missing and 4 partials ⚠️
storage/operation/collections.go	20.00%	4 Missing ⚠️
storage/store/transactions.go	0.00%	2 Missing ⚠️
cmd/observer/node_builder/observer_builder.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7093      +/-   ##
==========================================
- Coverage   41.24%   41.16%   -0.08%     
==========================================
  Files        2171     2176       +5     
  Lines      190061   190382     +321     
==========================================
- Hits        78391    78377      -14     
- Misses     105127   105451     +324     
- Partials     6543     6554      +11

Flag	Coverage Δ
unittests	`41.16% <6.38%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

peterargue · 2025-03-07T19:18:01Z

cmd/execution_builder.go

@@ -218,6 +217,9 @@ func (builder *ExecutionNodeBuilder) LoadComponentsAndModules() {
 		Module("blobservice peer manager dependencies", exeNode.LoadBlobservicePeerManagerDependencies).
 		Module("bootstrap", exeNode.LoadBootstrapper).
 		Module("register store", exeNode.LoadRegisterStore).
+		AdminCommand("get-transactions", func(conf *NodeConfig) commands.AdminCommand {


why move this?

Because the exeNode.collections was not initialized until exeNode.LoadCollections is called.

That said, this change should be in a different PR, let me check.

peterargue · 2025-03-07T19:22:23Z

storage/operation/collections.go

+// IndexCollectionPayload indexes the transactions within the collection payload
+// of a cluster block.


is this specific to collection cluster logic, or is this just indexing by blockID?

I'm wondering if we're overloading the codeIndexCollection to mean different things on ANs/ENs vs LNs

peterargue · 2025-03-07T19:25:46Z

storage/operation/collections_test.go

+		t.Run("Retrieve nonexistant", func(t *testing.T) {
+			var actual flow.LightCollection
+			err := operation.RetrieveCollection(db.Reader(), expected.ID(), &actual)
+			assert.Error(t, err)


Suggested change

assert.Error(t, err)

assert.ErrorIs(t, err, storage.ErrNotFound)

assert.Nil(t, actual)

peterargue · 2025-03-07T19:27:04Z

storage/operation/collections_test.go

+
+			var actual flow.LightCollection
+			err = operation.RetrieveCollection(db.Reader(), expected.ID(), &actual)
+			assert.Error(t, err)


can you assert the specific error here and wherever we have sentinels returned

peterargue · 2025-03-07T19:27:55Z

storage/operation/collections_test.go

+
+			_ = db.WithReaderBatchWriter(func(rw storage.ReaderBatchWriter) error {
+				err := operation.InsertCollection(rw.Writer(), &expected)
+				assert.Nil(t, err)


I think assert.NoError() communicates your intent more clearly

Suggested change

assert.Nil(t, err)

assert.NoError(t, err)

peterargue · 2025-03-07T19:29:18Z

storage/operation/transactions_test.go

+
+func TestTransactions(t *testing.T) {
+
+	dbtest.RunWithDB(t, func(t *testing.T, db storage.DB) {


can you add a not found test as well

peterargue · 2025-03-07T19:33:33Z

storage/store/collections.go

+}
+
+func NewCollections(db storage.DB, transactions *Transactions) *Collections {
+	c := &Collections{


what do you think about adding a cache? collections are commonly looked up on access nodes. totally fine to do later

peterargue · 2025-03-07T19:36:56Z

storage/store/collections.go

+
+func (c *Collections) Remove(colID flow.Identifier) error {
+	err := c.db.WithReaderBatchWriter(func(rw storage.ReaderBatchWriter) error {
+		return operation.RemoveCollection(rw.Writer(), colID)


should this also remove the index?

peterargue · 2025-03-07T19:39:18Z

storage/store/collections.go

+				// transaction is already indexed by a different collection, we should not index it again
+				// so that the access node will always return the same collection for a given transaction
+				// and return a consistent transaction result status.
+				continue


I think we should return an error here since LNs are supposed to prevent a tx from

appearing multiple times in the same collection

appearing in multiple collections

jordanschalm · 2025-03-12T17:42:29Z

cmd/access/node_builder/access_node_builder.go

+					notNil(builder.events),
+					notNil(builder.collections),
+					notNil(builder.transactions),
+					notNil(builder.lightTransactionResults),


What's the motivation for using notNil on only some inputs?

Shouldn't all the inputs be non-nil? And if an input is nil, won't the component using it quickly panic anyway?

jordanschalm · 2025-03-12T17:50:15Z

storage/operation/collections.go

+// RemoveCollectionTransactionIndices removes a collection id indexed by a transaction id
+// any error returned are exceptions
+func RemoveCollectionTransactionIndices(w storage.Writer, txID flow.Identifier) error {
+	return RemoveByKey(w, MakePrefix(codeIndexCollectionByTransaction, txID))
+}


Suggested change

// RemoveCollectionTransactionIndices removes a collection id indexed by a transaction id

// any error returned are exceptions

func RemoveCollectionTransactionIndices(w storage.Writer, txID flow.Identifier) error {

return RemoveByKey(w, MakePrefix(codeIndexCollectionByTransaction, txID))

}

// RemoveCollectionByTransactionIndex removes a collection id indexed by a transaction id,

// created by [UnsafeIndexCollectionByTransaction].

// Any error returned is an exception.

func RemoveCollectionByTransactionIndex(w storage.Writer, txID flow.Identifier) error {

return RemoveByKey(w, MakePrefix(codeIndexCollectionByTransaction, txID))

}

Naming to match the insert method for same index.

jordanschalm · 2025-03-12T17:55:10Z

storage/store/collections.go

+	if err != nil {
+		return nil, err
+	}
+


Suggested change

if err != nil {

return nil, err

}

The error is already checked above

jordanschalm · 2025-03-12T18:01:57Z

storage/operation/collections.go

@@ -52,3 +50,15 @@ func UnsafeIndexCollectionByTransaction(w storage.Writer, txID flow.Identifier,
 func RetrieveCollectionID(r storage.Reader, txID flow.Identifier, collectionID *flow.Identifier) error {


Suggested change

func RetrieveCollectionID(r storage.Reader, txID flow.Identifier, collectionID *flow.Identifier) error {

// LookupCollectionByTransaction looks up the collection indexed by the given transaction ID,

// which is the collection in which the given transaction was included.

// No errors are expected during normal operaion.

func LookupCollectionByTransaction(r storage.Reader, txID flow.Identifier, collectionID *flow.Identifier) error {

To match naming of other methods operating on the same index.

jordanschalm · 2025-03-12T18:08:10Z

storage/store/collections.go

+	err = c.db.WithReaderBatchWriter(func(rw storage.ReaderBatchWriter) error {
+		// remove transaction indices
+		for _, txID := range col.Transactions {
+			err = operation.RemoveCollectionTransactionIndices(rw.Writer(), txID)


Collections.Store above inserts transactions. Should Remove also remove the transactions?

storage/operation/collections.go

jordanschalm · 2025-03-12T18:16:20Z

storage/store/collections.go

-				}
-				continue
+			// the indexingByTx lock has ensured we are the only process indexing collection by transaction
+			err = operation.UnsafeIndexCollectionByTransaction(rw.Writer(), txID, collection.ID())


Suggested change

err = operation.UnsafeIndexCollectionByTransaction(rw.Writer(), txID, collection.ID())

err = operation.UnsafeIndexCollectionByTransaction(rw.Writer(), txID, cid)

Avoid re-computing the hash every loop iteration

jordanschalm · 2025-03-12T18:27:52Z

storage/store/collections.go

+		if err == nil {
+			// collection nodes have ensured that a transaction can only belong to one collection
+			// so if transaction is already indexed by a collection, check if it's the same collection.
+			// if not, return an error
+			if cid != differentColTxIsIn {
+				return fmt.Errorf("transaction %v is already indexed by a different collection %v", txID, differentColTxIsIn)
+			}


I think this is substantially changing the behaviour.

Previously, we would skip re-indexing TXID->COLLECTIONID, if any index entry for TXID already existed. Now we are throwing an exception.

The reason we specifically check for the case of the index already existing is to make sure that we don't overwrite the index with a different collection ID, so that the information served by the Access API is consistent (if not correct). Now this scenario will cause an exception and likely the node will enter a crash-loop. To match the previous behaviour, the case of err == nil on line 151 should be a no-op.

It is true that we don't currently expect this scenario to happen, absent a cluster consensus bug, but we have had such bugs in the past, and in the mature system we need to tolerate Byzantine clusters. So I don't think this should throw an exception.

zhangchiqing · 2025-03-13T19:09:34Z

cmd/access/node_builder/access_node_builder.go

@@ -575,6 +577,15 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
 		AdminCommand("read-execution-data", func(config *cmd.NodeConfig) commands.AdminCommand {
 			return stateSyncCommands.NewReadExecutionDataCommand(builder.ExecutionDataStore)
 		}).
+		Module("transactions and collections storage", func(node *cmd.NodeConfig) error {
+			// TODO: needs to be wrapped with ChainedCollections module, otherwise once we switch


Link the issue as TODO here #6523 (comment) .

Will be addressed separately. We can review and approve this PR, but not merge until the TODO is completed.

cc @fxamacker

zhangchiqing added 7 commits February 18, 2025 20:15

refactor collections

eca55cd

update execution builder to use store.collections

fde54c9

refactor collection node to read collections

a109af6

revert collection changes

f326355

adding lock to protect indexing collections

435b991

add concurrent test

6cd812c

refactor access node's transactions and collections storage module

208fb81

zhangchiqing mentioned this pull request Feb 25, 2025

Replacing Badger with Pebble DB #6515

Open

23 tasks

Merge branch 'master' into leo/refactor-storage-collections-for-an

a5f43fb

zhangchiqing force-pushed the leo/refactor-storage-collections-for-an branch from d72e501 to a5f43fb Compare March 6, 2025 17:09

zhangchiqing added 2 commits March 6, 2025 20:52

refactor using protocoldb

705399d

making collections and transactions private

b70d475

peterargue reviewed Mar 7, 2025

View reviewed changes

zhangchiqing added 2 commits March 9, 2025 17:45

Merge branch 'master' into leo/refactor-storage-collections-for-an

5013f21

fix lint

3287ed1

j1010001 mentioned this pull request Mar 10, 2025

Badger -> Pebble DB M2 - DB access refactoring for low-risk data (AN, EN, VN) #6527

Open

18 tasks

zhangchiqing added 3 commits March 10, 2025 16:56

update collections.Remove

07288a9

remove collections.StoreLight

f38988b

index collections by txs

1a15b10

jordanschalm reviewed Mar 12, 2025

View reviewed changes

zhangchiqing added 5 commits March 13, 2025 09:33

Merge branch 'master' into leo/refactor-storage-collections-for-an

e4bf8e5

update collections operations methods

99bbf77

refactor collection store and remove methods

14469d0

refactor StoreLightAndIndexByTransaction

db502ad

remove unused collections operations

35692d7

zhangchiqing force-pushed the leo/refactor-storage-collections-for-an branch from 8bdd882 to 35692d7 Compare March 13, 2025 18:06

zhangchiqing added 3 commits March 13, 2025 11:15

update mocks

127859b

add UpertCollection methods

cbb01aa

refactor StoreLightAndIndexByTransaction

311b44b

add TODO

a9c3140

zhangchiqing commented Mar 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Access] Refactor storage collections for access node #7093

[Access] Refactor storage collections for access node #7093

zhangchiqing commented Feb 25, 2025

codecov-commenter commented Mar 5, 2025 •

edited

Loading

peterargue Mar 7, 2025

zhangchiqing Mar 7, 2025

peterargue Mar 7, 2025

peterargue Mar 7, 2025

peterargue Mar 7, 2025

peterargue Mar 7, 2025

peterargue Mar 7, 2025

peterargue Mar 7, 2025

peterargue Mar 7, 2025

peterargue Mar 7, 2025

jordanschalm Mar 12, 2025

jordanschalm Mar 12, 2025

jordanschalm Mar 12, 2025

jordanschalm Mar 12, 2025

jordanschalm Mar 12, 2025

jordanschalm Mar 12, 2025

jordanschalm Mar 12, 2025

zhangchiqing Mar 13, 2025

		// IndexCollectionPayload indexes the transactions within the collection payload
		// of a cluster block.

	assert.Error(t, err)
	assert.ErrorIs(t, err, storage.ErrNotFound)
	assert.Nil(t, actual)


		func TestTransactions(t *testing.T) {

		dbtest.RunWithDB(t, func(t *testing.T, db storage.DB) {

		@@ -52,3 +50,15 @@ func UnsafeIndexCollectionByTransaction(w storage.Writer, txID flow.Identifier,
		func RetrieveCollectionID(r storage.Reader, txID flow.Identifier, collectionID *flow.Identifier) error {

-func RetrieveCollectionID(r storage.Reader, txID flow.Identifier, collectionID *flow.Identifier) error {
+// LookupCollectionByTransaction looks up the collection indexed by the given transaction ID,
+// which is the collection in which the given transaction was included.
+// No errors are expected during normal operaion.
+func LookupCollectionByTransaction(r storage.Reader, txID flow.Identifier, collectionID *flow.Identifier) error {

	err = operation.UnsafeIndexCollectionByTransaction(rw.Writer(), txID, collection.ID())
	err = operation.UnsafeIndexCollectionByTransaction(rw.Writer(), txID, cid)

[Access] Refactor storage collections for access node #7093

Are you sure you want to change the base?

[Access] Refactor storage collections for access node #7093

Conversation

zhangchiqing commented Feb 25, 2025

codecov-commenter commented Mar 5, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Mar 5, 2025 •

edited

Loading