feat(firewood/ffi): add FjallStore

[RodrigoVillar]

This PR adds FjallStore, an implementation of RootStore that allows for persistent storage of revisions (in contrast to MockStore, which keeps everything in-memory). The main changes of this PR are as follows:

• Adds FjallStore implementation
• Extends the DbConfig to take in a path to where FjallStore will be stored
• Extends the FFI database config to enable archival support
• Adds FjallStore tests in both Rust and Golang

[RodrigoVillar]

Note to reviewers: I've marked the PR as DO NOT MERGE since I'm working on getting benchmark results for Firewood + FjallStore when compared to LevelDB (and we should only merge once we have benchmark results). However, I figured we can get the code review process going since benchmarking can take place async.

[rkuris]

I'm not sure I love changing the reaping logic to fix this issue. The commit should have the ability to reap any revision at any time.

I think there are a few possible approaches here:

1 - Part of reaping is adding it to the RootStore
2 - Every commit adds to RootStore so entries are both in the RootStore and in the revision manager (preferred).

[rkuris]

Note to reviewers: I've marked the PR as DO NOT MERGE since I'm working on getting benchmark results for Firewood + FjallStore when compared to LevelDB (and we should only merge once we have benchmark results). However, I figured we can get the code review process going since benchmarking can take place async.

I think we determined that this PR does not affect performance, so I think this is mergeable at this point.

[rkuris]

I'm not sure I love the separation of concerns here, but I could be convinced it's okay.

Perhaps the Rootstore interface should be that it returns nodestores themselves, and then the RootStore can deal with its own caching (only in the real implementation).

I think that cleans up the manager itself to focus on doing what it's supposed to be doing (tracking nodestores) instead of constructing them.

[RodrigoVillar]

@rkuis I'm definitely on-board with having RootStore own constructing and maintaining its own nodestores. With our current code, refactoring would fail due to the following circular dependency:

stateDiagram-v2
    s1: Revision Manager
    s2: RootStore

    s1 --> s2: Wants historical revision X
    s2 --> s1: Wants latest revision

Loading

One way to fix this would be to pass in the latest revision to get():

fn get(
        &self,
        hash: &TrieHash,
        latest_revision: CommittedRevision,
    ) -> Result<Option<LinearAddress>, Box<dyn std::error::Error + Send + Sync>>;

However, passing in the latest revision seems odd (we expect RootStore to handle it's own nodestores). I think the better alternative would be to delegate the historical and by_hash fields of the RevisionManager into their own DS and have that be shared by both the RevisionManager and RootStore:

pub struct InMemoryRevisions {
    historical: VecDeque<CommittedRevision>,
    by_hash: HashMap<TrieHash, CommittedRevision>
}

Refactoring our code to use this DS would lead to the following dependency graph:

stateDiagram-v2
    s1: Revision Manager
    s2: RootStore
    s3: InMemoryRevisions

    s1 --> s3: Maintains latest revisions
    s1 --> s2: Wants historical revision
    s2 --> s3: Wants latest revision

Loading

[rkuris]

pub struct InMemoryRevisions {
    historical: VecDeque<CommittedRevision>,
    by_hash: HashMap<TrieHash, CommittedRevision>
}

Let's enter a new issue and do this. We should also have methods on this struct to manage and fetch from the list.

[rkuris]

Approved on creation of a followup for simplifying management of in memory revisions.

[RodrigoVillar]

Created #1470 which will track this.