Parallel bl db

[SirTyson]

Description

Builds on top of #4172 to enable parallel loads for BucketListDB.

This is an experimental draft PR that makes BucketListDB thread safe. The interface is as follows:

Each thread should call BucketManager::getSearchableBucketListSnapshot() to initialize their local snapshot. All loads from the thread should go through this snapshot object. After initialization, the BucketListSnapshot object automatically stays in sync with the current BucketList.

This change has a very simple, single threaded unit test. I've also run a watcher node successfully, but have yet to do a significant multi-threaded test.

Checklist

• Reviewed the contributing document
• Rebased on top of master (no merge commits)
• Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
• Compiles
• Ran all tests
• If change impacts performance, include supporting evidence per the performance document

[SirTyson]

There's been a few iterations on this, so here's an overview of the thread safe structure.

We've added BucketListSnapshots. This is a light weight wrapper around the BucketList which maintains pointers to all Buckets and an independent file stream for each Bucket. For thread safe BucketListDB reads, each thread has it's own BucketListSnapshot. These snapshots all share access to the same in-memory BucketIndex because indexes are guaranteed to always be immutable. Because each snapshot maintains it's own set of file streams, there are no conflicts at the file level.

Currently, most core subsystems assume single thread access only and are not thread safe. Due to this assumption, background threads should have no access to most core subsystems (BucketManager, BucketList, LedgerManager, Application, etc.). In order to maintain this separation, we've introduced BucketSnapshotManager. This class serves as the boundary between the non-threadsafe singletons and background threads that require thread safety.

The BucketSnapshotManager maintains a canonical BucketListSnapshot protected by a snapshot mutex. Whenever the main thread updates the BucketList (addBatch and assumeState), the main thread will update the BucketSnapshotManager snapshot. Background threads will then request a copy of the snapshot from the manager. Additionally, whenever a background thread performs a load from their snapshot, the background thread will check their snapshot against the current canonical snapshot from the BucketSnapshotManager. If the background thread snapshot is out of date, it will be updated, such that loads are always performed on the most up-to-date snapshot. This approach requires minimal concurrency overhead, where the main thread must only acquire a lock to update the snapshot.

[marta-lokhova]

Thanks for the PR, very nice changes! (that also improve bucketlist interface and separation of responsibilities). Please squash the commits, and we can merge.

[marta-lokhova]

Looks like there are some outdated files in the beginning of the commit history, and some small fixes at the end, please squash the commits (commit history doesn't need to be super precise, just make sure to remove obsolete interface changes etc)

[marta-lokhova]

r+ 916bb73