History Archive Support for State Archival #4610
Conversation
SirTyson
force-pushed
the
hot-archive-history
branch
2 times, most recently
from
c6d900c to
6ffde65
Compare
SirTyson
force-pushed
the
hot-archive-history
branch
from
6ffde65 to
39e74f8
Compare
SirTyson
force-pushed
the
hot-archive-history
branch
5 times, most recently
from
2f0f86a to
e258f51
Compare
SirTyson
force-pushed
the
hot-archive-history
branch
from
e258f51 to
69ca558
Compare
src/history/HistoryArchive.h
Outdated
| @@ -75,17 +82,37 @@ struct HistoryArchiveState | |||
| static constexpr size_t MAX_HISTORY_ARCHIVE_BUCKET_SIZE = | |||
| 1024ull * 1024ull * 1024ull * 100ull; // 100 GB | |||
|
|
|||
| static unsigned const HISTORY_ARCHIVE_STATE_VERSION; | |||
| static inline unsigned const HISTORY_ARCHIVE_STATE_VERSION_PRE_PROTOCOL_22 = | |||
There was a problem hiding this comment.
These names are a little odd. You have pre-22 and post-22 but no 22. Also I think 23 is the boundary? Maybe rename as ..VERSION_BEFORE_HOT_ARCHIVE and ...VERSION_WITH_HOT_ARCHIVE ?
| hash = mLiveBucketList->getHash(); | ||
| SHA256 hsh; | ||
| hsh.add(mLiveBucketList->getHash()); | ||
| hsh.add(mHotArchiveBucketList->getHash()); |
| @@ -1094,15 +1112,17 @@ BucketManager::startBackgroundEvictionScan(uint32_t ledgerSeq, | |||
| mSnapshotManager->copySearchableLiveBucketListSnapshot(); | |||
| auto const& sas = cfg.stateArchivalSettings(); | |||
|
|
|||
| using task_t = std::packaged_task<EvictionResultCandidates()>; | |||
| using task_t = | |||
| std::packaged_task<std::unique_ptr<EvictionResultCandidates>()>; | |||
There was a problem hiding this comment.
Curious: why the extra layer of indirection here with the unique_ptr?
There was a problem hiding this comment.
I threw a bunch of consts in EvictionResultCandidates, which removed the default copy constructor and copy assignment operator, which I need in BucketManager.cpp::1154. Eviction candidates can be pretty large, so we probably don't want to copy them anyway. I think the correct solution is to make EvictionResultCandidates and use unique_ptr everywhere.
src/bucket/BucketManager.cpp
Outdated
| @@ -1250,51 +1270,71 @@ BucketManager::assumeState(HistoryArchiveState const& has, | |||
| releaseAssert(threadIsMain()); | |||
| releaseAssertOrThrow(mConfig.MODE_ENABLES_BUCKETLIST); | |||
|
|
|||
| // TODO: Assume archival bucket state | |||
| // Dependency: HAS supports Hot Archive BucketList | |||
There was a problem hiding this comment.
I think this line of comment is residual from the removed TODO, and should also go?
src/bucket/BucketManager.cpp
Outdated
| auto b = getBucketByHashInternal(h, mSharedLiveBuckets); | ||
| if (!b) | ||
| // Returns filename, hash, and whether it's a hot archive bucket | ||
| auto loadFilenameAndHash = |
There was a problem hiding this comment.
Why make a lambda just to call it? There's only one caller.
SirTyson
force-pushed
the
hot-archive-history
branch
from
69ca558 to
c6aaaf2
Compare
SirTyson
force-pushed
the
hot-archive-history
branch
from
c6aaaf2 to
2504748
Compare
SirTyson
force-pushed
the
hot-archive-history
branch
from
2504748 to
cafb222
Compare
Description
Resolves #4593
This adds support for the Hot Archive BucketList in History Archives. This includes changes to publish, catchup, and assume state.
TODO: More robust p23 upgrade testing, specifically around upgrades that occur on or during checkpoint boundaries.Upgrade tests have been added, so unit tests should have complete coverage now.
Checklist
clang-formatv8.0.0 (viamake formator the Visual Studio extension)