Skip to content

feat: Separate location for ancillary for cardano database v1 #2380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Mar 25, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f2625cf
feat: add `ancillary_locations` and `ancillary_size` to all snapshot …
Alenar Mar 17, 2025
e9d51c9
feat(aggregator): add `snapshot_all_completed_immutables` to Snapshotter
Alenar Mar 18, 2025
8e5ffaa
feat(aggregator): snapshot only completed immutables in cardano db v1
Alenar Mar 18, 2025
b184a6d
refactor(aggregator): compute archive name separatly from the archive…
Alenar Mar 18, 2025
6870438
test(aggregator): make `DumbUploader` store multiple uploads instead …
Alenar Mar 18, 2025
216ec00
feat(aggregator): snapshot & upload ancillary in immutable files full…
Alenar Mar 18, 2025
b26d0ec
refactor(aggregator): use `LocalUploader` for cardano database v1 upl…
Alenar Mar 18, 2025
222a52b
test(client-lib): reorganize integration tests infrastructure
Alenar Mar 18, 2025
bba6a3f
test(client-lib): only add last ledger file and remove volatile from …
Alenar Mar 18, 2025
baa6ec5
test(client-lib): update cardano db v1 test to ancillary archive split
Alenar Mar 18, 2025
0c62b47
fix(client-cli): incorrect genesis vkey in shared dev.json param file
Alenar Mar 19, 2025
5d7958c
feat(client-lib+cli): download ancillary archive if found in cardano …
Alenar Mar 19, 2025
a1e9c11
feat(client-cli): include ancillary size in prerequisite disk space c…
Alenar Mar 19, 2025
ac6fca6
chore: update changelog
Alenar Mar 19, 2025
aa16dd2
refactor: rephrasings, change ancillary archive naming by moving 'anc…
dlachaume Mar 21, 2025
57aba51
refactor(client-lib): rename `fake` module to `fake_aggregator`
dlachaume Mar 21, 2025
add188c
chore(client-lib): add comment to clarify underscore prefix usage for…
dlachaume Mar 21, 2025
36b94bf
feat(common+client-cli): add `compute_total_size` function for snapsh…
dlachaume Mar 24, 2025
ae7765c
chore: upgrade crate versions and `openapi.yaml` version
dlachaume Mar 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,13 @@ As a minor extension, we have adopted a slightly different versioning convention

- End support for **macOS x64 pre-built binaries** for the client CLI.

- Cardano database certification: creation of two separate archives, one for the immutable files and another for the ancillary files.

- **UNSTABLE** Implement a slave signer registration mode in the aggregator.

- **UNSTABLE** Cardano database incremental certification:

- Implement the client library for the the signed entity type `CardanoDatabase` (download and prove snapshot).
- Implement the client library for the signed entity type `CardanoDatabase` (download and prove snapshot).
- Implement the client CLI commands for the signed entity type `CardanoDatabase` (snapshot list, snapshot show and download commands).
- Implement an example crate for the signed entity type `CardanoDatabase`.
- Lighter ancillary archive by only including what's strictly necessary: the latest ledger file and the last immutable file trio.
Expand Down
10 changes: 5 additions & 5 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/client-cardano-database/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[package]
name = "client-cardano-database"
description = "Mithril client Cardano database example"
version = "0.1.27"
version = "0.1.28"
authors = ["[email protected]", "[email protected]"]
documentation = "https://mithril.network/doc"
edition = "2021"
Expand Down
44 changes: 39 additions & 5 deletions examples/client-cardano-database/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ async fn main() -> MithrilResult<()> {
pub struct IndicatifFeedbackReceiver {
progress_bar: MultiProgress,
download_pb: RwLock<Option<ProgressBar>>,
ancillary_download_pb: RwLock<Option<ProgressBar>>,
certificate_validation_pb: RwLock<Option<ProgressBar>>,
}

Expand All @@ -111,9 +112,19 @@ impl IndicatifFeedbackReceiver {
Self {
progress_bar: progress_bar.clone(),
download_pb: RwLock::new(None),
ancillary_download_pb: RwLock::new(None),
certificate_validation_pb: RwLock::new(None),
}
}

fn new_download_bytes_progress_bar(size: u64) -> ProgressBar {
let pb = ProgressBar::new(size);
pb.set_style(ProgressStyle::with_template("{spinner:.green} [{elapsed_precise}] [{wide_bar:.cyan/blue}] {bytes}/{total_bytes} ({eta})")
.unwrap()
.with_key("eta", |state: &ProgressState, w: &mut dyn Write| write!(w, "{:.1}s", state.eta().as_secs_f64()).unwrap())
.progress_chars("#>-"));
pb
}
}

#[async_trait]
Expand All @@ -126,11 +137,7 @@ impl FeedbackReceiver for IndicatifFeedbackReceiver {
size,
} => {
println!("Starting download of snapshot '{digest}'");
let pb = ProgressBar::new(size);
pb.set_style(ProgressStyle::with_template("{spinner:.green} [{elapsed_precise}] [{wide_bar:.cyan/blue}] {bytes}/{total_bytes} ({eta})")
.unwrap()
.with_key("eta", |state: &ProgressState, w: &mut dyn Write| write!(w, "{:.1}s", state.eta().as_secs_f64()).unwrap())
.progress_chars("#>-"));
let pb = Self::new_download_bytes_progress_bar(size);
self.progress_bar.add(pb.clone());
let mut download_pb = self.download_pb.write().await;
*download_pb = Some(pb);
Expand All @@ -152,6 +159,33 @@ impl FeedbackReceiver for IndicatifFeedbackReceiver {
}
*download_pb = None;
}
MithrilEvent::SnapshotAncillaryDownloadStarted {
download_id: _,
size,
} => {
println!("Starting download of ancillary snapshot");
let pb = Self::new_download_bytes_progress_bar(size);
self.progress_bar.add(pb.clone());
let mut ancillary_download_pb = self.ancillary_download_pb.write().await;
*ancillary_download_pb = Some(pb);
}
MithrilEvent::SnapshotAncillaryDownloadProgress {
download_id: _,
downloaded_bytes,
size: _,
} => {
let ancillary_download_pb = self.ancillary_download_pb.read().await;
if let Some(progress_bar) = ancillary_download_pb.as_ref() {
progress_bar.set_position(downloaded_bytes);
}
}
MithrilEvent::SnapshotAncillaryDownloadCompleted { download_id: _ } => {
let mut ancillary_download_pb = self.ancillary_download_pb.write().await;
if let Some(progress_bar) = ancillary_download_pb.as_ref() {
progress_bar.finish_with_message("Snapshot ancillary download completed");
}
*ancillary_download_pb = None;
}
MithrilEvent::CertificateChainValidationStarted {
certificate_chain_validation_id: _,
} => {
Expand Down
2 changes: 1 addition & 1 deletion mithril-aggregator/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "mithril-aggregator"
version = "0.7.23"
version = "0.7.24"
description = "A Mithril Aggregator server"
authors = { workspace = true }
edition = { workspace = true }
Expand Down
Loading