Skip to content

Commit a7d98ba

Browse files
bwbushbwbush
authored
Analysis of simulations on large, realistic networks (#252)
* Refactored simulation workflow for realistic network * Reran 100-node simulations to 600 seconds * Analysis of rust simulation results for realistic network * Updated logbook
1 parent a44d881 commit a7d98ba

35 files changed

+67686
-209
lines changed

‎Logbook.md

+6
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,12 @@
2121

2222
- Initial trace verifier for Short Leios simulation traces in `leios-trace-verifier`
2323

24+
### Analysis of mainnet-scale simulation
25+
26+
The first analysis results for Leios on a mainnet-scale network simulation, using the Rust simulator, are available in [analysis/sims/2025w11xl/analysis.ipynb](analysis/sims/2025w11xl/analysis.ipynb). It is interesting to see that the 3000-node mainnet-scale network transports IBs faster than the artificial 100-node network that was previously analyzed: likely this is because the larger network has long-range "shortcut" edges that speed transport.
27+
28+
![In-flight time for IBs](analysis/sims/2025w11xl/plots/elapsed-IB-rust.png)
29+
2430
### Performance analysis of sharding
2531

2632
The Jupyter notebook [analysis/shard-performance.ipynb](analysis/shard-performance.ipynb) presents computations that elucidate the relation between the fraction of shards without an IB vs the expected number of extra IBs for the shard, for the simplest sharding scheme.

‎analysis/sims/2025w11/analysis.ipynb

+209-208
Large diffs are not rendered by default.

‎analysis/sims/2025w11/export-results.sh

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/usr/bin/env bash
2+
3+
source env.sh
4+
5+
mongoexport --host="$HOST" \
6+
--db="$DB" \
7+
--collection=receipts \
8+
--type=csv \
9+
--fields='elapsed,ib-body-avg-size-bytes,ib-generation-probability,item,kind,label,leios-stage-length-slots,network,producer,received,recipient,sent,simulator' \
10+
| gzip -9vc > results/receipts.csv
11+
12+
mongoexport --host="$HOST" \
13+
--db="$DB" \
14+
--collection=cpus \
15+
--type=csv \
16+
--fields 'duration,ib-body-avg-size-bytes,ib-generation-probability,label,leios-stage-length-slots,network,node,simulator,slot,task' \
17+
| gzip -9vc > results/cpus.csv
18+
19+
mongoexport --host="$HOST" \
20+
--db="$DB" \
21+
--collection=ibgen \
22+
--type=csv \
23+
--fields='eb-count,eb-first,eb-last,ib,ib-body-avg-size-bytes,ib-generation-probability,label,leios-stage-length-slots,network,node,simulator,time,size' \
24+
| gzip -9vc > results/ibgen.csv
25+
26+
mongoexport --host="$HOST" \
27+
--db="$DB" \
28+
--collection=ebgen \
29+
--type=csv \
30+
--fields='eb,ib-body-avg-size-bytes,ib-count,ib-generation-probability,label,leios-stage-length-slots,network,node,rb-count,rb-first,rb-last,simulator,size,time' \
31+
| gzip -9vc > results/ebgen.csv
32+
33+
mongoexport --host="$HOST" \
34+
--db="$DB" \
35+
--collection=rbgen \
36+
--type=csv \
37+
--fields='eb-count,ib-body-avg-size-bytes,ib-generation-probability,label,leios-stage-length-slots,network,rb,simulator,time,size' \
38+
| gzip -9vc > results/rbgen.csv

‎analysis/sims/2025w11/queries/haskell/complete.js

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
db.haskell.aggregate(
2+
[
3+
{
4+
$match: {
5+
"scenario.label": "default"
6+
}
7+
},
8+
{
9+
$group: {
10+
_id: "$scenario",
11+
time_s: {
12+
$max: "$time_s"
13+
}
14+
}
15+
},
16+
{
17+
$match: {
18+
time_s: {
19+
$gt: 599
20+
}
21+
}
22+
},
23+
{
24+
$replaceWith: {
25+
$mergeObjects: ["$$ROOT", "$_id"]
26+
}
27+
},
28+
{
29+
$unset: ["_id", "network", "label"]
30+
},
31+
{
32+
$sort: {
33+
"leios-stage-length-slots": 1,
34+
"ib-generation-probability": 1,
35+
"ib-body-avg-size-bytes": 1
36+
}
37+
},
38+
{
39+
$out: "haskell-complete"
40+
}
41+
]
42+
)
43+
44+
db["haskell-complete"].find()

‎analysis/sims/2025w11/queries/rust/complete.js

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
db.rust.aggregate(
2+
[
3+
{
4+
$match: {
5+
"scenario.label": "default"
6+
}
7+
},
8+
{
9+
$group: {
10+
_id: "$scenario",
11+
time_s: {
12+
$max: "$time_s"
13+
}
14+
}
15+
},
16+
{
17+
$match: {
18+
time_s: {
19+
$gt: 599
20+
}
21+
}
22+
},
23+
{
24+
$replaceWith: {
25+
$mergeObjects: ["$$ROOT", "$_id"]
26+
}
27+
},
28+
{
29+
$unset: ["_id", "network", "label"]
30+
},
31+
{
32+
$sort: {
33+
"leios-stage-length-slots": 1,
34+
"ib-generation-probability": 1,
35+
"ib-body-avg-size-bytes": 1
36+
}
37+
},
38+
{
39+
$out: "rust-complete"
40+
}
41+
]
42+
)
43+
44+
db["rust-complete"].find()

‎analysis/sims/2025w11/run-experiment.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ function mkScenario() {
2727
source env.sh
2828

2929
NETWORK=100
30-
MAX_SLOT=150
30+
MAX_SLOT=600
3131

3232

3333
#### CPU vs no CPU

‎analysis/sims/2025w11/run-queries.sh

100644100755
File mode changed.

‎analysis/sims/2025w11xl/analysis.ipynb

+1,788
Large diffs are not rendered by default.

‎analysis/sims/2025w11xl/build-experiments.sh

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/usr/bin/env bash
2+
3+
set -ev
4+
5+
LABEL=default
6+
NETWORK=realistic
7+
MAX_SLOT=600
8+
9+
for SIMULATOR in haskell rust
10+
do
11+
for IB_RATE in 0.3 3.0 30.0
12+
do
13+
for IB_SIZE in 32768 98304 163840
14+
do
15+
for EB_RATE in 1.5
16+
do
17+
for STAGE_LENGTH in 60
18+
do
19+
./make-experiment.sh $SIMULATOR $MAX_SLOT $LABEL $NETWORK $IB_RATE $IB_SIZE $EB_RATE $STAGE_LENGTH
20+
done
21+
done
22+
done
23+
done
24+
done

‎analysis/sims/2025w11xl/config.default.yaml

+188
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# yaml-language-server: $schema=./config.schema.json
2+
3+
################################################################################
4+
# Simulation Configuration File
5+
################################################################################
6+
#
7+
# This file contains the default configuration for running Leios simulations in
8+
# the Haskell simulation (`simulation/`) and the Rust simulation (`sim-rs/`).
9+
#
10+
################################################################################
11+
# Simulation Configuration
12+
################################################################################
13+
14+
relay-strategy: "request-from-first"
15+
tcp-congestion-control: true
16+
multiplex-mini-protocols: true
17+
simulate-transactions: false
18+
treat-blocks-as-full: false
19+
20+
################################################################################
21+
# Leios Protocol Configuration
22+
################################################################################
23+
24+
leios-stage-length-slots: 20
25+
leios-stage-active-voting-slots: 1
26+
leios-vote-send-recv-stages: false
27+
28+
################################################################################
29+
# Transaction Configuration
30+
################################################################################
31+
32+
tx-generation-distribution:
33+
distribution: exp
34+
lambda: 0.85
35+
scale: 1000.0
36+
tx-size-bytes-distribution:
37+
distribution: log-normal
38+
mu: 6.833
39+
sigma: 1.127
40+
tx-validation-cpu-time-ms: 1.5
41+
tx-max-size-bytes: 16384
42+
43+
################################################################################
44+
# Ranking Block Configuration
45+
################################################################################
46+
47+
# 1/leios-stage-length-slots, targeting one RB per pipeline.
48+
# Also 20s is current rate of praos blocks.
49+
rb-generation-probability: 5.0e-2
50+
# Eng. team targets 1kB as worst case upper bound.
51+
# Actual size fairly close.
52+
rb-head-size-bytes: 1024
53+
rb-body-max-size-bytes: 90112
54+
# Note: certificate generation/validation is not included in the
55+
# timings here, see cert-* fields.
56+
rb-generation-cpu-time-ms: 1.0
57+
rb-head-validation-cpu-time-ms: 1.0
58+
59+
# On average, no Txs directly embedded in blocks.
60+
rb-body-legacy-praos-payload-avg-size-bytes: 0
61+
rb-body-legacy-praos-payload-validation-cpu-time-ms-constant: 50.0
62+
# the -per-byte component is meant to be using size as a (bad)
63+
# proxy for the complexity of the Txs included.
64+
rb-body-legacy-praos-payload-validation-cpu-time-ms-per-byte: 0.0005
65+
66+
################################################################################
67+
# Input Block Configuration
68+
################################################################################
69+
70+
ib-generation-probability: 5.0
71+
ib-shards: 1
72+
73+
# ProducerId 32
74+
# SlotNo 64
75+
# VRF proof 80
76+
# Body hash 32
77+
# RB Ref 32
78+
# Signature 64
79+
# Total 304
80+
#
81+
# NOTE: using a KES Signature (like for Praos headers)
82+
# would instead more than double the total to 668.
83+
# And even 828 including Op Cert.
84+
ib-head-size-bytes: 304
85+
# 98KB to optimize for 3 TCP round trips
86+
ib-body-avg-size-bytes: 98304
87+
ib-body-max-size-bytes: 327680
88+
# Here we also use praos blocks as ballpark estimate.
89+
# Sec 2.3 Forging, of the benchmark cluster report, lists
90+
# * Slot start to announced: 0.12975s
91+
ib-generation-cpu-time-ms: 130.0
92+
ib-head-validation-cpu-time-ms: 1.0
93+
ib-body-validation-cpu-time-ms-constant: 50.0
94+
ib-body-validation-cpu-time-ms-per-byte: 0.0005
95+
ib-diffusion-strategy: "freshest-first"
96+
97+
# Haskell prototype relay mini-protocol parameters.
98+
ib-diffusion-max-bodies-to-request: 1
99+
ib-diffusion-max-headers-to-request: 100
100+
ib-diffusion-max-window-size: 100
101+
102+
################################################################################
103+
# Endorsement Block Configuration
104+
################################################################################
105+
106+
# We want one per pipeline, but not too many.
107+
eb-generation-probability: 1.5
108+
# ProducerId 32
109+
# SlotNo 64
110+
# VRF proof 80
111+
# Signature 64
112+
# Total 240
113+
#
114+
# See Note about signatures on ib-head-size-bytes.
115+
eb-size-bytes-constant: 240
116+
# IB hash
117+
eb-size-bytes-per-ib: 32
118+
# Collecting the IBs to reference and cryptography are the main tasks.
119+
# A comparable task is maybe mempool snapshotting.
120+
# Sec 2.3 Forging, of the benchmark cluster report, lists
121+
# * Mempool snapshotting: 0.07252s
122+
# 75ms then seems a generous estimate for eb generation.
123+
eb-generation-cpu-time-ms: 75.0
124+
# Validating signature and vrf proof, as in other headers.
125+
eb-validation-cpu-time-ms: 1.0
126+
127+
eb-diffusion-strategy: "peer-order"
128+
129+
# Haskell prototype relay mini-protocol parameters.
130+
eb-diffusion-max-bodies-to-request: 1
131+
eb-diffusion-max-headers-to-request: 100
132+
eb-diffusion-max-window-size: 100
133+
134+
################################################################################
135+
# Vote Configuration
136+
################################################################################
137+
138+
# Cryptography related values taken from [vote-spec](crypto-benchmarks.rs/Specification.md)
139+
# using weighted averages of 80% persistent and 20% non-persistent.
140+
141+
# vote-spec#Committe and quorum size
142+
#
143+
# Note: this is used as the expected amount of total weight of
144+
# generated votes in the sims.
145+
vote-generation-probability: 500.0
146+
# vote-spec#"Committe and quorum size"
147+
# 60% of `vote-generation-probability`
148+
vote-threshold: 400
149+
# vote-spec#"Generate vote" 0.8*135e-3 + 0.2*280e-3
150+
vote-generation-cpu-time-ms-constant: 164.0e-3
151+
# No benchmark yet.
152+
vote-generation-cpu-time-ms-per-ib: 0
153+
# vote-spec#"Verify vote" 0.8*670e-3 + 0.2*1.4
154+
vote-validation-cpu-time-ms: 816.0e-3
155+
# The `Vote` structure counted in the -per-eb already identifies slot
156+
# (in Eid) and voter. We can assume a vote bundle is all for the same
157+
# voter and slot, so for non-persistent voters we could factor their
158+
# PoolKeyHash (28bytes) here, but that is for 20% of cases.
159+
# More relevant if EB generation is very high.
160+
vote-bundle-size-bytes-constant: 0
161+
# vote-spec#Votes 0.8*90 + 0.2*164
162+
vote-bundle-size-bytes-per-eb: 105
163+
164+
vote-diffusion-strategy: "peer-order"
165+
166+
# Haskell prototype relay mini-protocol parameters.
167+
vote-diffusion-max-bodies-to-request: 1
168+
vote-diffusion-max-headers-to-request: 100
169+
vote-diffusion-max-window-size: 100
170+
171+
################################################################################
172+
# Certificate Configuration
173+
################################################################################
174+
175+
# vote-spec#"certificate bytes"
176+
cert-size-bytes-constant: 136
177+
# vote-spec#"certificate bytes" ((80/8) + 76 * (100 - 80))/100
178+
cert-size-bytes-per-node: 15
179+
180+
# For certificate timings we have bulk figures for realistic scenarios,
181+
# so we do not attempt to give -per-node (i.e. per-voter) timings.
182+
#
183+
# vote-spec#"Generate certificate"
184+
cert-generation-cpu-time-ms-constant: 90.0
185+
cert-generation-cpu-time-ms-per-node: 0
186+
# vote-spec#"Verify certificate"
187+
cert-validation-cpu-time-ms-constant: 130.0
188+
cert-validation-cpu-time-ms-per-node: 0

‎analysis/sims/2025w11xl/env.sh

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
export HOST=thelio
2+
export DB=leios2025w11xl

0 commit comments

Comments
 (0)