Expand storage for Holesky fleet #218
Comments
|
Current state: |
|
Re-sync itself can free some space too. |
Is re-sync what @jakubgs did here #219 (comment)? |
|
Yes, I think. You can read on this in geth docs:
https://geth.ethereum.org/docs/fundamentals/pruning Although I've never tried prune, just remove and run the node to sync from scratch. |
|
That is correct, I simply purged the data folder (excluding any key files) and allowed to sync from scratch. And yes, Holesky is way smaller than mainnet. |
|
For all of the Nethermind hosts I've purged the data and started sync from scratch for two of the four nodes per host. For all of them |
|
I've did the same for geth holesky hosts, 2 nodes per host have been resynced(I've purged the |
|
Done with all nethermind hosts. The layout looks like this: Devices sdc and sdd are combined into raid0 logical volume and mounted at |
|
On geth hosts I've noticed this in geth exporter logs: 2025/01/31 11:46:08 failed to get metrics: the method debug_metrics does not exist/is not available
2025/01/31 11:46:38 failed to get metrics: the method debug_metrics does not exist/is not availablewhich is weird considering that the metrics it exports all state # HELP geth_sync_txIndexFinishedBlocks_value metric exported from geth with debug.metricsAnother thing is this depends_on in its docker compose file: which doesn't make sense and results with an error if you try to stop container with compose down: [email protected]:~ % docker compose -f /docker/geth-holesky-02/docker-compose.exporter.yml down
WARN[0000] /docker/geth-holesky-02/docker-compose.exporter.yml: `version` is obsolete
service "metrics" depends on undefined service "geth": invalid compose project |
|
Done with the geth hosts, the layout is the same as nethermind: [email protected]:~ % lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 372.5G 0 disk
├─sda1 8:1 0 1G 0 part /boot/efi
└─sda2 8:2 0 371.5G 0 part /docker
/
sdb 8:16 0 1.5T 0 disk /data
/mnt/sdb
sdc 8:32 0 1.5T 0 disk
└─md0 9:0 0 2.9T 0 raid0 /mnt/sdc
/docker
sdd 8:48 0 1.5T 0 disk
└─md0 9:0 0 2.9T 0 raid0 /mnt/sdc
/docker
[email protected]:~ % df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 6.3G 2.6M 6.3G 1% /run
/dev/sda2 365G 42G 305G 12% /
tmpfs 32G 332K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sdb 1.5T 825G 568G 60% /data
/dev/sda1 1022M 6.1M 1016M 1% /boot/efi
/dev/md0 2.9T 6.3G 2.8T 1% /docker
tmpfs 6.3G 0 6.3G 0% /run/user/7010
|
|
Done with the erigon hosts, the layout is the same as nethermind and geth: [email protected]:~ % lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 372.5G 0 disk
├─sda1 8:1 0 1G 0 part /boot/efi
└─sda2 8:2 0 371.5G 0 part /docker
/
sdb 8:16 0 1.5T 0 disk /data
/mnt/sdb
sdc 8:32 0 1.5T 0 disk
└─md0 9:0 0 2.9T 0 raid0 /mnt/sdc
/docker
sdd 8:48 0 1.5T 0 disk
└─md0 9:0 0 2.9T 0 raid0 /mnt/sdc
/docker
[email protected]:~ % df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 6.3G 2.2M 6.3G 1% /run
/dev/sda2 365G 44G 303G 13% /
tmpfs 32G 316K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sdb 1.5T 824G 568G 60% /data
/dev/sda1 1022M 6.1M 1016M 1% /boot/efi
/dev/md0 2.9T 34G 2.7T 2% /docker
tmpfs 6.3G 0 6.3G 0% /run/user/7010
I've resynced them all, they will be synced in few hours. |
We are currently low on storage for EL nodes on
nimbus.holeskyfleet. Storage usage on/dockervolume varies from 72% up to 96% on some nodes.We need to:
You can see notes on previous task like this here:
The text was updated successfully, but these errors were encountered: