Fix inode exhaustion in benchmark bootstrap by reducing default bytes-per-inode

[Copilot]

LevelDB benchmarks fail with "no space left on device" due to inode exhaustion, not disk space exhaustion. The bootstrap script creates ext4 filesystems with 6MB per inode (6291456 bytes), which severely limits inode count on large NVMe devices. LevelDB's many small files quickly exhaust available inodes.

Changes

• build-environment.sh: Make bytes-per-inode configurable via --bytes-per-inode command-line option, default to 2MB (2097152 bytes) instead of hardcoded 6MB
• build-environment.sh: Add --help option to display usage information
• README.md: Document the new command-line option with guidance on when to use different values:
  • Default 2MB works well for LevelDB workloads with many small files (such as LevelDB with AvalancheGo)
  • Larger values like 6MB can be used for Firewood-only workloads to provide more database space

Usage

# Uses default 2MB per inode (good for LevelDB + AvalancheGo)
sudo bash build-environment.sh

# Use larger value for Firewood-only workloads (more database space)
sudo bash build-environment.sh --bytes-per-inode 6291456

# Display help
sudo bash build-environment.sh --help

The 2MB default provides 3x more inodes while maintaining sufficient space for Firewood's single-file database and LevelDB's numerous small files. For Firewood-only workloads without LevelDB, users can increase the value to 6MB for more database space.

• Fixes bug(benchmark/bootstrap): benchmark fails due to inode exhaustion #1444

Original prompt

---

This section details on the original issue you should resolve

<issue_title>bug(benchmark/bootstrap): benchmark fails due to inode exhaustion</issue_title>
<issue_description>While trying to gauge the performance of Firewood + RootStore against LevelDB + HashDB, I ran the following:

./aws-launch.sh --firewood-branch rodrigo/add-root-store-alpha --avalanchego-branch rodrigo/add-firewood-archive-config-in-reexecution --coreth-branch rodrigo/firewood-v0.0.14-with-rootstore --libevm-commit 1bccf4f2ddb2 --instance-type i4i.xlarge --config archive --nblocks 10m

I was able to get the reexecution test running on an EC2 instance, but the test eventually failed and I saw this in the error logs:

[11-08|08:00:13.388] ERROR <2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5 Chain> triedb/hashdb/database.go:420 Failed to write flush list to disk       err="open 
/mnt/nvme/ubuntu/exec-data/current-state/db/6725573.ldb: no space left on device"
[11-08|08:00:13.388] CRIT <2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5 Chain> core/blockchain.go:620 unable to flatten snapshot from acceptor blockHash=e7a18c..c04315 err="open 
/mnt/nvme/ubuntu/exec-data/current-state/db/6725573.ldb: no space left on device"

At first glance, I thought that the LevelDB instance had grown larger than the SSD size, but running df -h --total revealed the opposite:

Filesystem       Size  Used Avail Use% Mounted on
/dev/root         48G   26G   23G  53% /
tmpfs             16G     0   16G   0% /dev/shm
tmpfs            6.2G  948K  6.2G   1% /run
tmpfs            5.0M     0  5.0M   0% /run/lock
efivarfs         128K  3.9K  120K   4% /sys/firmware/efi/efivars
/dev/nvme1n1p16  881M  155M  665M  19% /boot
/dev/nvme1n1p15  105M  6.2M   99M   6% /boot/efi
/dev/nvme0n1     872G  407G  422G  50% /mnt/nvme
tmpfs            3.1G   12K  3.1G   1% /run/user/1007
total            945G  432G  469G  48% - 

Consulting with Claude, it seems that the test failed due to inode exhaustion - running df -i seems to confirm this:

/dev/nvme0n1   223424 223424     0  100% /mnt/nvme 

I'm somewhat hesitant to consider this a bug since aws-launch.sh is tailored towards running Firewood benchmarks but if we want to have consistent benchmarks of Firewood and LevelDB + HashDB, the script should also be able to setup a EC2 instance that tests LevelDB + HashDB.</issue_description>

<agent_instructions>Change the script to support configuring the bytes per inode, and set the default to 2097152</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@AminR443 This happens because of the way we create the filesystem in [firewood/benchmark/setup-scripts/build-environment.sh](https://github.com/ava-labs/firewood/blob/cae9899e7730a185fa0a3777b2c8b549cdc26835/benchmark/setup-scripts/build-environment.sh), and is not related to the `aws-launch.sh` script.

I think build-environment.sh script was meant for the benchmark scripts that we had before, and we reused it in aws-launch.sh. We didn't run into any issues as firewood is a single file database, and other files (like the blockdb ones) didn't seem to exhaust the inodes in our experiments.

I ran into this in my tests when I needed to compile perf from sources to do some profiling and cloning linux repo exhausted all of inodes.

The reason is this line:

firewood/benchmark/setup-scripts/build-environment.sh

Line 65 in cae9899

mkfs.ext4 -E nodiscard -i 6291456 "$DEVICE_TO_USE"

It sets bytes-per-inode to 6MB. The main reason I think @rkuris did this was to speedup file system creation when using large devices. Reducing that should solve your problem, default value for ext4 is 16KB iirc.

@rkuris I think we should actually change this for our experiments, because of the LevelDB stuff we get from S3, in some storage/block-range pairs we might run into this issue with firewood config as well. Let me know and I'll create a PR.</comment_new>
<comment_new>@RodrigoVillar
Ah thanks for the context @AminR443! Let's wait for @rkuris to respond but I would be in favor of parameterizing bytes-per-inode or lowering it altogether so that build-environment.sh can also be used for testing LevelDB + HashDB with archival mode enabled.</comment_new>
<comment_new>@rkuris
As discussed, our bytes per inode value is larger to allow for more space but doesn't work so well with leveldb. I think we can probably lower this by a factor of 3 to 2097152.</comment_new>

• Fixes bug(benchmark/bootstrap): benchmark fails due to inode exhaustion #1444

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[rkuris]

@copilot I prefer a command line over an environment variable. Can you add the command line option recommended in the original issue?

[Copilot]

@copilot I prefer a command line over an environment variable. Can you add the command line option recommended in the original issue?

Changed to use --bytes-per-inode command-line option instead of environment variable in commit 0680e00. The script now supports:

# Default usage (2MB per inode)
sudo bash build-environment.sh

# Custom value
sudo bash build-environment.sh --bytes-per-inode 1048576

# Help
sudo bash build-environment.sh --help

[rkuris]

@copilot Modify the comment to mention that if you're not using leveldb (for example, just using firewood without avalanchego), you don't need so many inodes, which gives you more room for the database itself, so you can and should use a larger value.

Change the example to show how to do that (use 6291456 instead of 1048576)

[Copilot]

Updated the documentation in commit 060565d:

• Added explanation that the 2MB default works well for LevelDB workloads with many small files
• Clarified that Firewood-only workloads can use larger values (6MB) for more database space
• Changed the example to use 6291456 instead of 1048576