The node was low on resource: ephemeral-storage

[simonmacklin]

EKS version 1.2.1

Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         4
  ephemeral-storage:           20619092Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      16072844Ki
  pods:                        58
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         3920m
  ephemeral-storage:           17928813332
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      15056012Ki
  pods:                        58

[root@admin]# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0          7:0    0 11.7M  1 loop /.bottlerocket/rootfs/var/lib/kernel-devel/.overlay/lower
loop1          7:1    0  312K  1 loop /.bottlerocket/rootfs/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
nvme0n1      259:0    0  100G  0 disk
|-nvme0n1p1  259:2    0    4M  0 part
|-nvme0n1p2  259:3    0    5M  0 part
|-nvme0n1p3  259:4    0   40M  0 part /.bottlerocket/rootfs/boot
|-nvme0n1p4  259:5    0  920M  0 part
|-nvme0n1p5  259:6    0   10M  0 part
|-nvme0n1p6  259:7    0   25M  0 part
|-nvme0n1p7  259:8    0    5M  0 part
|-nvme0n1p8  259:9    0   40M  0 part
|-nvme0n1p9  259:10   0  920M  0 part
|-nvme0n1p10 259:11   0   10M  0 part
|-nvme0n1p11 259:12   0   25M  0 part
`-nvme0n1p12 259:13   0   42M  0 part /.bottlerocket/rootfs/var/lib/bottlerocket
nvme1n1      259:1    0   20G  0 disk
`-nvme1n1p1  259:14   0   20G  0 part /.bottlerocket/rootfs/local
[root@admin]#

df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay          20G   15G  4.1G  79% /

I have been testing bottle rocket in EKS for a while now and since we rolled this out to one of our test clusters we have seen a lot of evictions that otherwise never happened with Amazon linux.

The eviction message is "The node was low on resource: ephemeral-storage". Usually this is due to low disk on the underlying host which may make sense looking at the details above as only 5GB is free on the root disk even though we have a 100GB disk EBS disk attached.

I was not able to find in the docs how we can adjust the values to increase this partitions. Does this mean we would need to extend using standard tools and rebuild a custom AMI or can this be done as startup time?

[stmcginnis]

Hi @simonmacklin. There are typically two disks attached to a Bottlerocket instance - the root disk for the system and a data disk for ephemeral/container storage. For that system disk, it really is not worth increasing the size of it as Bottlerocket has a read-only root filesystem that just contains two partitions to hold the Bottlerocket image. Any extra space allocated to the root disk is just wasted space, so you end up paying for extra space that can never be consumed.

The second disk is the interesting one. When Bottlerocket first boots, it will create a filesystem on that to use all available disk space. So in your example above, nvme1n1 is only 20G, so that is all the space that will be made available for your containers. So I believe what you actually want to do is change that first disk size back to its default, then extend the size of the second disk to be 100GB.

[simonmacklin]

Hey @stmcginnis

Thanks for the info. I think I figured out what was going on. The terraform we use to build the clusters mapped the root EBS volume to device /dev/xvda which was fine for amazon linux. It appears the bottle rocket AMIs use /dev/xvdb so the 100GB disk wasn't being used in the right place it appears.

Changing the device mapping in terraform appeared to solve the ephemeral storage issue but something else has created another additional 2GB EBS disk. So we have one 100GB disk created in terraform used correctly and another 2GB disk.

Does the bottle rocket software create this at boot time?

[stmcginnis]

Awesome! 🎉

Yes, by default when you spin up a Bottlerocket instance it will have two volumes attach. Unless you specify otherwise, this is a 2 GiB volume for the read-only system partition, and larger volume for a data partition. When Bottlerocket initializes it will extend that data partition to use whatever space is available. So if you swap out something like a 20GiB and decide to make it 100GiB, Bottlerocket will automatically extend the partition to use all of that available space and you should have lots of space for your container volume layers, images, and ephemeral storage.

[simonmacklin]

Amazing,

Thank you so much!