Skip to content

Network stack boot time optimization: decouple from DATA partition #638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 6, 2025
Merged

Network stack boot time optimization: decouple from DATA partition #638

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
951b136
release: add direct mount for PRIVATE partition at /.bottlerocket
ytsssun Sep 16, 2025
2022a51
netdog: use /run/netdog for network status files
ytsssun Sep 25, 2025
b3b2a98
netdog: update services for /.bottlerocket path
ytsssun Sep 25, 2025
651023c
systemd: ensure dbus is available for network services
ytsssun Sep 16, 2025
dd845f7
migrator: add explicit dependency on /var/lib/bottlerocket
ytsssun Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions packages/dbus-broker/dbus-broker.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
Description=D-Bus System Message Bus
Documentation=https://github.com/bus1/dbus-broker
DefaultDependencies=false
After=dbus.socket
# Ensure the dbus user is created before starting dbus service
After=dbus.socket systemd-sysusers.service
Wants=dbus.socket systemd-sysusers.service
Before=basic.target shutdown.target
Requires=dbus.socket
Conflicts=shutdown.target
Expand All @@ -13,7 +15,9 @@ Sockets=dbus.socket
OOMScoreAdjust=-900
LimitNOFILE=16384
ProtectSystem=full
PrivateTmp=true
# Disable private /tmp to avoid dependency on systemd-tmpfiles and consequently
# local-fs.target, allowing dbus to start earlier in the boot
PrivateTmp=no
PrivateDevices=true
ExecStart=/usr/bin/dbus-broker-launch --scope system
ExecReload=/usr/bin/busctl call org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus ReloadConfig
Expand Down
5 changes: 5 additions & 0 deletions packages/dbus-broker/dbus.socket
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
[Unit]
Description=D-Bus System Message Bus Socket
# Disable DefaultDependencies to allow D-Bus to be started earlier in the boot
# so that it is available for other services
DefaultDependencies=no
Before=shutdown.target sockets.target
Conflicts=shutdown.target

[Socket]
ListenStream=/run/dbus/system_bus_socket
Expand Down
3 changes: 2 additions & 1 deletion packages/filesystem/filesystem.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ mkdir -p %{buildroot}%{_cross_infodir}
mkdir -p %{buildroot}%{_cross_mandir}
mkdir -p %{buildroot}%{_cross_localstatedir}
mkdir -p %{buildroot}/{boot,dev,proc,run,sys,tmp}
mkdir -p %{buildroot}/{home,local,media,mnt,opt,srv}
mkdir -p %{buildroot}/{home,local,media,mnt,opt,srv,.bottlerocket}
mkdir -p %{buildroot}/media/cdrom
mkdir -p %{buildroot}/root/.aws

Expand Down Expand Up @@ -74,6 +74,7 @@ ln -s .%{_sbindir} %{buildroot}/sbin
/sys
/tmp

/.bottlerocket
/home
/local
/media
Expand Down
3 changes: 3 additions & 0 deletions packages/netdog/generate-network-config.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
[Unit]
Description=Generate network configuration
DefaultDependencies=no
# Block manual interactions with this service, since it could leave the system in an
# unexpected state
RefuseManualStart=true
RefuseManualStop=true
Before=network-pre.target
RequiresMountsFor=/.bottlerocket

[Service]
Type=oneshot
Expand Down
2 changes: 0 additions & 2 deletions packages/netdog/netdog-tmpfiles.conf
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions packages/netdog/netdog.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ Summary: Bottlerocket network configuration helper
License: Apache-2.0 OR MIT
URL: https://github.com/bottlerocket-os/bottlerocket

Source0: netdog-tmpfiles.conf

Source10: run-netdog.mount
Source11: write-network-status.service
Source12: generate-network-config.service
Expand Down Expand Up @@ -64,9 +62,6 @@ install -p -m 0755 ${HOME}/.cache/dogtag/%{__cargo_target}/release/10-reverse-dn
install -d %{buildroot}%{_cross_bindir}
install -p -m 0755 ${HOME}/.cache/networkd/%{__cargo_target}/release/netdog %{buildroot}%{_cross_bindir}/netdog

install -d %{buildroot}%{_cross_tmpfilesdir}
install -p -m 0644 %{S:0} %{buildroot}%{_cross_tmpfilesdir}/netdog.conf

install -d %{buildroot}%{_cross_unitdir}
install -p -m 0644 %{S:10} %{S:11} %{S:12} %{S:13} %{buildroot}%{_cross_unitdir}

Expand All @@ -75,7 +70,6 @@ install -d %{buildroot}%{_cross_libdir}/systemd/resolved.conf.d
install -p -m 0644 %{S:20} %{buildroot}%{_cross_libdir}/systemd/resolved.conf.d

%files
%{_cross_tmpfilesdir}/netdog.conf
%{_cross_unitdir}/generate-network-config.service
%{_cross_unitdir}/disable-udp-offload.service
%{_cross_unitdir}/run-netdog.mount
Expand Down
5 changes: 4 additions & 1 deletion packages/netdog/write-network-status.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
[Unit]
Description=Write network status
DefaultDependencies=no
# Block manual interactions with this service, since it could leave the system in an
# unexpected state
RefuseManualStart=true
RefuseManualStop=true
Before=early-boot-config.service
Before=early-boot-config.service network-online.target
Copy link
Contributor Author

@ytsssun ytsssun Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to add this explicit Before annotation, otherwise I get weird systemd ordering that write-network-status happens after network-online.target:

network-online.target @1.979s
└─systemd-networkd-wait-online.service @1.254s +722ms
  └─systemd-networkd.service @1.189s +61ms
    └─network-pre.target @1.186s
...

With this change, the write-network-status.service happens before the network-online.target.

network-online.target @1.788s
└─write-network-status.service @1.753s +33ms      <-- listed as dependency
  └─systemd-resolved.service @2.669s +217ms
    └─systemd-sysctl.service @753ms +22ms
...

# This service creates a symlink to the resolv.conf systemd-resolved creates
# and we would like it to exist first
RequiresMountsFor=/.bottlerocket /run/netdog
After=systemd-networkd-wait-online.service systemd-resolved.service
Wants=systemd-networkd-wait-online.service systemd-resolved.service

[Service]
Type=oneshot
ExecStart=mkdir -p /etc/sysctl.d
ExecStart=/usr/bin/netdog write-primary-interface-status
RemainAfterExit=true
StandardError=journal+console
Expand Down
2 changes: 2 additions & 0 deletions packages/os/migrator.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
[Unit]
Description=Bottlerocket data store migrator
DefaultDependencies=no
Before=apiserver.service mark-successful-boot.service storewolf.service
RequiresMountsFor=/var/lib/bottlerocket

RefuseManualStart=true
RefuseManualStop=true
Expand Down
8 changes: 6 additions & 2 deletions packages/os/whippet.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
[Unit]
Description=D-Bus System Message Bus
DefaultDependencies=false
After=dbus.socket
# Ensure the dbus user is created before starting dbus service
After=dbus.socket systemd-sysusers.service
Wants=dbus.socket systemd-sysusers.service
Before=basic.target shutdown.target
Requires=dbus.socket
Conflicts=shutdown.target
Expand All @@ -12,7 +14,9 @@ Sockets=dbus.socket
OOMScoreAdjust=-900
LimitNOFILE=16384
ProtectSystem=full
PrivateTmp=true
# Disable private /tmp to avoid dependency on systemd-tmpfiles and consequently
# local-fs.target, allowing dbus to start earlier in the boot
PrivateTmp=no
PrivateDevices=true
ExecStart=/usr/bin/whippet

Expand Down
19 changes: 19 additions & 0 deletions packages/release/bottlerocket.mount.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[Unit]
Description=Private Directory (/.bottlerocket)
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=dev-disk-by\x2dpartlabel-BOTTLEROCKET\x2dPRIVATE.device selinux-policy-files.service
Requires=dev-disk-by\x2dpartlabel-BOTTLEROCKET\x2dPRIVATE.device
Wants=selinux-policy-files.service
Copy link
Contributor

@arnaldo2792 arnaldo2792 Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not making this Required as well?

Copy link
Contributor Author

@ytsssun ytsssun Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used Wants here to be consistent with elsewhere https://github.com/search?q=repo%3Abottlerocket-os%2Fbottlerocket-core-kit+%3Dselinux-policy-files.service&type=code

RefuseManualStart=true
RefuseManualStop=true

[Mount]
What=/dev/disk/by-partlabel/BOTTLEROCKET-PRIVATE
Where=/.bottlerocket
Options=defaults,noexec,nosuid,nodev,noatime,private,context=system_u:object_r:private_t:s0
StandardError=journal+console

[Install]
WantedBy=local-fs.target
10 changes: 10 additions & 0 deletions packages/release/network-pre-target-dbus-dep.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[Unit]
# Ensure D-Bus is fully initialized before network services start.
# This prevents a race condition where network services could attempt
# to use D-Bus functionality (like networkctl commands) before the D-Bus
# broker service is fully ready to handle requests.
DefaultDependencies=no
After=dbus.service
Requires=dbus.service
Before=shutdown.target
Conflicts=shutdown.target
13 changes: 13 additions & 0 deletions packages/release/release.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ Source1082: usr-share-licenses.mount.in
Source1083: lib-modules.mount.in
Source1084: usr-bin.mount.in
Source1085: usr-libexec.mount.in
Source1086: bottlerocket.mount.in

# Drop-in units to override defaults
Source1100: systemd-tmpfiles-setup-service-debug.conf
Expand All @@ -108,6 +109,7 @@ Source1107: systemd-journald-compat.conf
Source1108: systemd-sysusers-selinux.conf
Source1109: modprobe-no-exit.conf
Source1110: tmp-mount-noexec.conf
Source1111: network-pre-target-dbus-dep.conf

# network link rules
Source1200: 80-release.link
Expand Down Expand Up @@ -269,6 +271,10 @@ install -d %{buildroot}%{_cross_unitdir}/tmp.mount.d
install -p -m 0644 %{S:1110} \
%{buildroot}%{_cross_unitdir}/tmp.mount.d/10-no-exec.conf

install -d %{buildroot}%{_cross_unitdir}/network-pre.target.d
install -p -m 0644 %{S:1111} \
%{buildroot}%{_cross_unitdir}/network-pre.target.d/00-dbus-dep.conf

# Empty (but packaged) directory. The FIPS packages for kernels will add drop-ins to
# this directory to arrange for the right modules to be loaded before the check runs.
install -d %{buildroot}%{_cross_unitdir}/check-fips-modules.service.d
Expand Down Expand Up @@ -302,6 +308,10 @@ LIBEXECDIRPATH=$(systemd-escape --path %{_cross_libexecdir})
sed -e 's|PREFIX|%{_cross_prefix}|g' %{S:1085} > ${LIBEXECDIRPATH}.mount
install -p -m 0644 ${LIBEXECDIRPATH}.mount %{buildroot}%{_cross_unitdir}

# Process bottlerocket mount template files with proper systemd naming
BOTTLEROCKET_PATH=$(systemd-escape --path /.bottlerocket)
install -p -m 0644 %{S:1086} %{buildroot}%{_cross_unitdir}/${BOTTLEROCKET_PATH}.mount

install -d %{buildroot}%{_cross_templatedir}
install -p -m 0644 %{S:200} %{buildroot}%{_cross_templatedir}/motd
install -p -m 0644 %{S:201} %{buildroot}%{_cross_templatedir}/proxy-env
Expand Down Expand Up @@ -355,6 +365,7 @@ ln -s preconfigured.target %{buildroot}%{_cross_unitdir}/default.target
%{_cross_unitdir}/prepare-opt.service
%{_cross_unitdir}/prepare-var.service
%{_cross_unitdir}/repart-local.service
%{_cross_unitdir}/\x2ebottlerocket.mount
%{_cross_unitdir}/var.mount
%{_cross_unitdir}/opt.mount
%{_cross_unitdir}/mnt.mount
Expand All @@ -373,6 +384,8 @@ ln -s preconfigured.target %{buildroot}%{_cross_unitdir}/default.target
%{_cross_unitdir}/mask-local-mnt.service
%{_cross_unitdir}/mask-local-opt.service
%{_cross_unitdir}/mask-local-var.service
%dir %{_cross_unitdir}/network-pre.target.d
%{_cross_unitdir}/network-pre.target.d/00-dbus-dep.conf
%{_cross_unitdir}/root-.aws.mount
%{_cross_unitdir}/repart-data-preferred.service
%{_cross_unitdir}/repart-data-fallback.service
Expand Down
8 changes: 4 additions & 4 deletions packages/release/var-lib-bottlerocket.mount
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@
Description=Private Directory (/var/lib/bottlerocket)
DefaultDependencies=no
Conflicts=umount.target
RequiresMountsFor=/var
RequiresMountsFor=/var /.bottlerocket
Before=local-fs.target umount.target

[Mount]
What=/dev/disk/by-partlabel/BOTTLEROCKET-PRIVATE
What=/.bottlerocket
Where=/var/lib/bottlerocket
Type=ext4
Options=defaults,nosuid,nodev,noexec,noatime,private,context=system_u:object_r:private_t:s0
Type=none
Options=rbind,rshared
Copy link
Contributor

@arnaldo2792 arnaldo2792 Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have to make this rshared? Why not just rprivate?

Copy link
Contributor Author

@ytsssun ytsssun Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linking @bcressey 's comments - #638 (comment)

Most of these options aren't valid for bind mounts because they share the same filesystem superblock.
Generally it's useful to set up the recursive options so mounts are propagated in both locations


[Install]
WantedBy=preconfigured.target
2 changes: 1 addition & 1 deletion sources/netdog/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ It contains two subcommands meant for use as settings generators:
The subcommand `set-hostname` sets the hostname for the system.

The subcommand `generate-net-config` generates the network interface configuration for the host. If
a `net.toml` file exists in `/var/lib/bottlerocket`, it is used to generate the configuration. If
a `net.toml` file exists in `/.bottlerocket`, it is used to generate the configuration. If
`net.toml` doesn't exist, the kernel command line `/proc/cmdline` is checked for the prefix
`netdog.default-interface`. If an interface is defined with that prefix, it is used to generate an
interface configuration. A single default interface may be defined on the kernel command line with
Expand Down
8 changes: 3 additions & 5 deletions sources/netdog/src/cli/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,8 @@ pub(crate) mod write_resolv_conf;

use crate::net_config::{self, Interfaces};
use crate::{
DEFAULT_NET_CONFIG_FILE, KERNEL_CMDLINE, OVERRIDE_NET_CONFIG_FILE, PRIMARY_INTERFACE,
PRIMARY_MAC_ADDRESS, PRIMARY_SYSCTL_CONF, SYSCTL_MARKER_FILE, SYSTEMD_SYSCTL, SYS_CLASS_NET,
USR_NET_CONFIG_FILE,
DEFAULT_NET_CONFIG_FILE, KERNEL_CMDLINE, PRIMARY_INTERFACE, PRIMARY_MAC_ADDRESS,
PRIMARY_SYSCTL_CONF, SYSCTL_MARKER_FILE, SYSTEMD_SYSCTL, SYS_CLASS_NET, USR_NET_CONFIG_FILE,
};
pub(crate) use generate_hostname::GenerateHostnameArgs;
pub(crate) use generate_net_config::GenerateNetConfigArgs;
Expand Down Expand Up @@ -147,12 +146,11 @@ where

// Gather net config from possible sources, returning both the config and the source
fn fetch_net_config() -> Result<(Option<Box<dyn Interfaces>>, PathBuf)> {
let override_path = PathBuf::from(OVERRIDE_NET_CONFIG_FILE);
let default_path = PathBuf::from(DEFAULT_NET_CONFIG_FILE);
let usr_path = PathBuf::from(USR_NET_CONFIG_FILE);
let cmdline = PathBuf::from(KERNEL_CMDLINE);

for path in [override_path, default_path, usr_path] {
for path in [default_path, usr_path] {
if Path::exists(&path) {
return Ok((
net_config::from_path(&path).context(error::NetConfigParseSnafu { path: &path })?,
Expand Down
11 changes: 5 additions & 6 deletions sources/netdog/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ It contains two subcommands meant for use as settings generators:
The subcommand `set-hostname` sets the hostname for the system.

The subcommand `generate-net-config` generates the network interface configuration for the host. If
a `net.toml` file exists in `/var/lib/bottlerocket`, it is used to generate the configuration. If
a `net.toml` file exists in `/.bottlerocket`, it is used to generate the configuration. If
`net.toml` doesn't exist, the kernel command line `/proc/cmdline` is checked for the prefix
`netdog.default-interface`. If an interface is defined with that prefix, it is used to generate an
interface configuration. A single default interface may be defined on the kernel command line with
Expand Down Expand Up @@ -42,12 +42,11 @@ use argh::FromArgs;
use std::process;

static KERNEL_HOSTNAME: &str = "/proc/sys/kernel/hostname";
static CURRENT_IP: &str = "/var/lib/netdog/current_ip";
static CURRENT_IP: &str = "/run/netdog/current_ip";
static KERNEL_CMDLINE: &str = "/proc/cmdline";
static PRIMARY_INTERFACE: &str = "/var/lib/netdog/primary_interface";
static PRIMARY_MAC_ADDRESS: &str = "/var/lib/netdog/primary_mac_address";
static DEFAULT_NET_CONFIG_FILE: &str = "/var/lib/bottlerocket/net.toml";
static OVERRIDE_NET_CONFIG_FILE: &str = "/var/lib/netdog/net.toml";
static PRIMARY_INTERFACE: &str = "/run/netdog/primary_interface";
static PRIMARY_MAC_ADDRESS: &str = "/run/netdog/primary_mac_address";
static DEFAULT_NET_CONFIG_FILE: &str = "/.bottlerocket/net.toml";
static USR_NET_CONFIG_FILE: &str = "/usr/share/bottlerocket/net.toml";
static PRIMARY_SYSCTL_CONF: &str = "/etc/sysctl.d/90-primary_interface.conf";
static SYSCTL_MARKER_FILE: &str = "/run/netdog/primary_sysctls_set";
Expand Down