Right now, when a new target image is identified, all BootcNodes are updated with desiredImageState: Staged. This means all the nodes in the pool simultaneously staging the update. This is an incredible I/O and traffic spike (related: #61).
We should be smarter than this and support only requesting staging slightly ahead of the reboot rollout. (One way I picture this is when watching a video online, you're only buffering a few seconds ahead of the playback position, you're not downloading the whole video.)
This probably should be a configurable "stage ahead N nodes" knob since the right value probably depends on various factors (e.g. how fast downloading updates are vs the drain and reboot step, how much the sysadmin cares about network spikes, etc.). Though we should have a basic reasonable default heuristic when unset.
Right now, when a new target image is identified, all BootcNodes are updated with
desiredImageState: Staged. This means all the nodes in the pool simultaneously staging the update. This is an incredible I/O and traffic spike (related: #61).We should be smarter than this and support only requesting staging slightly ahead of the reboot rollout. (One way I picture this is when watching a video online, you're only buffering a few seconds ahead of the playback position, you're not downloading the whole video.)
This probably should be a configurable "stage ahead N nodes" knob since the right value probably depends on various factors (e.g. how fast downloading updates are vs the drain and reboot step, how much the sysadmin cares about network spikes, etc.). Though we should have a basic reasonable default heuristic when unset.