[WIP] many: switch to bootc based anaconda (away from rpm) (HMS-9392)

[mvo5]

This spike show how we could use bootc based containers with anaconda. So instead of building our iso from rpm packages and just putting the bootc container into the rpm generated ISO this spike builds the ISO from a bootc container.

This requires that the container has the right content of course (i.e. anaconda). We would move some of the responsibility to the user (or to a tool that helps the user and hides the complexity here like bootc-iso-imagectl or something) to ensure that.

The Containerfile to generate an installer looks something like:

FROM {container_ref}
RUN dnf install -y \
     anaconda \
     anaconda-install-env-deps \
     anaconda-dracut \
     dracut-config-generic \
     dracut-network \
     net-tools \
     squashfs-tools \
     grub2-efi-x64-cdboot \
     python3-mako \
     lorax-templates-* \
     biosdevname \
     prefixdevname \
     && dnf clean all
# shim-x64 is marked installed but the files are not in the expected
# place for https://github.com/osbuild/osbuild/blob/v160/stages/org.osbuild.grub2.iso#L91, see
# workaround via reinstall, we could add a config to the grub2.iso
# stage to allow a different prefix that then would be used by
# anaconda.
# once https://github.com/osbuild/osbuild/pull/2202 is merged we
# can update images/ to set the correct efi_src_dir and this can
# be removed
RUN dnf reinstall -y shim-x64
# lorax wants to create a symlink in /mnt which points to /var/mnt
# on bootc but /var/mnt does not exist on some images.
#
# If https://gitlab.com/fedora/bootc/base-images/-/merge_requests/294
# gets merged this will be no longer needed
RUN mkdir /var/mnt

this basic template works for cs9,cs10,f42 (Note that the goal here is to essentially remove all quirks
and hide them in images/, ideally the containerfile would be just a "FROM \ndnf install anaconda...").

This branch also has a "--installer-payload" switch so to use this one would essentially write: bib --type iso --installer-payload quay.io/centos-bootc/centos-bootc:stream9 quay.io/centos-bootc-anaconda:stream9 (as currently the containerfile changes things like /root which makes it no longer a proper bootc container). We could of course explore making this more user friendly, either by a convention like "suffix -installer" for type iso or by creating a derrived constainer locally ourselfs with the above containerfile or helper or by fixing the issues that require /root to be a real dir (but even with that it would probably be undesired to install a container that contains a full anaconda env so something like installer-payload is probably still be needed).

Note that this is big departure from our current approach and we would need to think about a transition strategy/communication if we go down this route.

Thanks to Ondrej, Colin for their help/inputs in this, I hope it captures what they suggested to explore.

/jira-epic HMS-8839

JIRA: HMS-9392

[cgwalters]

AWESOME!!!! Thanks so much for starting this!

Even having an experimental/hidden option for this would be super useful.

shim-x64 is marked installed but the files are not in the expected
place for https://github.com/osbuild/osbuild/blob/v160/stages/org.osbuild.grub2.iso#L91

Yes because they're managed via bootupd; I think we could intersect bootupd with the osbuild ISO logic here.

Or...maybe more generally this is really intersecting bootc install to-filesystem and ISOs.

(as currently the containerfile changes things like /root which makes it no longer a proper bootc container).

Nah it's still a bootc container, we support content in /var (this is /var/roothome), it's just discouraged. https://bootc-dev.github.io/bootc/filesystem.html#var

[achilleas-k]

This is amazing work. I need to read it in more detail, but from a quick skim it looks great.
TY!

[mvo5]

AWESOME!!!! Thanks so much for starting this!

Thank you! Really happy that you like it!

[..]

(as currently the containerfile changes things like /root which makes it no longer a proper bootc container).
Nah it's still a bootc container, we support content in /var (this is /var/roothome), it's just discouraged. https://bootc-dev.github.io/bootc/filesystem.html#var

I managed to remove this (and more quirks), I think the only one left is that anaconda does not like it when /mnt is a dangling symlink in the base image. I created https://gitlab.com/fedora/bootc/base-images/-/merge_requests/294 but I'm not sure its the best approach given the discouraged nature of populating /var. OTOH the dangling symlink might also be considered not super nice, idk. Or we could create a stage in osbuild that runs systemd-tmpfiles in a tree, i.e. after we mount the container we run systemd-tmpfiles(?).

[mvo5]

Closing in favor of #1059 which is a much cleaner version of this (I used a separate branch to keep this spike around for reference)