Don't look for host policy with `--source-imageref`

[cgwalters]

This is like #1410 but for SELinux. Basically we shouldn't look at the host system policy

bootc/crates/lib/src/install.rs

Line 631 in 03fa72b

let sepolicy = state.load_policy()?;

until later, and then reconcile the labels on files we wrote using that.

[cgwalters]

The workaround for this is to just install a policy in the buildroot (e.g. selinux-policy-targeted in Fedora).

[elkoniu]

Hi @cgwalters , thanks for the update. I have tried the workaround and it is more complicated.

The 1st thing is that standalone package install is not enough as SELinux removal is part of the Lorax runtime removal template:
https://github.com/weldr/lorax/blob/913cb696fd992e867f6e6ea043293da6dad7ba6e/share/templates.d/99-generic/runtime-cleanup.tmpl#L21

To have it working I have applied custom Lorax template and sideloaded the selinux-policy-targeted rpm package. End result was OS not booting at all:

Thanks to the team debugging effort I was able to boot the image by manually adding selinux=0 to kernel parameters list in grub.

This solved the bootc SELinux policies complain but I reached the next error:

error: Installing to filesystem: Creating ostree deployment: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver

I am not sure if this is some authentication to registry issue (https://access.redhat.com/solutions/7086221) or yet another blocker we need to resolve.

[cgwalters]

Thanks to the team debugging effort I was able to boot the image by manually adding selinux=0 to kernel parameters list in grub.

Hmm...that will have other side effects. Doesn't Anaconda run today in permissive=1? A bit odd that including the policy package would cause this. I guess this does need some digging.

overlay' is not supported over overlayfs,

This is a different issue, it's not related to authentication to the registry. The problem is because Anaconda is mounting an overlayfs for /. Try just typing e.g. podman images - it fails with the same error.

Though Fedora Workstation is set up the same way, but doesn't fail. I am not sure why. It's very clear that /var/lib/containers/storage is an overlay there too which should fail but doesn't...I am not seeing any obvious changes to the podman root storage config to redirect it set up in Workstation.

It's notable though the "LiveOS" approach used for Fedora CoreOS is different because we don't default to writability for / so we make dedicated tmpfs for /etc and /var which avoids overlay-on-overlay consistently.

[cgwalters]

Try just typing e.g. podman images - it fails with the same error.

So a workaround here is mount -t tmpfs tmpfs /var/lib/containers e.g. (probably better done via a systemd .mount unit).

But

It's notable though the "LiveOS" approach used for Fedora CoreOS

To expand on this the really nice thing about the design we have for image-based derivatives is that we have a strong stance that the system should support an empty /var so we can sanely just make a tmpfs for all of var, and that's that.

[elkoniu]

Just for the record - I have tired this with various combinations of selinux=0/1 and enforcing=0/1. So far result is the same.

[elkoniu]

Hi @cgwalters , I have an update about the testing:

[anaconda root@fedora ~]# bootc install to-filesystem --stateroot=default --source-imgref=containers-storage:quay.io/centos-bootc/centos-bootc:stream9 --target-imgref=containers-storage:quay.io/centos-bootc/centos-bootc:stream9 /mnt/sysimage

Installing image: docker://containers-storage:quay.io/centos-bootc/centos-bootc:stream9
Initializing ostree layout
error: Installing to filesystem: Creating ostree deployment: Creating imgstorage: Initializing images: No such file or directory (os error 2)

This is with mount -t tmpfs tmpfs /var/lib/containers:

tmpfs on /var/lib/containers type tmpfs (rw,relatime,inode64)

I have checked content of the /var/lib/containers and tried variants with:

• empty dir
• copy of premounted content

The result is the same.

My other observation is that with mount in place bootc fails faster.
All the above has been tested with selinux=0 kernel parameter.

[elkoniu]

OK, it looks like it may be caused by small amount of RAM on my VM. After changing from 4G to 20G I reached new point:

error: Installing to filesystem: Creating ostree deployment: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]quay.io/centos-bootc/centos-bootc:stream9" does not resolve to an image ID

[jkonecny12]

This is starting to show that we need to store the container image on the disk instead of memory I would say. The RAM requirement might be insane...

[elkoniu]

 56 pub enum Transport {                                                            
 57     /// A remote Docker/OCI registry (`registry:` or `docker://`)               
 58     Registry,                                                                   
 59     /// A local OCI directory (`oci:`)                                          
 60     OciDir,                                                                     
 61     /// A local OCI archive tarball (`oci-archive:`)                            
 62     OciArchive,                                                                 
 63     /// A local Docker archive tarball (`docker-archive:`)                      
 64     DockerArchive,                                                              
 65     /// Local container storage (`containers-storage:`)                                                                                                                
 66     ContainerStorage,                                                           
 67     /// Local directory (`dir:`)                                                
 68     Dir,                                                                        
 69 }

Now I realized I was using wrong repo modifier. Now with this call:

bootc install to-filesystem --stateroot=default --source-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal --target-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal /mnt/sysimage

@cgwalters one question here - bootc expects to have an empty target directory, just to complain later there is no /boot directory presented. Can you elaborate a bit about bootc using the /boot directory? I guess it expects to find a kernel there together with the grub config? I tried with empty /boot directory on target partition and it ends up with:

[anaconda root@fedora ~]# bootc install to-filesystem --stateroot=default --source-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal --target-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal /mnt/sysimage
Installing image: docker://registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal
Initializing ostree layout
layers already present: 0; layers needed: 1 (438.5 MB)
Fetched layers: 418.23 MiB in 55 seconds (7.65 MiB/s)                                                                                                                        Deploying container image: done (9 seconds)                                                                                                                              Running bootupctl to install bootloader
> bootupctl backend install --write-uuid --update-firmware --auto --device /dev/vda /mnt/sysimage
error: Installing to filesystem: Installing bootloader: No such file or directory (os error 2)

We are almost there :)