examples: port the tests to test.thing

Drop our dependency on cockpit-bots (checked out from its git repository
and requiring libvirt and other heavy dependencies) and switch over to
using test.thing (vendored) via pytest.

We no longer install ssh keys into the images: test.thing generates an
ephemeral key on each run and feeds it into the guest.

Expand examples/README.md to describe how this is all intended to be
used.

Adjust our github workflows appropriately.  The systemd version on the
runner isn't new enough to have systemd-ssh-proxy, so install our
polyfill.  We also need to make sure the vhost-vsock is accessible to
the user in the same way as /dev/kvm.

Signed-off-by: Allison Karlitskaya <[email protected]>

Author: allisonkarlitskaya

.github/workflows/examples.yml

@@ -37,10 +37,12 @@ jobs:
 
     - name: Setup /dev/kvm
       run: |
+        set -eux
         echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm.rules
+        echo 'KERNEL=="vhost-vsock", GROUP="kvm", MODE="0666", OPTIONS+="static_node=vhost-vsock"' | sudo tee /etc/udev/rules.d/99-vhost-vsock.rules
         sudo udevadm control --reload-rules
-        sudo udevadm trigger --name-match=kvm --settle
-        ls -l /dev/kvm
+        sudo udevadm trigger --settle
+        ls -l /dev/kvm /dev/vhost-vsock
 
     - name: Install dependencies
       run: |
@@ -51,10 +53,8 @@ jobs:
           erofs-utils \
           fsverity \
           mtools \
-          libvirt-daemon \
-          libvirt-daemon-driver-qemu \
-          python3-libvirt \
-          qemu-system \
+          python3-pytest-asyncio \
+          qemu-kvm \
           systemd-boot-efi
 
     - name: Get a newer podman for heredoc support (from plucky)
@@ -80,6 +80,9 @@ jobs:
           examples/common/install-patched-tools ~/bin
         fi
 
+    - name: Install systemd-ssh-proxy polyfill
+      run: sudo cp examples/bls/test-thing.workarounds/systemd-ssh-proxy /usr/lib/systemd
+
     - name: Run example tests
       run: |
         export PATH="${HOME}/bin:${PATH}"

examples/.gitignore

@@ -1,8 +1,9 @@
+/*/*.qcow2
 /*/VARS_CUSTOM.secboot.fd*
 /*/cfsctl
 /*/extra/usr/lib/dracut/modules.d/37composefs/composefs-setup-root
-/*/*.qcow2
 /*/secureboot
 /*/tmp/
 /common/fix-verity/fix-verity.efi
 /test/bots
+__pycache__/

examples/README.md

@@ -12,3 +12,45 @@ a verified operating system image.
  - `unified`: similar to the `uki` example, but avoiding the intermediate `cfsctl` step by running `cfsctl` inside a build stage from the `Containerfile` itself.
    This involves bind-mounting the earlier build stage of the base image so that we can measure it from inside the stage that builds the UKI.
  - `unified-secureboot`: based on the `unified` example, adding signing for Secure Boot.
+
+## Using the examples
+
+The main use of the examples is to act as a scratch space for feature
+development (like initramfs integration) and to show how you can build a system
+image in various configurations using composefs.  They are also run from CI and
+are very useful for local testing, however.
+
+You can build the various images using the `build` script found in each
+subdirectory.  It takes a single argument: the OS to build the image from
+(`fedora`, `rawhide`, `arch`, `ubuntu`, `rhel9`, etc.).  You should not build
+multiple images in parallel due to conflicting feature flags and shared use of
+the tmp/ directory.  After the image is built, you can run tests against it by
+saying something like:
+
+```
+TEST_IMAGE=examples/bls/arch-bls-efi.qcow2 pytest examples/test
+```
+
+Building and running tests on a particular image is supported via the
+`examples/test/run` script, which you can use like:
+
+```
+examples/test/run bls rhel9
+```
+
+The tests are run using [`test.thing`](https://codeberg.org/lis/test.thing). We
+keep a copy of it in-tree.  You you can also use it to run the VM images for
+manual inspection:
+
+```
+examples/testthing.py examples/bls/fedora-bls-efi.qcow2
+```
+
+In that case, you should add this fragment to your ssh configuration:
+
+```
+Host tt.*
+        ControlPath ${XDG_RUNTIME_DIR}/test.thing/%h/ssh
+```
+
+So you can access the test machine via `ssh tt.0` and so on.

examples/bls/build

@@ -60,7 +60,7 @@ if [ "${FS_VERITY_MODE:-repart}" = "none" ]; then
     CFSCTL="$CFSCTL --insecure"
 fi
 
-${CFSCTL} oci prepare-boot "${BASE_ID}" --bootdir tmp/efi --cmdline console=ttyS0,115200 --entry-id=example --cmdline rw
+${CFSCTL} oci prepare-boot "${BASE_ID}" --bootdir tmp/efi --entry-id=example --cmdline rw
 
 ../common/install-systemd-boot
 ../common/make-image "${os}-bls-efi.qcow2"

examples/bls/extra/root/.ssh/authorized_keys

@@ -0,0 +1,5 @@
+[tool.pytest.ini_options]
+addopts = "--strict-markers -v"
+asyncio_mode = "auto"
+pythonpath = "."
+testpaths = ["test"]

examples/pyproject.toml

@@ -4,16 +4,8 @@ set -eux
 
 cd "${0%/*}/.."
 
-if [ ! -e test/bots ]; then
-    if [ -h ~/.config/cockpit-dev/bots ]; then
-        ln -sfT "$(realpath --relative-to=test ~/.config/cockpit-dev)/bots" test/bots
-    else
-        git clone https://github.com/cockpit-project/bots test/bots
-    fi
-fi
-
 EXAMPLE="$1"
 OS="$2"
 
 "${EXAMPLE}/build" "${OS}"
-test/run-tests "${EXAMPLE}/${OS}-${EXAMPLE}-efi.qcow2"
+TEST_IMAGE="${EXAMPLE}/${OS}-${EXAMPLE}-efi.qcow2" pytest test

examples/test/run

@@ -0,0 +1,43 @@
+#!/usr/bin/python3
+
+import os
+from collections.abc import AsyncGenerator
+
+import pytest
+
+import testthing
+
+
+@pytest.fixture
+async def machine() -> AsyncGenerator[testthing.VirtualMachine, None]:
+    image = os.getenv("TEST_IMAGE")
+    if not image:
+        raise RuntimeError("TEST_IMAGE environment variable must be set")
+    with testthing.IpcDirectory() as ipc:
+        async with testthing.VirtualMachine(image=image, ipc=ipc, verbose=True) as vm:
+            yield vm
+
+
+async def test_basic(machine: testthing.VirtualMachine) -> None:
+    m = machine
+
+    # root filesystem is read-only
+    with pytest.raises(testthing.SubprocessError):
+        await m.execute("touch /a")
+
+    # the content of /sysroot is what we expect
+    expected = set[str](("composefs", "state"))
+    if os.getenv("FS_FORMAT", "") in ("ext4", ""):
+        expected.add("lost+found")
+
+    output = await m.execute("ls /sysroot")
+    assert set(output.splitlines()) == expected
+
+    # make sure /etc and /var persist across a reboot
+    await m.write("/etc/persists.conf", "hihi conf")
+    await m.write("/var/persists.db", "hihi db")
+    await m.reboot()
+    assert await m.execute("cat /etc/persists.conf") == "hihi conf"
+    await m.execute("rm /etc/persists.conf")
+    assert await m.execute("cat /var/persists.db") == "hihi db"
+    await m.execute("rm /var/persists.db")