examples: port the tests to test.thing

Drop our dependency on cockpit-bots (checked out from its git repository
and requiring libvirt and other heavy dependencies) and switch over to
using test.thing (vendored) via pytest.

We can't yet use ephemeral key support for the UKI scenarios because
they enable selinux and there is an outstanding bug preventing sshd from
reading the key: https://bugzilla.redhat.com/show_bug.cgi?id=2374928

For this reason, we keep the old cockpit-tests ssh identity file around
and use it instead.  We do include the workarounds to enable ephemeral
key support in the images to make them easier to use with `test.thing`
interactively.

Expand examples/README.md to describe how this is all intended to be
used.

Signed-off-by: Allison Karlitskaya <[email protected]>

Author: allisonkarlitskaya

.github/workflows/examples.yml

@@ -37,10 +37,12 @@ jobs:
 
     - name: Setup /dev/kvm
       run: |
+        set -eux
         echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm.rules
+        echo 'KERNEL=="vhost-vsock", GROUP="kvm", MODE="0666", OPTIONS+="static_node=vhost-vsock"' | sudo tee /etc/udev/rules.d/99-vhost-vsock.rules
         sudo udevadm control --reload-rules
-        sudo udevadm trigger --name-match=kvm --settle
-        ls -l /dev/kvm
+        sudo udevadm trigger 
+        ls -l /dev/kvm /dev/vhost-vsock
 
     - name: Install dependencies
       run: |
@@ -51,10 +53,8 @@ jobs:
           erofs-utils \
           fsverity \
           mtools \
-          libvirt-daemon \
-          libvirt-daemon-driver-qemu \
-          python3-libvirt \
-          qemu-system \
+          python3-pytest-asyncio \
+          qemu-kvm \
           systemd-boot-efi
         sudo apt-get build-dep systemd e2fsprogs
 
@@ -71,6 +71,9 @@ jobs:
         mkdir ~/bin
         examples/common/install-patched-tools ~/bin
 
+    - name: Install systemd-ssh-proxy polyfill
+      run: sudo cp examples/bls/test-thing.workarounds/systemd-ssh-proxy /usr/lib/systemd
+
     - name: Run example tests
       run: |
         export PATH="${HOME}/bin:${PATH}"

examples/.gitignore

@@ -1,8 +1,9 @@
+/*/*.qcow2
 /*/VARS_CUSTOM.secboot.fd*
 /*/cfsctl
 /*/extra/usr/lib/dracut/modules.d/37composefs/composefs-setup-root
-/*/*.qcow2
 /*/secureboot
 /*/tmp/
 /common/fix-verity/fix-verity.efi
 /test/bots
+__pycache__/

examples/README.md

@@ -12,3 +12,45 @@ a verified operating system image.
  - `unified`: similar to the `uki` example, but avoiding the intermediate `cfsctl` step by running `cfsctl` inside a build stage from the `Containerfile` itself.
    This involves bind-mounting the earlier build stage of the base image so that we can measure it from inside the stage that builds the UKI.
  - `unified-secureboot`: based on the `unified` example, adding signing for Secure Boot.
+
+## Using the examples
+
+The main use of the examples is to act as a scratch space for feature
+development (like initramfs integration) and to show how you can build a system
+image in various configurations using composefs.  They are also run from CI and
+are very useful for local testing, however.
+
+You can build the various images using the `build` script found in each
+subdirectory.  It takes a single argument: the OS to build the image from
+(`fedora`, `rawhide`, `arch`, `ubuntu`, `rhel9`, etc.).  You should not build
+multiple images in parallel due to conflicting feature flags and shared use of
+the tmp/ directory.  After the image is built, you can run tests against it by
+saying something like:
+
+```
+TEST_IMAGE=examples/bls/arch-bls-efi.qcow2 pytest examples/test
+```
+
+Building and running tests on a particular image is supported via the
+`examples/test/run` script, which you can use like:
+
+```
+examples/test/run bls rhel9
+```
+
+The tests are run using [`test.thing`](https://codeberg.org/lis/test.thing). We
+keep a copy of it in-tree.  You you can also use it to run the VM images for
+manual inspection:
+
+```
+examples/testthing.py examples/bls/fedora-bls-efi.qcow2
+```
+
+In that case, you should add this fragment to your ssh configuration:
+
+```
+Host tt.*
+        ControlPath ${XDG_RUNTIME_DIR}/test.thing/%h/ssh
+```
+
+So you can access the test machine via `ssh tt.0` and so on.

examples/pyproject.toml

@@ -0,0 +1,5 @@
+[tool.pytest.ini_options]
+addopts = "-sv"
+asyncio_mode = "auto"
+testpaths = ["test"]
+pythonpath = '.'

examples/test/identity

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpQIBAAKCAQEA1DrTSXQRF8isQQfPfK3U+eFC4zBrjur+Iy15kbHUYUeSHf5S
+jXPYbHYqD1lHj4GJajC9okle9rykKFYZMmJKXLI6987wZ8vfucXo9/kwS6BDAJto
+ZpZSj5sWCQ1PI0Ce8CbkazlTp5NIkjRfhXGP8mkNKMEhdNjaYceO49ilnNCIxhpb
+eH5dH5hybmQQNmnzf+CGCCLBFmc4g3sFbWhI1ldyJzES5ZX3ahjJZYRUfnndoUM/
+TzdkHGqZhL1EeFAsv5iV65HuYbchch4vBAn8jDMmHh8G1ixUCL3uAlosfarZLLyo
+3HrZ8U/llq7rXa93PXHyI/3NL/2YP3OMxE8baQIDAQABAoIBAQCxuOUwkKqzsQ9W
+kdTWArfj3RhnKigYEX9qM+2m7TT9lbKtvUiiPc2R3k4QdmIvsXlCXLigyzJkCsqp
+IJiPEbJV98bbuAan1Rlv92TFK36fBgC15G5D4kQXD/ce828/BSFT2C3WALamEPdn
+v8Xx+Ixjokcrxrdeoy4VTcjB0q21J4C2wKP1wEPeMJnuTcySiWQBdAECCbeZ4Vsj
+cmRdcvL6z8fedRPtDW7oec+IPkYoyXPktVt8WsQPYkwEVN4hZVBneJPCcuhikYkp
+T3WGmPV0MxhUvCZ6hSG8D2mscZXRq3itXVlKJsUWfIHaAIgGomWrPuqC23rOYCdT
+5oSZmTvFAoGBAPs1FbbxDDd1fx1hisfXHFasV/sycT6ggP/eUXpBYCqVdxPQvqcA
+ktplm5j04dnaQJdHZ8TPlwtL+xlWhmhFhlCFPtVpU1HzIBkp6DkSmmu0gvA/i07Z
+pzo5Z+HRZFzruTQx6NjDtvWwiXVLwmZn2oiLeM9xSqPu55OpITifEWNjAoGBANhH
+XwV6IvnbUWojs7uiSGsXuJOdB1YCJ+UF6xu8CqdbimaVakemVO02+cgbE6jzpUpo
+krbDKOle4fIbUYHPeyB0NMidpDxTAPCGmiJz7BCS1fCxkzRgC+TICjmk5zpaD2md
+HCrtzIeHNVpTE26BAjOIbo4QqOHBXk/WPen1iC3DAoGBALsD3DSj46puCMJA2ebI
+2EoWaDGUbgZny2GxiwrvHL7XIx1XbHg7zxhUSLBorrNW7nsxJ6m3ugUo/bjxV4LN
+L59Gc27ByMvbqmvRbRcAKIJCkrB1Pirnkr2f+xx8nLEotGqNNYIawlzKnqr6SbGf
+Y2wAGWKmPyEoPLMLWLYkhfdtAoGANsFa/Tf+wuMTqZuAVXCwhOxsfnKy+MNy9jiZ
+XVwuFlDGqVIKpjkmJyhT9KVmRM/qePwgqMSgBvVOnszrxcGRmpXRBzlh6yPYiQyK
+2U4f5dJG97j9W7U1TaaXcCCfqdZDMKnmB7hMn8NLbqK5uLBQrltMIgt1tjIOfofv
+BNx0raECgYEApAvjwDJ75otKz/mvL3rUf/SNpieODBOLHFQqJmF+4hrSOniHC5jf
+f5GS5IuYtBQ1gudBYlSs9fX6T39d2avPsZjfvvSbULXi3OlzWD8sbTtvQPuCaZGI
+Df9PUWMYZ3HRwwdsYovSOkT53fG6guy+vElUEDkrpZYczROZ6GUcx70=
+-----END RSA PRIVATE KEY-----

examples/test/identity.pub

@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUOtNJdBEXyKxBB898rdT54ULjMGuO6v4jLXmRsdRhR5Id/lKNc9hsdioPWUePgYlqML2iSV72vKQoVhkyYkpcsjr3zvBny9+5xej3+TBLoEMAm2hmllKPmxYJDU8jQJ7wJuRrOVOnk0iSNF+FcY/yaQ0owSF02Nphx47j2KWc0IjGGlt4fl0fmHJuZBA2afN/4IYIIsEWZziDewVtaEjWV3InMRLllfdqGMllhFR+ed2hQz9PN2QcapmEvUR4UCy/mJXrke5htyFyHi8ECfyMMyYeHwbWLFQIve4CWix9qtksvKjcetnxT+WWrutdr3c9cfIj/c0v/Zg/c4zETxtp cockpit-test

examples/test/run

@@ -4,16 +4,8 @@ set -eux
 
 cd "${0%/*}/.."
 
-if [ ! -e test/bots ]; then
-    if [ -h ~/.config/cockpit-dev/bots ]; then
-        ln -sfT "$(realpath --relative-to=test ~/.config/cockpit-dev)/bots" test/bots
-    else
-        git clone https://github.com/cockpit-project/bots test/bots
-    fi
-fi
-
 EXAMPLE="$1"
 OS="$2"
 
 "${EXAMPLE}/build" "${OS}"
-test/run-tests "${EXAMPLE}/${OS}-${EXAMPLE}-efi.qcow2"
+TEST_IMAGE="${EXAMPLE}/${OS}-${EXAMPLE}-efi.qcow2" pytest test

examples/test/run-tests

@@ -0,0 +1,46 @@
+#!/usr/bin/python3
+
+import os
+from collections.abc import AsyncGenerator
+from pathlib import Path
+
+import pytest
+
+import testthing
+
+
+pytest_plugins = ["pytest_asyncio"]
+
+
+@pytest.fixture
+async def machine() -> AsyncGenerator[testthing.VirtualMachine, None]:
+    image = os.getenv("TEST_IMAGE")
+    if not image:
+        raise RuntimeError("TEST_IMAGE environment variable must be set")
+    private = Path(__file__).parent / "identity"
+    private.chmod(0o600)  # ssh will ignore it otherwise
+    with testthing.IpcDirectory() as ipc:
+        async with testthing.VirtualMachine(
+            image=image, ipc=ipc, identity=(private, None), verbose=True
+        ) as vm:
+            yield vm
+
+
+async def test_basic(machine: testthing.VirtualMachine) -> None:
+    m = machine
+
+    # root filesystem is read-only
+    with pytest.raises(testthing.SubprocessError):
+        await m.execute("touch /a")
+
+    # the content of /sysroot is what we expect
+    assert await m.execute("ls /sysroot") == "composefs\nlost+found\nstate\n"
+
+    # make sure /etc and /var persist across a reboot
+    await m.write("/etc/persists.conf", "hihi conf")
+    await m.write("/var/persists.db", "hihi db")
+    await m.reboot()
+    assert await m.execute("cat /etc/persists.conf") == "hihi conf"
+    await m.execute("rm /etc/persists.conf")
+    assert await m.execute("cat /var/persists.db") == "hihi db"
+    await m.execute("rm /var/persists.db")