exttests: New container image with upstream tests

This is aiming to finally, finally close the loop around
running upstream tests as part of our CI.

We landed support for "exttests" in here for a while and
since then we are successfully running upstream test suites
using kola on upstream pull requests.

But we also want the inverse: run upstream tests on builds outside of PRs to those repos.
For example,
I really want to run ostree's "transactionality" test on FCOS builds so that if a kernel change breaks it, we know that.

In practice, it's likely that down the line we
will need to version at least some of our tests along
with the OS content - by building them as RPMs or
sticking them in containers that are versioned with the OS
or something.

But for now, building fixed upstream releases will
give us a lot of reproducibility/control, at the
cost of needing to periodically bump them.

Author: cgwalters

exttests/Dockerfile

@@ -0,0 +1,6 @@
+FROM registry.svc.ci.openshift.org/coreos/cosa-buildroot:latest as builder
+COPY build.sh .
+RUN ./build.sh
+
+FROM quay.io/coreos-assembler/coreos-assembler:latest
+COPY --from=builder /usr/lib/coreos-assembler/tests/kola/ /usr/lib/coreos-assembler/tests/kola/

exttests/README.md

@@ -0,0 +1,31 @@
+# POC for gathering upstream tests into a container
+
+We landed support for "exttests" in coreos-assembler:
+https://github.com/coreos/coreos-assembler/blob/98d40e6bb13adc02bcd5f02f1d5bff7bafa0780d/mantle/kola/README-kola-ext.md
+
+Since then we are successfully running upstream test suites using
+kola on upstream pull requests.
+
+But we also want the inverse: run upstream tests on builds outside
+of PRs to those repos.
+
+For example, I really want to run ostree's "transactionality"
+test on FCOS builds so that if a kernel change breaks it, we
+know that.
+
+## Proposal
+
+- Build this container in api.ci like we're building the cosa buildroot;
+  currently done at `registry.svc.ci.openshift.org/cgwalters/cosa-exttests:latest`
+- Change the FCOS/RHCOS pipelines to pull this container and do: `kola run ... ext.*`
+  (This raises some issues like the fact that we'd need to store/share the images
+   in a way that would allow a secondary container to access them)
+
+### Alternative: coreos-assembler only
+
+Fold this into coreos-assembler.
+
+### Alternative: Make RPMs of tests and install those in coreos-assembler
+
+We have an `ostree-tests` RPM...but I'm not sure I want to deal with
+packaging for it.

exttests/build.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+set -xeuo pipefail
+declare -A repos
+# Pin versions for now for reproducibility
+repos[ostreedev/ostree]=v2020.6
+repos[coreos/rpm-ostree]=v2020.5
+for repo in "${!repos[@]}"; do
+    bn=$(basename ${repo})
+    git clone --depth=1 -b ${repos[${repo}]} https://github.com/${repo} ${bn}
+    # Our "standard" for ext-installed-tests
+    cd ${bn}/tests/kolainst
+    make -j 4
+    make install
+    cd -
+    rm ${bn} -rf
+done