K8s 1.30 provider slim s390x (kubevirt#1252)

* docs: Updated K8S.md and KUBEVIRTCI_LOCAL_TESTNIG.md for s390x changes

Signed-off-by: chandramerla <[email protected]>

* feat: Updated Centos9 Dockerfile and its CMD script to support s390x arch

- Added conditional execution in Dockerfile based on arch
- Updated Dockerfile CMD script (vm.sh) with qemu-system-s390x and its args as per what are supported on s390x

Signed-off-by: chandramerla <[email protected]>

* feat: Added s390x default kernel args

The defaults here are copied from within whatever are set in the generic cloud image of centos9 for s390x. One can get those values from zipl command under 'kernel parmline'. I've copied them from there to this file to mainly keep parity with x86 based kernel args, which exists for configuring args for kernel. For 'root' key value is set using actual path of file than its UUID.

Signed-off-by: chandramerla <[email protected]>

* feat: gocli and its Dockerfile changes to support s390x

- In gocli codebase, updated hardcoded vagrant username to be fetched calling function, which returns username based on arch.
- Use virtio-net-ccw device incase of s390x Architecture, as virtio-net-pci isn’t supported.
- Skipping provisioning sound cards for s390x as they are not supported.
- Updated method signature of waitForVMToBeUp() to include arg docker client, so that it can be re-used outside run.go
- Bumped docker-registry container image version from 2.7 to 2.8.2 as 2.7 wasn’t supporting s390x

Signed-off-by: chandramerla <[email protected]>

* feat: K8S 1.30 provider provisioning script changes to support s390x

- Updated username to be obtained conditionally based on arch.
- Removed CPU manager related kubelet arguments, as CPU manager args aren’t supported on s390x.
- Installing openvswitch for s390x from kojipkgs.fedoraproject.org, as for s390x same isn’t available from default centos repos
- Explicitly installing gettext rpm to make envsubst command available for s390x environments.

Signed-off-by: chandramerla <[email protected]>

* feat: Publish multiarch manifests for provider related container images

Multi-arch images are enabled for gocli, centos9 and k8s providers

Signed-off-by: chandramerla <[email protected]>

* feat: Updated cluster-up/check.sh for s390x special checks

On an IBM Z system (s390x), huge-page backing storage and nested virtualization cannot be used at the same time.

Ref: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_and_managing_virtualization/creating-nested-virtual-machines_configuring-and-managing-virtualization

Signed-off-by: chandramerla <[email protected]>

* feat: Updated check-cluster-up.sh and beneath scripts to work for s390x and slim mode

As in the case of s390x provider, which currently only supports slim mode, it doesn't have optional componets like CNAO, ISTIO, PROMETHEUS, CDI, etc. enabled and also extra-pre-pull-images is also not used. So, skipping all those components in case of slim mode.
Also using arch speicific manfest/binary files.

Signed-off-by: chandramerla <[email protected]>

* bug: Updated default ARTIFACTS env var to work for local run

As cp command copying  to ARTIFACTS dir was failing in local run with error that source and desnitation are the same file, changed the value of ARTIFACTS when not defined in local run case, unlike in prow jobs where it is defined.

Signed-off-by: chandramerla <[email protected]>

* feat: Updated multi-arch centos9 base image with changes from the PR

As once the PR changes are merged, k8s 1.30 provider build needs latest centos9 base image which is multi-arch, so that provider image can be built for multi-arch(x86, s390x) as well

Signed-off-by: chandramerla <[email protected]>

---------

Signed-off-by: chandramerla <[email protected]>

Author: chandramerla

K8S.md

@@ -48,13 +48,14 @@ make cluster-up
 # Attach to node01 console                           
 docker exec -it ${KUBEVIRT_PROVIDER}-node01 screen /dev/pts/0
 ```                                                 
-Use `vagrant:vagrant` to login.  
+Use `vagrant:vagrant` for x86 and root:root for s390x to login.
 Note: it is sometimes `/dev/pts/1` or `/dev/pts/2`, try them in case you don't get a prompt.
 
 Make sure you don't leave open screens, else the next screen will be messed up.  
 `screen -ls` shows the open screens.  
-`screen -XS <ID> quit` closes an open session.  
+`screen -XS <session-id> quit` closes an open session.
 Close all zombies and shutdown screen gracefully if you plan to open a new one instead.
+Ctrl+A and Ctrl+D will detach your screen session and `screen -r <session-id>` reattach to a detached screen session.
 
 ## Container image cache
 In order to have a local cache of container images:

KUBEVIRTCI_LOCAL_TESTING.md

@@ -21,7 +21,7 @@ cd $KUBEVIRTCI_DIR
 
 ```bash
 # Build a provider. This includes starting it with cluster-up for verification and shutting it down for cleanup.
-(cd cluster-provision/k8s/1.27; ../provision.sh)
+(cd cluster-provision/k8s/1.30; ../provision.sh)
 ```
 
 Note: 
@@ -34,7 +34,7 @@ please use `export BYPASS_PMAN_CHANGE_CHECK=true` to bypass provision-manager ch
 # set local provision test flag (mandatory)
 export KUBEVIRTCI_PROVISION_CHECK=1
 ```
-
+This ensures to set container-registry to quay.io and container-suffix to :latest
 If `KUBEVIRTCI_PROVISION_CHECK` is not used,
 you can set `KUBEVIRTCI_CONTAINER_REGISTRY` (default: `quay.io`), `KUBEVIRTCI_CONTAINER_ORG` (default: `kubevirtci`) and `KUBEVIRTCI_CONTAINER_SUFFIX` (default: according gocli tag),
 in order to use a custom image.
@@ -134,12 +134,16 @@ For that we have phased mode.
 Usage: export the required mode, i.e `export PHASES=linux` or `export PHASES=k8s`
 and then run the provision. the full flow will be:
 
-`export PHASES=linux; (cd cluster-provision/k8s/1.21; ../provision.sh)`  
-`export PHASES=k8s; (cd cluster-provision/k8s/1.21; ../provision.sh)`  
+`export PHASES=linux; (cd cluster-provision/k8s/1.30; ../provision.sh)`  
+`export PHASES=k8s; (cd cluster-provision/k8s/1.30; ../provision.sh)`  
 Run the `k8s` step as much as needed. It reuses the intermediate image that was created
-by the `linux` phase.  
+by the `linux` phase.
+Note :
+1. By default when you run `k8s` phase alone, it uses centos9 image specified in cluster-provision/k8s/base-image, not the one built locally in the `linux` phase. So, to make `k8s` phase use the locally built centos9 image, update cluster-provision/k8s/base-image with the locally built image name and tag (default: quay.io/kubevirtci/centos9:latest)
+2. Also note if you run both `linux,k8s` phases, then it doesn't save the intermediate container image generated post linux image. So, for the centos9 image required for k8s stage, you've to run the linux phase alone.
+
 Once you are done, either check the cluster manually, or use:  
-`export PHASES=k8s; export CHECK_CLUSTER=true; (cd cluster-provision/k8s/1.21; ../provision.sh)`
+`export PHASES=k8s; export CHECK_CLUSTER=true; (cd cluster-provision/k8s/1.30; ../provision.sh)`
 
 ### provision without pre-pulling images
 

cluster-provision/centos9/Dockerfile

@@ -1,33 +1,56 @@
+FROM quay.io/fedora/fedora:39 AS base
 
-FROM quay.io/kubevirtci/fedora@sha256:e3a6087f62f288571db14defb7e0e10ad7fe6f973f567b0488d3aac5e927035a
+RUN dnf -y install jq iptables iproute dnsmasq qemu socat openssh-clients screen bind-utils tcpdump iputils libguestfs-tools-c && dnf clean all
 
-ARG centos_version
 
-RUN dnf -y install jq iptables iproute dnsmasq qemu openssh-clients screen bind-utils tcpdump iputils && dnf clean all
+FROM base AS imageartifactdownload
+
+ARG BUILDARCH
+
+ARG centos_version
 
 WORKDIR /
 
-COPY vagrant.key /vagrant.key
+RUN echo "Centos9 version $centos_version"
 
-RUN chmod 700 vagrant.key
+COPY scripts/download_box.sh /
 
-ENV DOCKERIZE_VERSION v0.6.1
+RUN if test "$BUILDARCH" != "s390x"; then \
+      /download_box.sh https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-Vagrant-9-$centos_version.x86_64.vagrant-libvirt.box && \
+      curl -L -o /initramfs-amd64.img http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/pxeboot/initrd.img && \
+      curl -L -o /vmlinuz-amd64 http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/pxeboot/vmlinuz; \
+    else \
+      /download_box.sh https://cloud.centos.org/centos/9-stream/s390x/images/CentOS-Stream-GenericCloud-9-$centos_version.s390x.qcow2 && \
+      # Access virtual machine disk images directly by using LIBGUESTFS_BACKEND=direct, instead of libvirt
+      export LIBGUESTFS_BACKEND=direct && \
+      guestfish --ro --add box.qcow2 --mount /dev/sda1:/ ls /boot/ | grep -E '^vmlinuz-|^initramfs-' | xargs -I {} guestfish --ro --add box.qcow2 -i copy-out /boot/{} / ; \
+    fi
 
-RUN curl -LO https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
-  && tar -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
-  && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
-  && chmod u+x dockerize \
-  && mv dockerize /usr/local/bin/
 
-COPY scripts/download_box.sh /
+FROM base AS nodecontainer
 
-RUN echo "Centos9 version $centos_version"
+ARG BUILDARCH
+       
+WORKDIR /
+
+COPY vagrant.key /vagrant.key
+
+RUN chmod 700 vagrant.key
 
-ENV CENTOS_URL https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-Vagrant-9-$centos_version.x86_64.vagrant-libvirt.box
+ENV DOCKERIZE_VERSION=v0.8.0
 
-RUN /download_box.sh ${CENTOS_URL}
+RUN if test "$BUILDARCH" != "s390x"; then \
+      curl -L -o dockerize-linux-$BUILDARCH.tar.gz https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz; \
+    else \
+      curl -L -o dockerize-linux-$BUILDARCH.tar.gz https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-s390x-$DOCKERIZE_VERSION.tar.gz; \
+    fi && \
+    tar -xzvf dockerize-linux-$BUILDARCH.tar.gz && \
+    rm dockerize-linux-$BUILDARCH.tar.gz && \
+    chmod u+x dockerize && \
+    mv dockerize /usr/local/bin/
 
-RUN curl -L -o /initrd.img http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/pxeboot/initrd.img
-RUN curl -L -o /vmlinuz http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/pxeboot/vmlinuz
+COPY --from=imageartifactdownload /box.qcow2 box.qcow2
+COPY --from=imageartifactdownload /vmlinuz-* /vmlinuz
+COPY --from=imageartifactdownload /initramfs-* /initrd.img
 
-COPY scripts/* /
+COPY scripts/* /

cluster-provision/centos9/build.sh

@@ -4,4 +4,4 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 
 centos_version="$(cat $DIR/version | tr -d '\n')"
 
-docker build --build-arg centos_version=$centos_version . -t quay.io/kubevirtci/centos9
+docker build --build-arg BUILDARCH=$(uname -m) --build-arg centos_version=$centos_version . -t quay.io/kubevirtci/centos9

cluster-provision/centos9/scripts/download_box.sh

@@ -3,6 +3,13 @@
 set -e
 set -o pipefail
 
-curl -L $1 | tar -zxvf - box.img
-qemu-img convert -O qcow2 box.img box.qcow2
-rm box.img
+ARCH=$(uname -m)
+
+#For the s390x architecture, instead of vagrant box image, generic cloud (qcow2) image is used directly.
+if [ "$ARCH" == "s390x" ]; then
+    curl -L $1 -o box.qcow2
+else
+    curl -L $1 | tar -zxvf - box.img
+    qemu-img convert -O qcow2 box.img box.qcow2
+    rm box.img
+fi

cluster-provision/centos9/scripts/kernel.s390x.args

@@ -0,0 +1 @@
+root=/dev/vda1 ro no_timer_check console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

cluster-provision/centos9/scripts/vm.sh

@@ -7,17 +7,21 @@ MEMORY=3096M
 CPU=2
 NUMA=1
 QEMU_ARGS=""
+QEMU_MONITOR_ARGS=""
 KERNEL_ARGS=""
 NEXT_DISK=""
 BLOCK_DEV=""
 BLOCK_DEV_SIZE=""
+VM_USER=$( [ "$(uname -m)" = "s390x" ] && echo "cloud-user" || echo "vagrant" )
+VM_USER_SSH_KEY="vagrant.key"
 
 while true; do
   case "$1" in
     -m | --memory ) MEMORY="$2"; shift 2 ;;
     -a | --numa ) NUMA="$2"; shift 2 ;;
     -c | --cpu ) CPU="$2"; shift 2 ;;
     -q | --qemu-args ) QEMU_ARGS="${2}"; shift 2 ;;
+    -qm | --qemu-monitor-args ) QEMU_MONITOR_ARGS="${2}"; shift 2 ;;
     -k | --additional-kernel-args ) KERNEL_ARGS="${2}"; shift 2 ;;
     -n | --next-disk ) NEXT_DISK="$2"; shift 2 ;;
     -b | --block-device ) BLOCK_DEV="$2"; shift 2 ;;
@@ -39,6 +43,12 @@ function calc_next_disk {
   if [ -n "$NEXT_DISK" ]; then next=${NEXT_DISK}; fi
   if [ "$last" = "00" ]; then
     last="box.qcow2"
+    # Customize qcow2 image using virt-sysprep (with KVM accelerator)
+    if [ "$(uname -m)" = "s390x" ]; then
+      export LIBGUESTFS_BACKEND=direct
+      export LIBGUESTFS_BACKEND_SETTINGS=force_kvm
+      virt-sysprep -a box.qcow2 --run-command 'useradd -m cloud-user' --append '/etc/cloud/cloud.cfg:runcmd:' --append '/etc/cloud/cloud.cfg: - hostnamectl set-hostname ""' --root-password password:root --ssh-inject cloud-user:string:"ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key"
+    fi
   else
     last=$(printf "/disk%02d.qcow2" $last)
   fi
@@ -51,7 +61,7 @@ cat >/usr/local/bin/ssh.sh <<EOL
 #!/bin/bash
 set -e
 dockerize -wait tcp://192.168.66.1${n}:22 -timeout 300s &>/dev/null
-ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no vagrant@192.168.66.1${n} -i vagrant.key -p 22 -q \$@
+ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ${VM_USER}@192.168.66.1${n} -i ${VM_USER_SSH_KEY} -p 22 -q \$@
 EOL
 chmod u+x /usr/local/bin/ssh.sh
 echo "done" >/ssh_ready
@@ -195,15 +205,79 @@ if [ "${NUMA}" -gt 1 ]; then
     done
 fi
 
-exec qemu-system-x86_64 -enable-kvm -drive format=qcow2,file=${next},if=virtio,cache=unsafe ${block_dev_arg} \
-  -device virtio-net-pci,netdev=network0,mac=52:55:00:d1:55:${n} \
-  -netdev tap,id=network0,ifname=tap${n},script=no,downscript=no \
-  -device virtio-rng-pci \
-  -initrd /initrd.img \
-  -kernel /vmlinuz \
-  -append "$(cat /kernel.args) $(cat /additional.kernel.args) ${KERNEL_ARGS}" \
-  -vnc :${n} -cpu host,migratable=no,+invtsc -m ${MEMORY} -smp ${CPU} ${numa_arg} \
-  -serial pty -M q35,accel=kvm,kernel_irqchip=split \
-  -device intel-iommu,intremap=on,caching-mode=on -device intel-hda -device hda-duplex -device AC97 \
-  -uuid $(cat /proc/sys/kernel/random/uuid) \
-  ${QEMU_ARGS}
+if [ "$(uname -m)" == "s390x" ]; then
+  # As per https://www.qemu.org/docs/master/system/s390x/bootdevices.html#booting-without-bootindex-parameter -drive if=virtio can't be specified with bootindex for s390x
+  qemu_system_cmd="qemu-system-s390x \
+    -enable-kvm \
+    -drive format=qcow2,file=${next},if=none,cache=unsafe,id=drive1 ${block_dev_arg} \
+    -device virtio-blk,drive=drive1,bootindex=1 \
+    -device virtio-net-ccw,netdev=network0,mac=52:55:00:d1:55:${n} \
+    -netdev tap,id=network0,ifname=tap${n},script=no,downscript=no \
+    -device virtio-rng \
+    -initrd /initrd.img \
+    -kernel /vmlinuz \
+    -append \"$(cat /kernel.s390x.args) $(cat /additional.kernel.args) ${KERNEL_ARGS}\" \
+    -vnc :${n} \
+    -cpu host \
+    -m ${MEMORY} \
+    -smp ${CPU} ${numa_arg} \
+    -serial pty \
+    -machine s390-ccw-virtio,accel=kvm \
+    -uuid $(cat /proc/sys/kernel/random/uuid) \
+    -monitor unix:/tmp/qemu-monitor.sock,server,nowait \
+    ${QEMU_ARGS}"
+else
+  #Docs: https://www.qemu.org/docs/master/system/invocation.html
+  qemu_system_cmd="qemu-system-x86_64 \
+    -enable-kvm \
+    -drive format=qcow2,file=${next},if=virtio,cache=unsafe ${block_dev_arg} \
+    -device virtio-net-pci,netdev=network0,mac=52:55:00:d1:55:${n} \
+    -netdev tap,id=network0,ifname=tap${n},script=no,downscript=no \
+    -device virtio-rng-pci \
+    -initrd /initrd.img \
+    -kernel /vmlinuz \
+    -append \"$(cat /kernel.args) $(cat /additional.kernel.args) ${KERNEL_ARGS}\" \
+    -vnc :${n} \
+    -cpu host,migratable=no,+invtsc \
+    -m ${MEMORY} \
+    -smp ${CPU} ${numa_arg} \
+    -serial pty \
+    -machine q35,accel=kvm,kernel_irqchip=split \
+    -device intel-iommu,intremap=on,caching-mode=on \
+    -device intel-hda \
+    -device hda-duplex \
+    -device AC97 \
+    -uuid $(cat /proc/sys/kernel/random/uuid) \
+    -monitor unix:/tmp/qemu-monitor.sock,server,nowait \
+    ${QEMU_ARGS}"
+fi
+
+PID=0
+eval "nohup $qemu_system_cmd &"
+PID=$!
+
+# Function to check if QEMU monitor socket is ready
+is_qemu_monitor_ready() {
+  socat - UNIX-CONNECT:/tmp/qemu-monitor.sock < /dev/null > /dev/null 2>&1
+}
+
+# Wait for the QEMU monitor socket to be ready
+elapsed=0
+while ! is_qemu_monitor_ready; do
+  if [ $elapsed -ge 60 ]; then
+    echo "QEMU monitor socket did not become available within 60 seconds."
+    exit 1
+  fi
+  sleep 1
+  elapsed=$((elapsed + 1))
+done
+echo "QEMU monitor socket is ready."
+
+if [[ -n "$QEMU_MONITOR_ARGS" ]]; then
+  IFS=';' read -ra ADDR <<< "$QEMU_MONITOR_ARGS"
+  for QEMU_MONITOR_CMD in "${ADDR[@]}"; do
+      echo "$QEMU_MONITOR_CMD" | socat - UNIX-CONNECT:/tmp/qemu-monitor.sock
+  done
+fi
+
+wait $PID

cluster-provision/gocli/Makefile

@@ -2,6 +2,7 @@ SHELL := /bin/bash
 
 IMAGES_FILE ?= images.json
 KUBEVIRTCI_IMAGE_REPO ?= quay.io/kubevirtci
+GOARCH ?= $$(uname -m | grep -q s390x && echo s390x || echo amd64)
 
 export GO111MODULE=on
 export GOPROXY=direct
@@ -23,7 +24,7 @@ test-gocli:
 
 .PHONY: gocli
 cli:
-	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 $(GO) build -ldflags "-X 'kubevirt.io/kubevirtci/cluster-provision/gocli/images.SUFFIX=:$(KUBEVIRTCI_TAG)'" -o $(BIN_DIR)/cli ./cmd/cli
+	CGO_ENABLED=0 GOOS=linux GOARCH=${GOARCH} $(GO) build -ldflags "-X 'kubevirt.io/kubevirtci/cluster-provision/gocli/images.SUFFIX=:$(KUBEVIRTCI_TAG)'" -o $(BIN_DIR)/cli ./cmd/cli
 .PHONY: fmt
 fmt:
 	$(GO) fmt ./cmd/...

cluster-provision/gocli/cmd/provision.go

@@ -6,6 +6,7 @@ import (
 	"os"
 	"os/signal"
 	"path/filepath"
+	"runtime"
 	"strconv"
 	"strings"
 
@@ -23,6 +24,7 @@ import (
 	containers2 "kubevirt.io/kubevirtci/cluster-provision/gocli/containers"
 
 	"kubevirt.io/kubevirtci/cluster-provision/gocli/cmd/utils"
+	"kubevirt.io/kubevirtci/cluster-provision/gocli/pkg/libssh"
 	"kubevirt.io/kubevirtci/cluster-provision/gocli/docker"
 )
 
@@ -51,6 +53,7 @@ func NewProvisionCommand() *cobra.Command {
 
 func provisionCluster(cmd *cobra.Command, args []string) (retErr error) {
 	var base string
+	sshUser := libssh.GetUserByArchitecture(runtime.GOARCH)
 	packagePath := args[0]
 	versionBytes, err := os.ReadFile(filepath.Join(packagePath, "version"))
 	if err != nil {
@@ -228,13 +231,13 @@ func provisionCluster(cmd *cobra.Command, args []string) (retErr error) {
 	}
 
 	// Wait for ssh.sh script to exist
+	logrus.Info("Wait for ssh.sh script to exist")
 	err = _cmd(cli, nodeContainer(prefix, nodeName), "while [ ! -f /ssh_ready ] ; do sleep 1; done", "checking for ssh.sh script")
 	if err != nil {
 		return err
 	}
 
-	// Wait for the VM to be up
-	err = _cmd(cli, nodeContainer(prefix, nodeName), "ssh.sh echo VM is up", "waiting for node to come up")
+	err = waitForVMToBeUp(cli, prefix, nodeName)
 	if err != nil {
 		return err
 	}
@@ -252,21 +255,21 @@ func provisionCluster(cmd *cobra.Command, args []string) (retErr error) {
 		if err != nil {
 			return err
 		}
-		err = _cmd(cli, nodeContainer(prefix, nodeName), "if [ -f /scripts/extra-pre-pull-images ]; then scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/extra-pre-pull-images vagrant@192.168.66.101:/tmp/extra-pre-pull-images; fi", "copying /scripts/extra-pre-pull-images if existing")
+		err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("if [ -f /scripts/extra-pre-pull-images ]; then scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/extra-pre-pull-images %s@192.168.66.101:/tmp/extra-pre-pull-images; fi", sshUser), "copying /scripts/extra-pre-pull-images if existing")
 		if err != nil {
 			return err
 		}
-		err = _cmd(cli, nodeContainer(prefix, nodeName), "if [ -f /scripts/fetch-images.sh ]; then scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/fetch-images.sh vagrant@192.168.66.101:/tmp/fetch-images.sh; fi", "copying /scripts/fetch-images.sh if existing")
+		err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("if [ -f /scripts/fetch-images.sh ]; then scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/fetch-images.sh %s@192.168.66.101:/tmp/fetch-images.sh; fi", sshUser), "copying /scripts/fetch-images.sh if existing")
 		if err != nil {
 			return err
 		}
 
-		err = _cmd(cli, nodeContainer(prefix, nodeName), "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key vagrant@192.168.66.101 'mkdir -p /tmp/ceph /tmp/cnao /tmp/nfs-csi /tmp/nodeports /tmp/prometheus /tmp/whereabouts /tmp/kwok'", "Create required manifest directories before copy")
+		err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key %s@192.168.66.101 'mkdir -p /tmp/ceph /tmp/cnao /tmp/nfs-csi /tmp/nodeports /tmp/prometheus /tmp/whereabouts /tmp/kwok'", sshUser), "Create required manifest directories before copy")
 		if err != nil {
 			return err
 		}
 		// Copy manifests to the VM
-		err = _cmd(cli, nodeContainer(prefix, nodeName), "scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/manifests/* vagrant@192.168.66.101:/tmp", "copying manifests to the VM")
+		err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/manifests/* %s@192.168.66.101:/tmp", sshUser), "copying manifests to the VM")
 		if err != nil {
 			return err
 		}