Skip to content

How To: Setup a Dev Environment (Minikube)

Jump to bottom
Joe McCormick edited this page Nov 1, 2024 · 6 revisions

Note this will not attempt to harden your Docker or Minikube configuration in any way. These steps should only be used to develop and test the driver in an already secure isolated environment.

Purpose:

Quick start guide to setting up a development environment to work on and test the driver.

Prerequisites:

  • Ubuntu 22.04

Steps:

  • Install the BeeGFS client.
  • Install Docker Engine
  • Install Minikube
  • Add your user to the docker group: sudo usermod -aG docker $USER && newgrp docker
  • Start Minikube: minikube start
  • Install prerequisites into Minikube container (adjust BeeGFS version as needed):
minikube ssh
sudo su
apt-get update
apt-get install wget -y
wget -q https://www.beegfs.io/release/beegfs_7.4.5/gpg/GPG-KEY-beegfs -O- | apt-key add -
wget -P /etc/apt/sources.list.d/ https://www.beegfs.io/release/beegfs_7.4.5/dists/beegfs-focal.list
apt-get update
apt-get install beegfs-utils -y
wget -P /etc/beegfs/ https://raw.githubusercontent.com/ThinkParQ/beegfs/master/client_module/build/dist/etc/beegfs-client.conf
exit
exit
  • (optional) If required install kubectl and setup an symbolic link:
    • minikube kubectl -- get pods -A
    • sudo ln -s $(which minikube) /usr/local/bin/kubectl (note simply setting up an alias doesn't work with the OLM setup script)
  • Verify installation: kubectl get pod -A
  • (optional) If you want to work on the operator:

Configuration when Minikube and BeeGFS services are running on the same machine.

  • Ensure the BeeGFS services running on your host are either be bound to all IP’s (0.0.0.0) and interfaces, or to the IP and interface your VM is bridged against (e.g., br-33bbfd2e50bd).
    • If the service is bound only to localhost (127.0.0.1), this will not work (reference). For BeeGFS this is configured using the connInterface* settings (just don't set it to bind to all interfaces).
  • Then anywhere the BeeGFS management IP/hostname is configured, use the IP assigned to the bridge interface. Typically either the sysMgmtdHost in storage classes (e.g., dyn-sc.yaml) or in the volumeHandle for statically provisioned volumes (e.g., static-pv.yaml).
    • Note alternatively you could also use host.minikube.internal instead of the IP but then you would need to setup an /etc/hosts/ entry in the base OS/host machine for the IP.

Development Process

While the developer docs suggest pushing test containers to some registry that are then pulled into your test K8s cluster, if you're working on the driver on the same machine where Minikube is installed you can skip pushing the containers to an external registry by doing the following:

  • Point your terminal to use the docker daemon inside Minikube run eval $(minikube docker-env) (reference).
  • Copy the default developer K8s deployment manifest from deploy/k8s/overlays/dev-default to deploy/k8s/overlays/dev and modify dev/patches/image-pull-policy.yaml so that imagePullPolicy: Never (for both the StatefulSet and DaemonSet.

Now you can just modify/rebuild the driver then directly build containers with make container and skip pushing them to an external registry before running kubectl apply -k deploy/k8s/overlays/dev/.

Clone this wiki locally