dockerizing hyperd

[matti]

I'd like to run hyperd inside a docker image for deployment reasons.

I've managed to get it working by adding lots of privileges (that is okay for my scenario):

$ docker run --privileged --net=host -v /dev:/dev --cap-add=ALL -it myimage bash
$ hyperd >/dev/null &
$ hyperctl run -t alpine:3.2 uptime
 09:42:41 up  ....

The hyperd prints out following errors:

WARN[0000] devmapper: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors. For more information, see https://docs.docker.com/reference/commandline/daemon/#daemon-storage-driver-option
WARN[0000] devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section.
WARN[0000] devmapper: XFS is not supported in your system. Either the kernel doesnt support it or mkfs.xfs is not in your PATH. Defaulting to ext4 filesystem
INFO[0000] devmapper: Creating filesystem ext4 on device docker-0:69-235-base
INFO[0001] devmapper: Successfully created filesystem ext4 on device docker-0:69-235-base
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support swap memory limit.
INFO[0001] Loading containers: start.

INFO[0001] Loading containers: done.
E0406 09:43:16.823246       1 dm.go:193] losetup: /var/lib/hyper/lib/data: failed to set up loop device: Device or resource busy

My Dockerfile is as follows:

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND=noninteractive
ENV GOPATH=/root/go

RUN apt-get update && apt-get upgrade -y

RUN apt-get install -y software-properties-common && \
    add-apt-repository -y ppa:longsleep/golang-backports && \
    apt-get update && \
    apt-get install -y golang-1.8 && \
    ln -s /usr/lib/go-1.8/bin/go /usr/local/bin/go

WORKDIR /root/go/src/github.com/hyperhq

RUN apt-get install -y git
RUN git clone https://github.com/hyperhq/hyperd.git hyperd  && \
    git clone https://github.com/hyperhq/hyperstart.git hyperstart

RUN apt-get install -y \
  autotools-dev libdevmapper-dev libsqlite3-dev libvirt-dev automake \
  cpio


WORKDIR /root/go/src/github.com/hyperhq/hyperd


#TODO: btrfs is yes, but:
#checking btrfs/ioctl.h, usability... no
#checking btrfs/ioctl.h, presence... no
#checking for btrfs/ioctl.h,... no

RUN apt-get install -y gawk btrfs-tools
RUN ./autogen.sh && \
    ./configure --without-xen  && \
    make

RUN cp hyperd /usr/local/bin && \
    cp hyperctl /usr/local/bin && \
    mkdir -p /etc/hyper && \
    cp package/dist/etc/hyper/config /etc/hyper

WORKDIR /root/go/src/github.com/hyperhq/hyperstart

RUN ./autogen.sh && \
    ./configure  && \
    make

    RUN mkdir -p /var/lib/hyper && \
        cp build/kernel /var/lib/hyper && \
        cp build/hyper-initrd.img /var/lib/hyper

    RUN apt-get install -y \
        aufs-tools libvirt-bin qemu-kvm

#TODO: move
#E0406 08:48:28.327679      16 dm.go:222] /bin/sh: 1: dmsetup: not found
RUN apt-get install -y dmsetup

# WARN[0001] Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-66-generic/modules.dep.bin'
RUN apt-get install -y kmod
RUN apt-get install -y linux-image-4.4.0-66-generic

How could I get rid of those errors?

[matti]

(related #467)

[gnawux]

Tried but give up -- It is too hard to operate low level storage functionalities inside docker container.

[matti]

any pointers where to look for a solution?

[gao-feng]

Is /dev/loop6 and /dev/loop7 used in your host? hyperd devicemapper storage driver uses these two devices by default.

https://github.com/hyperhq/hyperd/blob/master/daemon/storage.go#L108
https://github.com/hyperhq/hyperd/blob/master/storage/defaults.go#L6

[matti]

Okay, so this seems to work:

docker run -it --privileged --net=host -v /var/lib/hyper myimage hyperd

So: -v /dev:/dev is not needed and also --cap-add=ALL is not (because of --privileged)

IMO this looks very promising, it's so convenient to run hyperd as a docker image!

[gnawux]

We should release an official image for hyperd with the v0.9.0 release

[neerdoc]

@matti Did you benchmark your setup? I followed this to create a hyper-in-docker (hid) installation of the 0.8.1 version and it worked fine, but when running a simple benchmark I get very low disk-IO performance. I.e., 'tar xf linux-4.12.5.tar.xz' takes around 15 seconds on host, also around 15 seconds in docker but around 200 seconds on hid. All my setup uses btrfs (host, docker and hyperd).

[matti]

@neerdoc didn’t do any IO benchmarks. Just compared startup times (ran uptime from alpine image) - from around 6s on GCP, virtualbox to 1.7s with baremetal kabylake 3.8ghz..

[kfox1111]

Where did you do the benchmark? Running the tar within the container at /var/lib/hyper with the passed through volume will perform vastly differently then in the rest of the container as it will be doing cow operations.