merge virtme into kernelcraft

Maintain virtme code directly inside kernelcraft.

The source code of virtme has been imported from:
https://github.com/arighi/virtme

Signed-off-by: Andrea Righi <[email protected]>

DCO-1.1.txt

@@ -0,0 +1,29 @@
+The text 'Signed-off-by:' in a commit message indicates that the signer
+agrees to the Developer's Certificate of Origin 1.1, reproduced below.
+
+Developer's Certificate of Origin 1.1
+
+By making a contribution to this project, I certify that:
+
+(a) The contribution was created in whole or in part by me and I
+    have the right to submit it under the open source license
+    indicated in the file; or
+
+(b) The contribution is based upon previous work that, to the best
+    of my knowledge, is covered under an appropriate open source
+    license and I have the right under that license to submit that
+    work with modifications, whether created in whole or in part
+    by me, under the same open source license (unless I am
+    permitted to submit under a different license), as indicated
+    in the file; or
+
+(c) The contribution was provided directly to me by some other
+    person who certified (a), (b) or (c) and I have not modified
+    it.
+
+(d) I understand and agree that this project and the contribution
+    are public and that a record of the contribution (including all
+    personal information I submit with it, including my sign-off) is
+    maintained indefinitely and may be redistributed consistent with
+    this project or the open source license(s) involved.
+

README.md

@@ -21,6 +21,9 @@ Kernels produced with KernelCraft are lacking lots of features, in order to
 reduce the build time to the minimum and still provide you a usable kernel
 capable of running your tests and experiments.
 
+KernelCraft is based on virtme, written by Andy Lutomirski <[email protected]>
+([web][korg-web] | [git][korg-git]).
+
 Quick start
 ===========
 
@@ -60,18 +63,30 @@ repository):
 Requirements
 ============
 
-KernelCraft requires a customized version of virtme, available here
-[web][arighi-virtme].
+ * You need Python 3.8 or higher
 
-You may also need to install `crash` to use the memory dump inspection feature
-(see example below).
+ * QEMU 1.6 or higher is recommended (QEMU 1.4 and 1.5 are partially supported
+   using a rather ugly kludge)
+   * You will have a much better experience if KVM is enabled.  That means that
+     you should be on bare metal with hardware virtualization (VT-x or SVM)
+     enabled or in a VM that supports nested virtualization.  On some Linux
+     distributions, you may need to be a member of the "kvm" group.  Using
+     VirtualBox or most VPS providers will fall back to emulation.
 
-If you are using Ubuntu you can install all the required packages and dependencies from this ppa:
-[web][kernelcraft-ppa].
+ * Depending on the options you use, you may need a statically linked `busybox`
+   binary somewhere in your path.
+
+ * You may need to install `crash` to use the memory dump inspection feature
+   (see example below).
 
 Examples
 ========
 
+ - Build and run a kernel from a local git repository:
+```
+   $ kc
+```
+
  - Build and run v6.1-rc3 from the public mainline git repository:
 ```
    $ kc -r mainline -c v6.1-rc3
@@ -192,16 +207,16 @@ virtme-configkernel.
 
 It is possible to specify a set of custom configs (.config chunk) in
 ~/.kc.config, these user-specific settings will override the default settings
-of virtme-configkernel (except for the mandatory configs that are required to
-boot and test the kernel inside qemu, using virtme).
+of virtme-configkernel (except for the mandatory configs that are required to boot
+and test the kernel inside qemu, using virtme-run).
 
 Then the kernel is compiled either locally or on an external build host (if the
 `--build-host` option is used); once the build is done only the required files
 needed to test the kernel are copied from the remote host if an external build
 host is used.
 
-Then the kernel is executed using virtme. This allows to test the kernel using
-a safe copy-on-write snapshot of the entire host filesystem.
+Then the kernel is executed using the virtme module. This allows to test the
+kernel using a safe copy-on-write snapshot of the entire host filesystem.
 
 All the kernels compiled with KernelCraft have a `-rc` suffix to their kernel
 version, this allows to easily determine if you're inside a KernelCraft kernel
@@ -281,6 +296,11 @@ Troubleshooting
   $ kc --clean --build-host HOSTNAME
 ```
 
+Contributing
+============
+
+Please see DCO-1.1.txt.
+
 Credits
 =======
 
@@ -291,5 +311,5 @@ KernelCraft is based on virtme, written by Andy Lutomirski <[email protected]>
 
 [korg-web]: https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git "virtme on kernel.org"
 [korg-git]: git://git.kernel.org/pub/scm/utils/kernel/virtme/virtme.git "git address"
-[arighi-virtme]: https://github.com/arighi/virtme "arighi virtme"
+[virtme]: https://github.com/amluto/virtme "virtme"
 [kernelcraft-ppa]: https://launchpad.net/~arighi/+archive/ubuntu/kernelcraft "kernelcraft ppa"

kc → bin/kc

@@ -10,8 +10,8 @@
 # source distribution.
 
 import sys
-if sys.version_info < (3, 3):
-    sys.stderr.write('ERROR: kernelcraft requires Python 3.3 or higher\n')
+if sys.version_info < (3,8):
+    print('KernelCraft requires Python 3.8 or higher')
     sys.exit(1)
 
 from kernelcraft import run

bin/virtme-configkernel

@@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+# -*- mode: python -*-
+# virtme-configkernel: Configure a kernel for virtme
+# Copyright © 2014 Andy Lutomirski
+# Licensed under the GPLv2, which is available in the virtme distribution
+# as a file called LICENSE with SHA-256 hash:
+# 8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643
+
+# This file is not installed; it's just used to run virtme from inside
+# a source distribution.
+
+import sys
+if sys.version_info < (3,8):
+    print('KernelCraft requires Python 3.8 or higher')
+    sys.exit(1)
+
+from virtme.commands import configkernel
+exit(configkernel.main())

bin/virtme-mkinitramfs

@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+# -*- mode: python -*-
+# virtme-mkinitramfs: Generate an initramfs image for virtme
+# Copyright © 2019 Marcos Paulo de Souza
+# Licensed under the GPLv2, which is available in the virtme distribution
+# as a file called LICENSE with SHA-256 hash:
+# 8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643
+
+import sys
+if sys.version_info < (3,8):
+    print('KernelCraft requires Python 3.8 or higher')
+    sys.exit(1)
+
+from virtme.commands import mkinitramfs
+exit(mkinitramfs.main())

bin/virtme-prep-kdir-mods

@@ -0,0 +1,68 @@
+#!/bin/sh
+
+# This is still a bit of an experiment.
+FAKEVER=0.0.0
+MODDIR=".virtme_mods/lib/modules/$FAKEVER"
+
+# Some distro don't have /sbin or /usr/sbin in user's default path. Make sure
+# to setup the right path to find all the commands needed to setup the modules
+# (depmod, etc.).
+PATH=$PATH:/sbin:/usr/sbin
+
+COPY_MODULES=${COPY_MODULES:-"false"}
+
+print_help() {
+    script_name=$(basename "$0")
+    echo "usage: ${script_name} [-h | --help] [-c | --copy-modules]"
+    echo ""
+    echo "optional arguments:"
+    echo "  -h, --help          show this help message and exit"
+    echo "  -c, --copy-modules  copy kernel instead of linking"
+}
+
+while ":"; do
+    case "$1" in
+	-h | --help)
+	    print_help
+	    exit 0
+	    ;;
+	-c | --copy-modules)
+	    COPY_MODULES="true"
+	    shift
+	    ;;
+	*)
+	    break
+    esac
+done
+
+if ! [ -f "modules.order" ]; then
+    echo 'kc-prep-kdir-mods must be run from a kernel build directory' >&2
+    echo "modules.order is missing.  Your kernel may be too old or you didn't make modules." >&2
+    exit 1
+fi
+
+# Set up .virtme_mods/lib/modules/0.0.0 as a module directory for this kernel,
+# but fill it with symlinks instead of actual modules.
+
+mkdir -p "$MODDIR/kernel"
+ln -srfT . "$MODDIR/build"
+
+# Remove all preexisting symlinks and add symlinks to all modules that belong
+# to the build kenrnel.
+find "$MODDIR/kernel" -type l -print0 |xargs -0 rm -f --
+# from v6.2, modules.order lists .o files, we need the .ko ones
+sed 's:\.o$:.ko:' modules.order | while read -r i; do
+    [ ! -e "$i" ] && i=$(echo "$i" | sed s:^kernel/::)
+    mkdir -p "$MODDIR/kernel/$(dirname "$i")"
+    if [ "$COPY_MODULES" = "true" ]; then
+        cp "$i" "$MODDIR/kernel/$i"
+    else
+        ln -sr "$i" "$MODDIR/kernel/$i"
+    fi
+done
+
+# Link in the files that make modules_install would copy
+ln -srf modules.builtin modules.builtin.modinfo modules.order "$MODDIR/"
+
+# Now run depmod to collect dependencies
+depmod -ae -F System.map -b .virtme_mods "$FAKEVER"

bin/virtme-run

@@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+# -*- mode: python -*-
+# virtme-run: The main command-line virtme frontend
+# Copyright © 2014 Andy Lutomirski
+# Licensed under the GPLv2, which is available in the virtme distribution
+# as a file called LICENSE with SHA-256 hash:
+# 8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643
+
+# This file is not installed; it's just use to run virtme from inside
+# a source distribution.
+
+import sys
+if sys.version_info < (3,8):
+    print('KernelCraft requires Python 3.8 or higher')
+    sys.exit(1)
+
+from virtme.commands import run
+exit(run.main())

debian/control

@@ -13,7 +13,6 @@ Depends:
  ${shlibs:Depends},
  ${misc:Depends},
  ${python3:Depends},
- virtme,
  crash,
  gcc-12,
  autoconf,

kernelcraft/run.py

@@ -150,6 +150,8 @@ def arg_fail(message, show_usage=True):
 def create_root(destdir, arch):
     if os.path.exists(destdir):
         return
+    # Use Ubuntu's cloud images to create a rootfs, these images are fairly
+    # small and they provide a nice environment to test kernels.
     release = check_output('lsb_release -s -c', shell=True).decode(sys.stdout.encoding).rstrip()
     url = f'https://cloud-images.ubuntu.com/{release}/current/{release}-server-cloudimg-{arch}-root.tar.xz'
     prevdir = os.getcwd()
@@ -347,7 +349,7 @@ def run(self, arch=None, root=None, cpus=None, memory=None, network=None, disk=N
             opts = ' '.join(opts)
         else:
             opts = ''
-        # Start VM using virtme
+        # Start VM using virtme-run
         rw_dirs = ' '.join(f'--overlay-rwdir {d}' for d in ('/boot', '/etc', '/home', '/opt', '/srv', '/usr', '/var'))
         cmd = f'virtme-run {arch} --name {hostname} --kdir ./ {mods} {rw_dirs} {pwd} {username} {root} {execute} {network} {disk} {opts} --qemu-opts -m {memory} -smp {cpus} -s -qmp tcp:localhost:3636,server,nowait'
         check_call(cmd, shell=True)

setup.py

@@ -5,32 +5,44 @@
 from setuptools import setup
 from kernelcraft.utils import VERSION, CONF_PATH
 
-if sys.version_info < (3,3):
-    print('kernelcraft requires Python 3.3 or higher')
+if sys.version_info < (3,8):
+    print('KernelCraft requires Python 3.8 or higher')
     sys.exit(1)
 
 setup(
     name='kernelcraft',
     version=VERSION,
     author='Andrea Righi',
     author_email='[email protected]',
-    description='',
+    description='Build and run a kernel inside a virtualized snapshot of your live system',
     url='https://git.launchpad.net/~arighi/+git/kernelcraft',
     license='GPLv2',
     long_description=open(os.path.join(os.path.dirname(__file__),
                                        'README.md'), 'r').read(),
     long_description_content_type="text/markdown",
-    packages=['kernelcraft'],
-    install_requires=['argcomplete', 'virtme'],
-    include_package_data=True,
+    packages=['kernelcraft', 'virtme', 'virtme.commands', 'virtme.guest'],
+    install_requires=['argcomplete'],
     entry_points = {
         'console_scripts': [
             'kc = kernelcraft.run:main',
+            'virtme-run = virtme.commands.run:main',
+            'virtme-configkernel = virtme.commands.configkernel:main',
+            'virtme-mkinitramfs = virtme.commands.mkinitramfs:main',
         ]
     },
     data_files = [
         (str(CONF_PATH), ['cfg/kernelcraft.conf']),
     ],
+    scripts = [
+        'bin/virtme-prep-kdir-mods',
+        ],
+    package_data = {
+        'virtme.guest': [
+            'virtme-init',
+            'virtme-udhcpc-script',
+        ],
+    },
+    include_package_data=True,
     classifiers=['Environment :: Console',
                  'Intended Audience :: Developers',
                  'Intended Audience :: System Administrators',