libmicrovmi python bindings

[Wenzel]

Create Python bindings to call libmicrovmi from Python

TODO (updated):

• decide how to integrate this new library in our build system
• expose DriverType and DriverInitParam
• expose read_physical
• package and upload to PyPI
• update setup.py to accept --features parameter, and set features in RustExtension
• create new Dockerfile based on quay.io/pypa/manylinux2014_x86_64 and install all the drivers required development headers
• update build-wheels.sh to accept --features and pass them to setup.py
• read all relevant fields of python/Cargo.toml to fill setup.py metadata

We should use the rust-cpython project

rust-cpython will help us to generate a Python extension module, wrapping the libmicrovmi Rust API.

how to integrate this new library

1. create a separate repository
2. use cargo workspace to manage both Rust library and Python extension in the same git repository, in tandem.
3. maybe generate both with the same cargo.toml (I doubt this is possible)

@rageagainsthepc if you have an opinion

[Wenzel]

After looking at rust-cpython, a couple of things:

• there is no need to create a workspace, we can produce the same libmicrovmi.so that will be considered both the official C API, and the native Python extension
• PyO3 is a fork of rust-cpython, at the time when it was maintained, and it proposes a better abstraction in my opinion. the only drawback was that it required nightly rust, but 6 months ago, it became available with stable rust

[rageagainsthepc]

the only drawback was that it required nightly rust, but 6 months ago, it became available with stable rust

that's awesome, my info was not up to date then, because I thought it was still nightly only. 👍

[Wenzel]

there is no need to create a workspace, we can produce the same libmicrovmi.so that will be considered both the official C API, and the native Python extension

actually not a good idea, since the library produced has undefined symbols to the Python API, requiring that any binary would link with Python libs.

                 U PyBaseObject_Type
                 U PyBytes_AsString
                 U PyBytes_Size
                 U _Py_Dealloc
                 U PyDict_Copy
                 U PyDict_DelItem
                 U PyDict_GetItem
                 U PyDict_Next
                 U PyDict_Size
                 U PyErr_Fetch
                 U PyErr_GivenExceptionMatches
                 U PyErr_NewException
                 U PyErr_NormalizeException
                 U PyErr_Print
                 U PyErr_PrintEx
                 U PyErr_Restore
                 U PyEval_SaveThread
                 U PyEval_ThreadsInitialized
                 U PyExc_AttributeError
                 U PyExc_BaseException
                 U PyExc_SystemError
                 U PyExc_TypeError
                 U PyExc_ValueError
                 U Py_Finalize
                 U PyGILState_Ensure
                 U PyGILState_Release
                 U Py_InitializeEx

so workspace and splitting is a good idea.

[Wenzel]

I investigated how to distribute the generate bindings.

maturin seems like the way to go for PyO3 bindings.
The tooling is good, however, I'm limited because maturin generates a wheel based on the crate name, and this isn't configurable.

The crate name right now is pymicrovmi, however this is only a conflict free value for the whole workspace, and the python package name should be microvmi.
A user should be able to do

pip install microvmi
and
from microvmi import Microvmi

Unfortunately, maturin doesn't allow to override this package name at the moment, and only uses the Cargo lib.name field.
This is an opened issue on maturin repo:
PyO3/maturin#313

Looking at this comment it looks like setuptools-rust is more flexible (yet older) alternative for now.

I will have a look at it

[rageagainsthepc]

What's wrong with pymicrovmi? After all many python bindings are named similarly if I remember correctly.

[Wenzel]

What's wrong with pymicrovmi?

I prefer the simplicity of having microvmi when you look for it on pypi and when you use the module than having a useless py prefix.
it's a python module, we know it's for python, there is no need for a prefix like this :)
it's just removing useless noise.

[rageagainsthepc]

Sure, I get that sentiment. I am just not sure whether it's worth pursuing other routes over. ;)

[Wenzel]

Sure, I get that sentiment. I am just not sure whether it's worth pursuing other routes over. ;)

I managed to get setuptools-rust working.
Maybe in a few months, maturin will have improved enough to edit the package name, who knows

[rageagainsthepc]

Are there any drawbacks at all if we are using setuptools-rust instead of maturin?

[Wenzel]

Maturin is described as:

This project is meant as a zero configuration replacement for setuptools-rust and milksnake.

So no, we are just not following the hype ... yet :)

[Wenzel]

Publishing wheels on PyPI is a pain in the ass.

PyPI will refuse linux_x86_64 platform tag

Uploading distributions to https://upload.pypi.org/legacy/
Uploading microvmi-0.0.1-cp36-cp36m-linux_x86_64.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.15M/1.15M [00:01<00:00, 755kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Client Error: Binary wheel 'microvmi-0.0.1-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://upload.pypi.org/legacy/

reason: https://stackoverflow.com/a/59586096/3017219
so we need to generate the manylinux1 platform tag now, using this docker image:
https://github.com/pypa/manylinux

However, Rust can't be installed on the manylinux1 docker image, the Glibc is too old:

(venv)  wenzel@Strix  ~/Projets/rust/libmicrovmi   python_bindings  docker run --rm -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/python/build-wheels.sh
+ curl https://sh.rustup.rs -sSf
+ sh -s -- --default-toolchain stable -y
info: downloading installer
/tmp/tmp.ZfVxKoeK20/rustup-init: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/tmp.ZfVxKoeK20/rustup-init)
/tmp/tmp.ZfVxKoeK20/rustup-init: /lib64/libc.so.6: version `GLIBC_2.9' not found (required by /tmp/tmp.ZfVxKoeK20/rustup-init)

with manylinux2010, PyO3 won't compile:

       Fresh microvmi v0.1.10 (/io)
error: failed to run custom build command for `pyo3 v0.13.1`

Caused by:
  process didn't exit successfully: `/io/target/release/build/pyo3-66b94e3e1352bcf0/build-script-build` (exit code: 1)
  --- stderr
  /io/target/release/build/pyo3-66b94e3e1352bcf0/build-script-build: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by /io/target/release/build/pyo3-66b94e3e1352bcf0/build-script-build)
  /io/target/release/build/pyo3-66b94e3e1352bcf0/build-script-build: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /io/target/release/build/pyo3-66b94e3e1352bcf0/build-script-build)
cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module --release --verbose -- --crate-type cdylib
error: cargo failed with code: 101

and with manylinux2014, auditwheel is unable to convert the platform tag to manylinux1:
➡️ actually now it works, and i have no idea why 😶