Merge #1023

1023: Add support for Android versions 5-11. r=Emilgardis a=Alexhuszagh

**Configuration**

We provide the following arguments to configure the Android version:
- `ANDROID_NDK`
- `ANDROID_SDK` (note that this is dependent on `ANDROID_VERSION)
- `ANDROID_VERSION`
- `ANDROID_SYSTEM_COMPLETE`: do a complete Android build
- `ANDROID_SYSTEM_NONE`: do not build the Android system, disables runner support

**Version Support**

We now support NDK versions r10e-r25b, SDK versions 21-33, and Android versions 5.0, 5.1, 6.0, 7.0, 8.0, 8.1, 9.0, 10.0, and 11.0. We also validate that the NDK, SDK, and Android versions are compatible.

For Android 12+, we do support using the NDK with the proper API level, allowing cross-compilation for the desired target, however, running or testing the generated binary is currently not supported without a complete build. Note that support for non-complete builds with Android versions 12+ is unlikely to ever occur, due to issues with APEX in the build system.

**Implementation Details**

Next, we've improved the removal of unittests during the build process, to ensure fast builds while maintaining compatibility with various newer Android versions. To do this, we've implemented a Python library and a script. The Python library contains a complete parser (correctly parses all valid input) for Soong blueprint files, using an LALR grammar, and a rudimentary parser for Makefiles.

The Soong parser removes any `gtest` dependencies, as well as any subdirectories or scope names containing `test`. For example:

```go
cc_library {
    name: "lib",
    srcs: ["lib.cc",],
}
cc_test {
    name: "test",
    defaults: ["target"],
    srcs: ["test.cc"],
}
```

Will become:

```go
cc_library {
    name: "lib",
    srcs: ["lib.cc",],
}
```

The Makefile parser first splits the file based on conditional directives (`ifeq`, `endif`, etc.) to ensure any subsequent processing doesn't lead to unbalanced directives. Next, we split the text within each directive based on comment sections used in the Android source tree. For example:

```Makefile
test_tags := tests

include $(call subdir,$(LOCAL_PATH))

c_flags := \
  -g \
  -Wall
```

We can therefore remove the `Benchmarks` and `Unit tests` sections without removing the `Other section`.

The Python library is reasonably performant (it adds no noticeable overhead to the build process) and is compact (in total, < 60KB). Also, it is much more resilient than a series of `sed` scripts.

Finally, extensive unittests have been added for the Python library, for both code linting (`flake8`) and unittests (via `tox`). Since we cannot assume the presence of Python on the host machine, the tests can be optionally enabled via the `--python` flag (or `PYTHON` environment variable, to hook into the git hooks), and custom overrides for the `flake8` and `tox` commands are provided (since the user may wish to specify a specific Python version, such as `python3.7 -m flake8`).

**Linker**

For Android 10+, since we use a minimal Android build, we only support the bootstrap and ASAN linkers unless using a complete Android build. Supporting the APEX linker requires a nearly complete Android runtime, requiring 60+GB image sizes, slow builds, and other prohibitive factors.

**Complete Builds**

Complete builds are currently untested, for Androids version 12+ they exceed the default storage capacity of WSL2 images (250GB). They should work and properly install both the `/system` and `/apex` directories, however, the builds are slow and untested.

Co-authored-by: Alex Huszagh <[email protected]>

Author: bors[bot]

.changes/1023.json

@@ -0,0 +1,4 @@
+{
+    "description": "support different Android NDK, API, and Android versions using Docker build args.",
+    "type": "added"
+}

.gitignore

@@ -4,3 +4,12 @@
 **/*.log
 /cargo-timing*.html
 CHANGELOG.md.draft
+
+# python stuff
+__pycache__/
+.pytest_cache/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+.tox

Cargo.lock

@@ -0,0 +1,11 @@
+# don't copy any of the python artifacts to the docker context
+__pycache__/
+.pytest_cache/
+*.py[cod]
+*$py.class
+**/*.egg-info/
+*.egg
+.tox
+
+# also skip our test suite
+android/tests/

docker/.dockerignore

@@ -10,27 +10,51 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+COPY qemu.sh /
+RUN /qemu.sh aarch64
+
+ARG ANDROID_NDK=r25b
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_NONE=0
+ARG ANDROID_SYSTEM_COMPLETE=0
+ARG PYTHON_TMPDIR=/tmp/android
+
+COPY validate-android-args.sh /
+RUN /validate-android-args.sh arm64
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh arm64 28
+RUN /android-ndk.sh arm64
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY android-system.sh /
+RUN mkdir -p $PYTHON_TMPDIR
+COPY android $PYTHON_TMPDIR
 RUN /android-system.sh arm64
 
-COPY qemu.sh /
-RUN /qemu.sh aarch64
-
 ENV CROSS_SYSROOT=/android-ndk/sysroot
-RUN cp $CROSS_SYSROOT/usr/lib/aarch64-linux-android/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh aarch64 aarch64-linux-android
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
     CARGO_TARGET_AARCH64_LINUX_ANDROID_RUNNER="/android-runner aarch64" \
+    AR_aarch64_linux_android=aarch64-linux-android-ar \
+    AS_aarch64_linux_android=aarch64-linux-android-as \
     CC_aarch64_linux_android=aarch64-linux-android-gcc \
     CXX_aarch64_linux_android=aarch64-linux-android-g++ \
+    LD_aarch64_linux_android=aarch64-linux-android-ld \
+    NM_aarch64_linux_android=aarch64-linux-android-nm \
+    OBJCOPY_aarch64_linux_android=aarch64-linux-android-objcopy \
+    OBJDUMP_aarch64_linux_android=aarch64-linux-android-objdump \
+    RANLIB_aarch64_linux_android=aarch64-linux-android-ranlib \
+    READELF_aarch64_linux_android=aarch64-linux-android-readelf \
+    SIZE_aarch64_linux_android=aarch64-linux-android-size \
+    STRINGS_aarch64_linux_android=aarch64-linux-android-strings \
+    STRIP_aarch64_linux_android=aarch64-linux-android-strip \
     BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android="--sysroot=$CROSS_SYSROOT" \
     DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include"/ \
     RUST_TEST_THREADS=1 \

docker/Dockerfile.aarch64-linux-android

@@ -10,27 +10,51 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+COPY qemu.sh /
+RUN /qemu.sh arm
+
+ARG ANDROID_NDK=r25b
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_NONE=0
+ARG ANDROID_SYSTEM_COMPLETE=0
+ARG PYTHON_TMPDIR=/tmp/android
+
+COPY validate-android-args.sh /
+RUN /validate-android-args.sh arm
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh arm 28
+RUN /android-ndk.sh arm
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY android-system.sh /
+RUN mkdir -p $PYTHON_TMPDIR
+COPY android $PYTHON_TMPDIR
 RUN /android-system.sh arm
 
-COPY qemu.sh /
-RUN /qemu.sh arm
-
 ENV CROSS_SYSROOT=/android-ndk/sysroot
-RUN cp $CROSS_SYSROOT/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh arm arm-linux-androideabi
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_ARM_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
     CARGO_TARGET_ARM_LINUX_ANDROIDEABI_RUNNER="/android-runner arm" \
+    AR_arm_linux_androideabi=arm-linux-androideabi-ar \
+    AS_arm_linux_androideabi=arm-linux-androideabi-as \
     CC_arm_linux_androideabi=arm-linux-androideabi-gcc \
     CXX_arm_linux_androideabi=arm-linux-androideabi-g++ \
+    LD_arm_linux_androideabi=arm-linux-androideabi-ld \
+    NM_arm_linux_androideabi=arm-linux-androideabi-nm \
+    OBJCOPY_arm_linux_androideabi=arm-linux-androideabi-objcopy \
+    OBJDUMP_arm_linux_androideabi=arm-linux-androideabi-objdump \
+    RANLIB_arm_linux_androideabi=arm-linux-androideabi-ranlib \
+    READELF_arm_linux_androideabi=arm-linux-androideabi-readelf \
+    SIZE_arm_linux_androideabi=arm-linux-androideabi-size \
+    STRINGS_arm_linux_androideabi=arm-linux-androideabi-strings \
+    STRIP_arm_linux_androideabi=arm-linux-androideabi-strip \
     BINDGEN_EXTRA_CLANG_ARGS_arm_linux_androideabi="--sysroot=$CROSS_SYSROOT" \
     DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
     RUST_TEST_THREADS=1 \

docker/Dockerfile.arm-linux-androideabi

@@ -10,27 +10,51 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+COPY qemu.sh /
+RUN /qemu.sh arm
+
+ARG ANDROID_NDK=r25b
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_NONE=0
+ARG ANDROID_SYSTEM_COMPLETE=0
+ARG PYTHON_TMPDIR=/tmp/android
+
+COPY validate-android-args.sh /
+RUN /validate-android-args.sh arm
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh arm 28
+RUN /android-ndk.sh arm
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY android-system.sh /
+RUN mkdir -p $PYTHON_TMPDIR
+COPY android $PYTHON_TMPDIR
 RUN /android-system.sh arm
 
-COPY qemu.sh /
-RUN /qemu.sh arm
-
 ENV CROSS_SYSROOT=/android-ndk/sysroot
-RUN cp $CROSS_SYSROOT/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh arm arm-linux-androideabi
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
     CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_RUNNER="/android-runner arm" \
+    AR_armv7_linux_androideabi=arm-linux-androideabi-ar \
+    AS_armv7_linux_androideabi=arm-linux-androideabi-as \
     CC_armv7_linux_androideabi=arm-linux-androideabi-gcc \
     CXX_armv7_linux_androideabi=arm-linux-androideabi-g++ \
+    LD_armv7_linux_androideabi=arm-linux-androideabi-ld \
+    NM_armv7_linux_androideabi=arm-linux-androideabi-nm \
+    OBJCOPY_armv7_linux_androideabi=arm-linux-androideabi-objcopy \
+    OBJDUMP_armv7_linux_androideabi=arm-linux-androideabi-objdump \
+    RANLIB_armv7_linux_androideabi=arm-linux-androideabi-ranlib \
+    READELF_armv7_linux_androideabi=arm-linux-androideabi-readelf \
+    SIZE_armv7_linux_androideabi=arm-linux-androideabi-size \
+    STRINGS_armv7_linux_androideabi=arm-linux-androideabi-strings \
+    STRIP_armv7_linux_androideabi=arm-linux-androideabi-strip \
     BINDGEN_EXTRA_CLANG_ARGS_armv7_linux_androideabi="--sysroot=$CROSS_SYSROOT" \
     DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
     RUST_TEST_THREADS=1 \

docker/Dockerfile.armv7-linux-androideabi

@@ -10,34 +10,59 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
-COPY android-ndk.sh /
-RUN /android-ndk.sh x86 28
-ENV PATH=$PATH:/android-ndk/bin
-
-COPY android-system.sh /
-RUN /android-system.sh x86
-
 # We could supposedly directly run i686 binaries like we do for x86_64, but
 # doing so generates an assertion failure:
 #   ... assertion failed: signal(libc::SIGPIPE, libc::SIG_IGN) != libc::SIG_ERR
 #   ... src/libstd/sys/unix/mod.rs
 #   fatal runtime error: failed to initiate panic, error 5
 #
-# Running with qemu works as expected
+# Running with qemu works as expected. it also ensures that're we're
+# running on a CPU that only supports 32-bit x86 systems.
 COPY qemu.sh /
 RUN /qemu.sh i386
 
+ARG ANDROID_NDK=r25b
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_NONE=0
+ARG ANDROID_SYSTEM_COMPLETE=0
+ARG PYTHON_TMPDIR=/tmp/android
+
+COPY validate-android-args.sh /
+RUN /validate-android-args.sh x86
+
+COPY android-ndk.sh /
+RUN /android-ndk.sh x86
+ENV PATH=$PATH:/android-ndk/bin
+
+COPY android-system.sh /
+RUN mkdir -p $PYTHON_TMPDIR
+COPY android $PYTHON_TMPDIR
+RUN /android-system.sh x86
+
 ENV CROSS_SYSROOT=/android-ndk/sysroot
-RUN cp $CROSS_SYSROOT/usr/lib/i686-linux-android/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh i386 i686-linux-android
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_I686_LINUX_ANDROID_LINKER=i686-linux-android-gcc \
     CARGO_TARGET_I686_LINUX_ANDROID_RUNNER="/android-runner i686" \
+    AR_i686_linux_android=i686-linux-android-ar \
+    AS_i686_linux_android=i686-linux-android-as \
     CC_i686_linux_android=i686-linux-android-gcc \
     CXX_i686_linux_android=i686-linux-android-g++ \
+    LD_i686_linux_android=i686-linux-android-ld \
+    NM_i686_linux_android=i686-linux-android-nm \
+    OBJCOPY_i686_linux_android=i686-linux-android-objcopy \
+    OBJDUMP_i686_linux_android=i686-linux-android-objdump \
+    RANLIB_i686_linux_android=i686-linux-android-ranlib \
+    READELF_i686_linux_android=i686-linux-android-readelf \
+    SIZE_i686_linux_android=i686-linux-android-size \
+    STRINGS_i686_linux_android=i686-linux-android-strings \
+    STRIP_i686_linux_android=i686-linux-android-strip \
     BINDGEN_EXTRA_CLANG_ARGS_i686_linux_android="--sysroot=$CROSS_SYSROOT" \
     DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
     LIBZ_SYS_STATIC=1 \

docker/Dockerfile.i686-linux-android

@@ -10,18 +10,31 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+COPY qemu.sh /
+RUN /qemu.sh arm
+
+ARG ANDROID_NDK=r25b
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_NONE=0
+ARG ANDROID_SYSTEM_COMPLETE=0
+ARG PYTHON_TMPDIR=/tmp/android
+
+COPY validate-android-args.sh /
+RUN /validate-android-args.sh arm
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh arm 28
+RUN /android-ndk.sh arm
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY android-system.sh /
+RUN mkdir -p $PYTHON_TMPDIR
+COPY android $PYTHON_TMPDIR
 RUN /android-system.sh arm
 
-COPY qemu.sh /
-RUN /qemu.sh arm
-
 ENV CROSS_SYSROOT=/android-ndk/sysroot
-RUN cp $CROSS_SYSROOT/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh arm arm-linux-androideabi
 
 COPY android-runner /
 

docker/Dockerfile.thumbv7neon-linux-androideabi

@@ -10,28 +10,52 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+# Using qemu allows older host cpus (without sse4) to execute the target binaries
+COPY qemu.sh /
+RUN /qemu.sh x86_64
+
+ARG ANDROID_NDK=r25b
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_NONE=0
+ARG ANDROID_SYSTEM_COMPLETE=0
+ARG PYTHON_TMPDIR=/tmp/android
+
+COPY validate-android-args.sh /
+RUN /validate-android-args.sh x86_64
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh x86_64 28
+RUN /android-ndk.sh x86_64
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY android-system.sh /
+RUN mkdir -p $PYTHON_TMPDIR
+COPY android $PYTHON_TMPDIR
 RUN /android-system.sh x86_64
 
-# Using qemu allows older host cpus (without sse4) to execute the target binaries
-COPY qemu.sh /
-RUN /qemu.sh x86_64
-
 ENV CROSS_SYSROOT=/android-ndk/sysroot
-RUN cp $CROSS_SYSROOT/usr/lib/x86_64-linux-android/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh x86_64 x86_64-linux-android
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_X86_64_LINUX_ANDROID_LINKER=x86_64-linux-android-gcc \
     CARGO_TARGET_X86_64_LINUX_ANDROID_RUNNER="/android-runner x86_64" \
+    AR_x86_64_linux_android=x86_64-linux-android-ar \
+    AS_x86_64_linux_android=x86_64-linux-android-as \
     CC_x86_64_linux_android=x86_64-linux-android-gcc \
     CXX_x86_64_linux_android=x86_64-linux-android-g++ \
+    LD_x86_64_linux_android=x86_64-linux-android-ld \
+    NM_x86_64_linux_android=x86_64-linux-android-nm \
+    OBJCOPY_x86_64_linux_android=x86_64-linux-android-objcopy \
+    OBJDUMP_x86_64_linux_android=x86_64-linux-android-objdump \
+    RANLIB_x86_64_linux_android=x86_64-linux-android-ranlib \
+    READELF_x86_64_linux_android=x86_64-linux-android-readelf \
+    SIZE_x86_64_linux_android=x86_64-linux-android-size \
+    STRINGS_x86_64_linux_android=x86_64-linux-android-strings \
+    STRIP_x86_64_linux_android=x86_64-linux-android-strip \
     BINDGEN_EXTRA_CLANG_ARGS_x86_64_linux_android="--sysroot=$CROSS_SYSROOT" \
     DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
     RUST_TEST_THREADS=1 \