[Rollout issue] Docker images failing to build due to failure in build-rootfs.sh

[riarenas]

• This issue is blocking

This issue is blocking rollouts for the docker images.

• This issue is causing unreasonable pain

The latest docker rollout https://dnceng.visualstudio.com/internal/_build/results?buildId=2085885&view=logs&j=1c4e26f6-d871-56a6-8075-f831838eaa01&t=c6369efb-1c13-5347-3009-6c7a7740b620&l=7402 is failing with error:

--2023-01-12 12:58:02--  https://github.com/alpinelinux/apk-tools/releases/download/v2.9.1/apk-tools-2.9.1-x86_64-linux.tar.gz
...

2023-01-12 12:58:02 (85.6 MB/s) - ‘/tmp/tmp.sZ4OiI52Uk/apk-tools-2.9.1-x86_64-linux.tar.gz’ saved [814809/814809]
...
'/usr/bin/qemu-arm-static' -> '/rootfs/arm/usr/bin/qemu-arm-static'
fetch http://dl-cdn.alpinelinux.org/alpine/v3.13/community/armv7/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.13/main/armv7/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  numactl-dev (missing):
    required by: world[numactl-dev]
-- EXECUTION ELAPSED TIME: 00:00:04.5055328
-- EXECUTING: docker logout dotnetdocker.azurecr.io
Removing login credentials for dotnetdocker.azurecr.io
-- EXECUTION ELAPSED TIME: 00:00:00.0257663
Unhandled exception: System.InvalidOperationException: Failed to execute build hook '/repo/dotnet-dotnet-buildtools-prereqs-docker/src/ubuntu/22.04/cross/arm-alpine/hooks/pre-build'

The docker pipeline downloads this script directly from arcade, so once a fix is understood the pipeline should be unblocked.

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

[riarenas]

@michellemcdaniel @ChadNedzlek any ideas?

[riarenas]

Some thoughts from @mthalman

https://github.com/dotnet/arcade/blob/main/eng/common/cross/build-rootfs.sh defaults to alpine3.13 if you just specify "alpine" for the CodeName. And Alpine 3.13 is EOL as of Nov 2022.

arcade/eng/common/cross/build-rootfs.sh

Line 260 in a6a3a70

alpine|alpine3.13)

[mthalman]

Note that there was a successful build of this just earlier this week on Tuesday. See the successful build output at https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/2084293/logs/792. Search for /repo/dotnet-dotnet-buildtools-prereqs-docker/src/ubuntu/22.04/cross/arm-alpine/hooks/pre-build in the log.

Given the error message and the timeframe from when it was working last, it appears that this recent change is the cause: 2561b84.

cc @am11

[michellemcdaniel]

This issue is a really good reason for eliminating rootfs where ever and whenever we can. We have linux arm/arm64 build queues now, so we should just be able to build arm/arm64 docker images for the various OSes that are rootfs for that reason and build "natively" like how we run our tests.

[jkotas]

This issue is a really good reason for eliminating rootfs where ever and whenever we can

We are going to use more rootfs going forward to fulfill the secure supply chain (building on fully supported OS) and product/business requirements (supporting old distros that may be no longer officially supported). Check discussion in dotnet/runtime#69361 for more context.

Trying to eliminate rootfs wherever we can does not make sense.

[jkotas]

Given the error message and the timeframe from when it was working last, it appears that this recent change is the cause: 2561b84.

We should revert this change to sort this out without blocking eng system.

[am11]

@jkotas fix is already up dotnet/dotnet-buildtools-prereqs-docker#766

[riarenas]

Given how the current docker build picks up the live script from arcade, I still prefer that this change gets reverted while we get the new docker images out the door. The change, and the subsequent change to the docker repo can be revisited afterwards?

[jkotas]

There are questions that I would like to see answered about the impact of the fix: dotnet/dotnet-buildtools-prereqs-docker#766 (comment)

[michellemcdaniel]

The rootfs scripts do not fit in line with secure supply chain. They are external and download random things from random places on the internet. That is against our principles. That is the problem we working on solving with the work we are doing in the docker space.

I'll be honest, I don't really understand much about rootfs, but there are many pieces of if that seem very insecure, whether or not we are relying on the host OS to be secure or not. So long as there are bits like

wget -O - https://ftp.gnu.org/gnu/gcc/gcc-8.4.0/gcc-8.4.0.tar.xz
wget -O - "https://github.com/freebsd/pkg/archive/${__FreeBSDPkg}.tar.gz"
wget "https://github.com/alpinelinux/apk-tools/releases/download/v$__ApkToolsVersion/apk-tools-$__ApkToolsVersion-x86_64-linux.tar.gz"

etc that we are just assuming aren't being stomped on, I'm going to be nervous about it. On our VM images, we have been able to make sure we get all of the tools we can't get from package manager from a secure location that can't be stomped on. What is the process for doing that in the rootfs space?

[am11]

They are external and download random things from random places on the internet.

Those places are source repositories and not any different than non-rootfs images: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/09c9b571629dcc0c13eec49ce07c5dbec6892e41/src/centos/7/Dockerfile#L56-L62 (the CentOS dockerfile is used to create linux-x64 image used by official builds since .NET Core 2x).

I'll be honest, I don't really understand much about rootfs

Lets take an example of ubuntu-18.04-cross-arm-alpine. The build will use clang-9 from host ubuntu-18.04 with -sysroot set to /crossrootfs/arm. Meaning, the compiled binaries will only link against the symbols from native libs available in rootfs. The VM will not (and cannot) execute any binary from the rootfs because they are of different OS, libc, architecture or all of the above. The product does not include anything from the rootfs location (or from the host OS for that matter). When the end-user installs the product, the dynamic loader finds the compatible libraries on their system and locates the required symbols.

[michellemcdaniel]

Those places are source repositories and not any different than non-rootfs images: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/09c9b571629dcc0c13eec49ce07c5dbec6892e41/src/centos/7/Dockerfile#L56-L62 (the CentOS dockerfile is used to create linux-x64 image used by official builds since .NET Core 2x).

Changing that is currently the scope of the epic that @ChadNedzlek and I are working on, see dotnet/dnceng#1313 for more details.

[jkotas]

The rootfs scripts do not fit in line with secure supply chain.

I agree that the current rootfs scripts have supply chain problems and we should fix those. We should have a discussion about the best way to do that.

rootfs as a build technology is critical for us (#12189 (comment)).

[michellemcdaniel]

We are currently in the process of converting docker images to be in line with our policies, but let's see if we can put together a meeting about rootfs, ownership, and SSC.