3.14: Windows on Arm build fails since windows-11-arm migrated to windows-11-vs2026-arm

[chris-eibl]

The Windows Arm64 jobs in the CI recently started failing on 3.14 (for example). Noted by @StanFromIreland in #158188 (comment): the _freeze_module

cpython/PCbuild/pcbuild.proj

Lines 20 to 21 in 9def455

	<FreezeProjects>
	<Platform>$(PreferredToolArchitecture)</Platform>

can no longer be compiled for PreferredToolArchitecture=x86 and PlatformToolset=v143:

The build tools for Visual Studio 2022 (Platform Toolset = 'v143') cannot be found.

PreferredToolArchitecture=x86 is chosen, because we use the x86 version of msbuild in CI:

"C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin\msbuild.exe"

which is chosen by build.bat if no MSBUILD environment variable is set and msbuild is not in the path. ¹

Compiling the _freeze_module for arm64 on an arm64 host by just using the arm64 version of msbuild would solve this, by e.g. just adding

      env:
        MSBUILD: ${{ inputs.arch == 'arm64' && 'C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin\arm64\msbuild.exe' || '' }}

to

cpython/.github/workflows/reusable-windows.yml

Lines 38 to 40 in e16b12b

	- name: Build CPython
	run: >-
	.\\PCbuild\\build.bat

because then PreferredToolArchitecture=arm64 and also the arm64 version of the compiler is chosen.

I think running the _freeze_module natively on an arm64 host is anyway the better fit?

Footnotes

1. noticed already in https://github.com/python/cpython/issues/153668#issuecomment-5304195008 ↩

[chris-eibl]

Note, we cannot override PreferredToolArchitecture if we use the x86 version of msbuild via

build.bat "/p:PreferredToolArchitecture=arm64"

due PROCESSOR_ARCHITECTURE=x86 in this case, and via C:\Program Files\Microsoft Visual Studio\18\Community\MSBuild\Microsoft\VC\v180\Microsoft.Cpp.ToolsetLocation.props:

  <PropertyGroup>
    <!-- UseNativeEnvironment and _IsNativeEnvironment was used in prev versions to define if we want to use 64-bit tools when building for x64 platform. -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and ('$(UseNativeEnvironment)' == 'true' or '$(_IsNativeEnvironment)' == 'true')">x64</PreferredToolArchitecture>

    <!-- By default we use the same bitness as the hosting platform -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and ('$(PROCESSOR_ARCHITECTURE)' == 'AMD64' and '$(Platform)' == 'x64')">x64</PreferredToolArchitecture>
    <!-- prefer arm64 tools when on arm64 -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and '$(PROCESSOR_ARCHITECTURE)' == 'ARM64' and '$(HostARM64Suported)' != 'false'">arm64</PreferredToolArchitecture>

    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == ''">x86</PreferredToolArchitecture>

    <!-- If OS is not x64, we cannot use x64 tools even if preferred -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == 'x64' and ('$(PROCESSOR_ARCHITECTURE)' != 'AMD64' and '$(PROCESSOR_ARCHITEW6432)' != 'AMD64' and '$(PROCESSOR_ARCHITECTURE)' != 'ARM64')">x86</PreferredToolArchitecture>

    <!-- If OS is not arm64, we cannot use arm64 tools even if preferred -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == 'arm64' and '$(PROCESSOR_ARCHITECTURE)' != 'ARM64'">x86</PreferredToolArchitecture>
  </PropertyGroup>

we always get PreferredToolArchitecture=x86. This is also the default, hence in all builds so far:

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostX86\x86\CL.exe /c /IC:\a\cpython\cpython\PCbuild\obj\314win32_Debug\_freeze_module\

Why this does not work anymore for the latest windows-11-arm image is not clear to me, especially because these compilers are part of it

Toolset     HostX86_X86 HostX86_ARM64 HostARM64_X86 HostARM64_ARM64
-------     ----------- ------------- ------------- ---------------
14.29.30133        True          True         False           False
14.44.35207        True          True         False            True
14.51.36231        True          True          True            True

so 14.44.35207\bin\HostX86\x86 should be found.

Details

=== Compiler paths ===
--- 14.29.30133 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\cl.exe
--- 14.44.35207 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostarm64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostarm64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx86\x86\cl.exe
--- 14.51.36231 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\x86\cl.exe

See https://github.com/chris-eibl/cpython/actions/runs/36241148789/job/108401760423 in my fork.

[chris-eibl]

Short summary:

Running the _freeze_module natively on an arm64 host could be done in main as well and would fix 3.14 without the need of migrating to v145, which was done to get 3.14 CI green via backporting:

• [3.14] GH-144679: When building with VS 2026 on Windows, use PlatformToolset v145 by default (GH-144680) #158188
  • including also Silence experimental coroutine deprecation warnings #150788
• [3.14] gh-153668: Temporarily disable unicodectype.c optimization on WinARM64 due to compiler bug (GH-153669) #158221

which could be reverted again.

Otherwise:

The current situation in 3.14 CI is that

• v145 is used for WoA builds
• but still v143 for AMD64 and x86, because they use windows-2022 that does not have VS2026.

Regarding official releases, AFAIU we could achieve the same (or even stick with v143 for Arm) since we cross-compile and only run the PGO step on @zooba's Azure VM (at least that's my understanding): we'd simply have to use the respective image, that is windows-2022 instead of windows-2025. For the first we get v143, for the latter v145, since this is chosen when VS 2026 is available now on 3.14. Previously, we've used v143 in this case, too.

Compiling and running the _freeze_module natively works on the new windows-11-arm image, tested here:
https://github.com/chris-eibl/cpython/actions/runs/36245087168/job/108412696751
note the

 C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostArm64\arm64\CL.exe

[chris-eibl]

@zware I have not yet found out why on your ARM64 Windows 3.x bot

bin\HostArm64\arm64\CL.exe

is chosen to build the _freeze_module, see e.g. https://buildbot.python.org/#/builders/729/builds/11426/steps/2/logs/stdio.

On my D4ps v6 VM in Azure, bin\HostX86\x86\CL.exe is taken by default like in CI.

Only one of

• setting the environment variable MSBUILD to the arm64 version
• adding the arm64 msbuild path to PATH
• calling vcvarsarm64.bat

tricks me into bin\HostArm64\arm64\CL.exe there.

Have you done anything of the above or maybe something even simpler?