Library building for Windows compilers

[partouf]

Part of #1332

Will need to address the following:

• Building with CMake on Windows
• Building with MingW compilers
• Building with MSVC compilers
• Sending packages to Conan
• Whatever else comes up

Will not include (things to be added later):

• Windows builder packer image
• Infra setup for Windows builder
• Non-C++ stuff

[partouf]

Does not introduce a new LibraryBuilder class in order to reuse as much as possible from existing (proven) code

[partouf]

msvc builds done, now remains only conan stuff which is probably the hardest

[partouf]

TODO

• Copy some extra files to Win SDK directory

[partouf]

Am at a a bit of deadlock now. Will write it down in the hopes that I miraculously think of the correct solution.

The problem:

When building the libraries for Windows, I force the recipe for conan to be a 'package install' which means it packages the install directory and you say to cmake hey install it to that directory.

This is done to avoid having headers on disk permanently like we have for most of our Linux libraries.

But the consequence of that is that the conan recipe changes, and that's replaced on the conan server for that library version.

Now, technically, CE doesn't actually use the recipes. It's only for telling conan what to zip up and upload. But if the recipe is changed, will it also say it needs reuploading for compilers that already have an upload? I need to figure this out first. I think I can check this by uploading a msvc build and then try to upload a build for a gcc that we've already done in the past. But I do need to disable force uploading before I do that.

Doing that now before continuing this writeup.

[partouf]

Ok, so changing the recipe doesn't cause issues with checking if a build has already been done.

Next issue:

Not having a directory where all the libraries are installed is obviously an issue for our ce_install build command. Because it first has to be installed before being able to build it.

I have tested the process simply by calling ce_install install first for the library and version that I was testing, but obviously this is terrible for automation.

So we need a way to temporary install the library version before building it.

The question is how, options are:

• Have an option to install as part of the build command
• Handle the installation outside of the python script

Either way, it will always cause installation of the library and will slow down the build even if there's nothing to build.
Then again, compared to looping over all the compilers and checking if it needs to be built, it's probably negligible.

I think it might be best to integrate it into the python script, since the installable is already available here https://github.com/compiler-explorer/infra/blob/main/bin/lib/ce_install.py#L478

Will give that a go.

[partouf]

Was pretty easy to add

[partouf]

Starting to work well.

MSVC x64 and x86 work, arm does not because supportsBinary is set to false. We can figure that out in the future.

Next todo:

• Some paths for the buildscript are resolved to the network share, this is not really a thing we want to use (in case we want to add sandboxing or caching in the future or something), and it's not consistent.

Example:

cmake --install-prefix "C:\tmp\staging\8566353b-7420-4392-88eb-8fe6d4be4159\install"
-GNinja "-DCMAKE_VERBOSE_MAKEFILE=ON"
  "-DCMAKE_BUILD_TYPE=Debug"
 "-DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=\\172.30.0.254\winshared\compilers\msvc\14.20.27508-14.20.27525.0\bin\Hostx64"
 "-DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=\\172.30.0.254\winshared\compilers\msvc\14.20.27508-14.20.27525.0\bin\Hostx64"
  "-DCMAKE_CXX_FLAGS_DEBUG=/EHsc /utf-8    "
 -D "CMAKE_C_COMPILER=Z:/compilers/msvc/14.20.27508-14.20.27525.0/bin/Hostx64/x86/cl.exe"
 -D "CMAKE_CXX_COMPILER=Z:/compilers/msvc/14.20.27508-14.20.27525.0/bin/Hostx64/x86/cl.exe"
 C:\tmp\staging\libs\fmt\4.1.0 > cecmakelog.txt 2>&1

[partouf]

so Path.resolve() also resolves drives that are on a network share
os.path.abspath() does not (looks like it also normalizes, so that might be the way to go)
testing...

[partouf]

Todo:

• Something to fix after_stage_script (for example https://github.com/compiler-explorer/infra/blob/main/bin/yaml/libraries.yaml#L78) - in some cases can be executed by powershell, but in other cases we just just fail that one build and not the entire build session like here https://github.com/compiler-explorer/infra/actions/runs/12876168262/job/35898716567#step:3:1024
• Similar for Pre and Post build scripts (example https://github.com/compiler-explorer/infra/blob/main/bin/yaml/libraries.yaml#L153)
• Installing tar.xz files fails (e.g. for boost_bin)

[partouf]

• cannot supply pwsh scripts in yaml file due to escaping issue of { and }, help needed

[mattgodbolt]

Hi @partouf - can you give an example of something that repro's the issue you're facing?
Something like :

- after_stage_script
  - '{ moo }'

? where you don't want the { to be anything but a literal { ? I think this isn't a YAML problem if so but the config_expand.py not supporting escaping. I'll look to add that.

[partouf]

Hi @partouf - can you give an example of something that repro's the issue you're facing? Something like :

- after_stage_script
  - '{ moo }'

? where you don't want the { to be anything but a literal { ? I think this isn't a YAML problem if so but the config_expand.py not supporting escaping. I'll look to add that.

The problem starts when I want to allow alternatives to shellscripting, for example for boost_bin https://github.com/compiler-explorer/infra/blob/main/bin/yaml/libraries.yaml#L137
where -Werror is replaced in CMake files so we can actually build the library

An example for a Powershell alternative (according to our AI friends) would be:

      after_stage_script_pwsh:
      - Get-ChildItem -Recurse -Filter "CMakeLists.txt, *.cmake" | ForEach-Object {
      -   (Get-Content $_.FullName) -replace '-Werror', '-Wno-error' | Set-Content $_.FullName
      - }