Add libunwind to the toolchain

[ParkMyCar]

Progress towards #1

This PR adds libunwind to the toolchain by building it with cmake via rules_foreign_cc. libunwind is an important runtime library for the clang toolchain (docs) and AFAIK is not natively included with macOS. Compiled it is <100Kb so I don't believe it will increase the size of the bundle.

Using my local machine's C toolchain I verified the libunwind build succeeds, but I can't get the bundled zig-cc toolchain to work. I also tried creating a free Buildbuddy account but that failed with the following error:

INFO: Invocation ID: 505ef73e-1616-47f6-be90-b15684758ceb
INFO: Streaming build results to: https://app.buildbuddy.io/invocation/505ef73e-1616-47f6-be90-b15684758ceb
WARNING: Build options --action_env, --experimental_platform_in_output_dir, --extra_execution_platforms, and 4 more have changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //:for_all_platforms (112 packages loaded, 169374 targets configured).
ERROR: /private/var/tmp/_bazel_parker.timmerman/09e98b04a37762c62ebd6addff782992/external/llvm_zstd/BUILD.bazel:22:11: Compiling lib/dictBuilder/divsufsort.c failed: (Exit 1): c++ failed: error executing CppCompile command (from target @@llvm_zstd//:zstd) external/hermetic_cc_toolchain~3.0.1~toolchains~zig_sdk/tools/x86_64-macos-none/c++ -MD -MF bazel-out/darwin_amd64-opt/bin/external/llvm_zstd/_objs/zstd/divsufsort.d ... (remaining 21 arguments skipped)
exec external/hermetic_cc_toolchain~3.0.1~toolchains~zig_sdk/tools/x86_64-macos-none/c++: exec format error
Target //:for_all_platforms failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 21.336s, Critical Path: 13.83s
INFO: 2423 processes: 2390 internal, 4 local, 29 remote.
ERROR: Build did NOT complete successfully
FAILED:

@dzbarsky if you can provide guidance for these errors I would be more than happy to get this building locally!

[dzbarsky]

@ParkMyCar hi! Thanks for giving this a shot. I haven't had great luck in the past with rules_cmake, we saw make segfaulting on osx build :) But if this build and helps you adopt the toolchain, we can certainly stick this in the non-minimal dist target. (I want it non-minimal to avoid confusion from having multiple libunwind available for folks who are using sysroot).

In terms of getting this building, I'm not sure about your local error, I suspect it wasn't the same as the RBE one. Can you include it?

For the one you hit on Buildbuddy, it has to do with the cross-compilation toolchains not being setup correctly. Note from the error, RBE is invoking the darwin x86_64 compiler, but the executor is Linux ;)

When building from Mac, I suggest to run build_docker.sh. That way the host will be detected as linux, and the toolchain selection should work better.

[ParkMyCar]

Ahhh yeah we've run into similar issues with rules_cmake :) It seems to be working for now though, and definitely more than happy to put it only in non-minimal!

Sweet thanks for the tips! I'll try running locally with build_docker.sh.

Also I realized we could probably build libcxx and libcxxabi in the same way, any thoughts on including those in non-minimal as well? That should fix #1 entirely then?

[dzbarsky]

Ahhh yeah we've run into similar issues with rules_cmake :) It seems to be working for now though, and definitely more than happy to put it only in non-minimal!

Sweet thanks for the tips! I'll try running locally with build_docker.sh.

Also I realized we could probably build libcxx and libcxxabi in the same way, any thoughts on including those in non-minimal as well? That should fix #1 entirely then?

I think I'm ok with them going in non-minimal if we can prevent the segfaults and if they build correctly in RBE.

[marvin-hansen]

MacOS 15
Bazel 7.3

I just ran full tilt into this issue of missing libunwind.

My MODULE.bazel:

llvm = use_extension("@toolchains_llvm//toolchain/extensions:llvm.bzl", "llvm")
llvm.toolchain(
    llvm_version = "18.1.2",
    sha256 = {
        "darwin-aarch64": "a6dfd8d98f0753c96e74382f93520ead9ab4c1fa5c52676c6bdbe32cd4fbc89f",
        "linux-x86_64"  : "6511968b1b5c4369fe5032c7a470fddc0da290cf740518a5c03c048adc5163c2",
    },
    stdlib = {
        "linux-x86_64": "stdc++",
        "linux-aarch64": "stdc++",
    },
    urls = {
        "darwin-aarch64": ["https://github.com/dzbarsky/static-clang/releases/download/v18.1.2-3/darwin_arm64.tar.xz"],
        "linux-x86_64"  : ["https://github.com/dzbarsky/static-clang/releases/download/v18.1.2-3/linux_amd64.tar.xz"],
    },
)

use_repo(llvm, "llvm_toolchain", "llvm_toolchain_llvm")

register_toolchains("@llvm_toolchain//:all")

Then a local build throws this error:

...
  = note: -macosx_version_min has been renamed to -macos_version_min
          ld: warning: ignoring duplicate libraries: '-lSystem', '-lm'
          ld: library 'unwind' not found
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

The regular LLVM distro compiles the repo just fine so it would be great to get libunwind to this toolchain.

[dzbarsky]

MacOS 15 Bazel 7.3

I just ran full tilt into this issue of missing libunwind.

My MODULE.bazel:

llvm = use_extension("@toolchains_llvm//toolchain/extensions:llvm.bzl", "llvm")
llvm.toolchain(
    llvm_version = "18.1.2",
    sha256 = {
        "darwin-aarch64": "a6dfd8d98f0753c96e74382f93520ead9ab4c1fa5c52676c6bdbe32cd4fbc89f",
        "linux-x86_64"  : "6511968b1b5c4369fe5032c7a470fddc0da290cf740518a5c03c048adc5163c2",
    },
    stdlib = {
        "linux-x86_64": "stdc++",
        "linux-aarch64": "stdc++",
    },
    urls = {
        "darwin-aarch64": ["https://github.com/dzbarsky/static-clang/releases/download/v18.1.2-3/darwin_arm64.tar.xz"],
        "linux-x86_64"  : ["https://github.com/dzbarsky/static-clang/releases/download/v18.1.2-3/linux_amd64.tar.xz"],
    },
)

use_repo(llvm, "llvm_toolchain", "llvm_toolchain_llvm")

register_toolchains("@llvm_toolchain//:all")

Then a local build throws this error:

...
  = note: -macosx_version_min has been renamed to -macos_version_min
          ld: warning: ignoring duplicate libraries: '-lSystem', '-lm'
          ld: library 'unwind' not found
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

The regular LLVM distro compiles the repo just fine so it would be great to get libunwind to this toolchain.

I wrote some BUILD files for libunwind here: 5fcd53b#diff-250cf229a385f1287e3e2f86607f15328ab80302918e91bade694aca33d0b004R21

@marvin-hansen could you possibly make me a small repro that I can play with? I want to make sure I include the libs in the right place in the archive (and also that they actually work for linking the end binary)

[marvin-hansen]

@dzbarsky , here you go:

https://github.com/marvin-hansen/bazel_rust_cross_llvm

There are three branches:

1. main: The working build with the full LLVM toolchain
2. small-clang: Replaces the full LLVM toolchain with a smaller version that only contains CLang and a few tools.
3. llvm_musl: Only declares a small host clang toolchain and uses MUSL to cross compile all other targets.

The main branch builds as expected with the other two currently both failing with the missing libunwind error.

The Readme contains a few more notes, but in a nutshell, the repro is based on a cross compiling example I made recently so it comes with a bunch of common dependencies and tests that all build on Mac and Linux. The tests check binaries build for Linux on Arm and Intel.

bazel build //...

bazel test //...

Should get you covered on each branch.

Hope that helps.

[Kreijstal]

so what is missing?

[marvin-hansen]

Just as quick follow up, In the end, I've compiled clang specifically for my CI provider using their new default CI image mainly to get around some libc incompatibility. Be warned, on the default GH action runner this may take between 2 to 4 hours. https://github.com/marvin-hansen/buildbuddy-clang I've forked the GitHub action and Dockerfile from @parker who did a fantastic job setting this up. If you don't have a libc picky CI image, you can easily use his excellent clang builds.

…

On Thu, Feb 6, 2025 at 16:38 Kreijstal ***@***.***> wrote: so what is missing? — Reply to this email directly, view it onGitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFYR7XBOQTJOEWPDJGMPCXD2OMNQ5AVCNFSM6AAAAABK3YQRM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZZGE3DEOBSGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Kreijstal]

ah so it isn't static?

[marvin-hansen]

Well, no because for me reduced download and unzip time was more important to cut my CI time. Although if you modify the build container, you might be able to compile clang fully statically.