Compiling an eBPF in docker

[Togtja]

Hi, I been trying to compile the aya-template XDP program in side the Rust's Alpine docker container
Basically this:

FROM rust:alpine


RUN apk add --no-cache musl-dev gcc llvm clang llvm-dev openssl-dev pkgconf

RUN rustup toolchain install nightly --component rust-src
RUN rustup target add x86_64-unknown-linux-musl

 #0.9.13 as it looks for LLVM 19 where the 0.9.14 looks for 20, and currently the alpine comes with LLVM 19
RUN cargo install [email protected] --no-default-features


RUN mkdir -p /app
WORKDIR /app
CMD [ "cargo", "build", "--release" ]

Then mounting the template repo to /app

Then it fails when linking with bpf-linker

< cut some downloading compiling warning for brevity >
builder-1  | warning: [email protected]:    Compiling aya_demo-common v0.1.0 (/app/aya_demo-common)
builder-1  | warning: [email protected]:    Compiling aya-log-ebpf v0.1.1
builder-1  | warning: [email protected]: error: linking with `bpf-linker` failed: signal: 11 (SIGSEGV) (core dumped)
builder-1  | warning: [email protected]:   |
builder-1  | warning: [email protected]:   = note:  "bpf-linker" "--export-symbols" "/tmp/rustcP9S365/symbols" "/tmp/rustcP9S365/symbols.o" "<1 object files omitted>" "/app/target/release/build/aya_demo-25c9624e43a6d70f/out/aya_demo-ebpf/bpfel-unknown-none/release/deps/{libaya_log_ebpf-c64862894bb1e85a.rlib,libaya_log_common-7734ed872e8af7d8.rlib,libnum_enum-168d50c9b00513de.rlib,libaya_ebpf-b8ce06dab19b6ade.rlib,libaya_ebpf_bindings-d3d2b6f439e3db4e.rlib,libaya_ebpf_cty-e41fa803f3cfc8a4.rlib,librustc_std_workspace_core-68f422dc3dd90c17.rlib,libcore-4769d38d00cba92f.rlib,libcompiler_builtins-f202cac87a12366a.rlib}.rlib" "-L" "/tmp/rustcP9S365/raw-dylibs" "--cpu" "generic" "-o" "/app/target/release/build/aya_demo-25c9624e43a6d70f/out/aya_demo-ebpf/bpfel-unknown-none/release/deps/aya_demo-83d007ab76177aa5" "-O3" "--debug"
builder-1  | warning: [email protected]:   = note: some arguments are omitted. use `--verbose` to show all linker arguments
builder-1  | warning: [email protected]:   = note: 
builder-1  | warning: [email protected]: 
builder-1  | warning: [email protected]: 
builder-1  | warning: [email protected]: error: could not compile `aya_demo-ebpf` (bin "aya_demo") due to 1 previous error
builder-1  | error: failed to run custom build command for `aya_demo v0.1.0 (/app/aya_demo)`
builder-1  | 
builder-1  | Caused by:
builder-1  |   process didn't exit successfully: `/app/target/release/build/aya_demo-ea0dbf32eda949e0/build-script-build` (exit status: 1)

It then seems to re run and ends up with just about the same output but the Error being

builder-1  |   Error: env -u RUSTC -u RUSTC_WORKSPACE_WRAPPER CARGO_CFG_BPF_TARGET_ARCH="x86_64" "cargo" "+nightly" "build" "--package" "aya_demo-ebpf" "-Z" "build-std=core" "--bins" "--message-format=json" "--release" "--target" "bpfel-unknown-none" "--target-dir" "/app/target/release/build/aya_demo-25c9624e43a6d70f/out/aya_demo-ebpf" failed: ExitStatus(unix_wait_status(25856))

I have tried with the newest bpf-linker without the --no-default-features (as the newest requires LLVM 20)
Then I get some different errors when compiling and the linking happens, that I think was fixed by using the --no-default-features option

builder-1  | warning: [email protected]: error: linking with `bpf-linker` failed: signal: 6 (SIGABRT) (core dumped)
builder-1  | warning: [email protected]:   |
builder-1  | warning: [email protected]:   = note:  "bpf-linker" "--export-symbols" "/tmp/rustcas8trV/symbols" "/tmp/rustcas8trV/symbols.o" "<1 object files omitted>" "/app/target/release/build/aya_demo-25c9624e43a6d70f/out/aya_demo-ebpf/bpfel-unknown-none/release/deps/{libaya_log_ebpf-c64862894bb1e85a.rlib,libaya_log_common-7734ed872e8af7d8.rlib,libnum_enum-168d50c9b00513de.rlib,libaya_ebpf-b8ce06dab19b6ade.rlib,libaya_ebpf_bindings-d3d2b6f439e3db4e.rlib,libaya_ebpf_cty-e41fa803f3cfc8a4.rlib,librustc_std_workspace_core-68f422dc3dd90c17.rlib,libcore-4769d38d00cba92f.rlib,libcompiler_builtins-f202cac87a12366a.rlib}.rlib" "-L" "/tmp/rustcas8trV/raw-dylibs" "--cpu" "generic" "-o" "/app/target/release/build/aya_demo-25c9624e43a6d70f/out/aya_demo-ebpf/bpfel-unknown-none/release/deps/aya_demo-83d007ab76177aa5" "-O3" "--debug"
builder-1  | warning: [email protected]:   = note: some arguments are omitted. use `--verbose` to show all linker arguments
builder-1  | warning: [email protected]:   = note: unable to read dir /usr/local/cargo/lib: No such file or directory (os error 2)
builder-1  | warning: [email protected]:           unable to open LLVM shared lib /usr/lib/libLLVM-19.so: Dynamic loading not supported
builder-1  | warning: [email protected]:           unable to open LLVM shared lib /usr/lib/libLLVM.so.19.1: Dynamic loading not supported
builder-1  | warning: [email protected]:           unable to open LLVM shared lib /usr/lib/libLLVM-19.so: Dynamic loading not supported
builder-1  | warning: [email protected]:           unable to open LLVM shared lib /usr/lib/libLLVM.so.19.1: Dynamic loading not supported
< removed a stack backtrace >

As this is docker hopefully it's very easy to re-create.

Please let me know if this is generally not possible

(FYI)
Also tried with

FROM rustlang/rust:nightly-alpine


RUN apk add --no-cache musl-dev gcc llvm clang llvm-dev openssl-dev pkgconf

RUN rustup component add rust-src --toolchain nightly-x86_64-unknown-linux-musl

RUN cargo install [email protected] --no-default-features


RUN mkdir -p /app
WORKDIR /app
CMD [ "cargo", "build", "--release" ]

Same issue as abvoe

[tamird]

I believe you are using latest nightly, which uses LLVM 20, but an older bpf-linker which uses 19. You need to install LLVM 20, not 19.

[vadorovsky]

You can use the trick of downloading LLVM 20 from Rust CI, like I did here:

https://gitlab.com/taricorp/llvm-sys.rs/-/blob/f88520ce8f94773982ffa9f87e008796fe878fdb/.gitlab-ci.yml#L21-34

You basically need to:

• Download the tarball, using the commit hash of github.com/rust-lang/rust, which bumps their LLVM version.
• Unpack it somewhere, let's say /usr/lib/llvm-rustc.
• Build bpf-linker with LLVM_SYS_201_PREFIX=/usr/lib/llvm-rustc (or whichever directory you unpacked the tarball to).

And as @tamird said, you need to use LLVM version which matches the current Rust nightly. Attempts to use distro packaging and pinning bpf-linker or Rust toolchain to something ancient usually fails. It's better to stick to what's the newest.

[Togtja]

I'll try to update to LLVM 20 and the newer bpf-linker, so my understanding since I use the flag --no-default-features I have pinned bpf-linker to whatever LLVM i have installed on the system, because I want to note I was able to compile on my Ubuntu with LLVM 18, and newest Rust (1.85), though I did not need to use --no-default-features

 stable-x86_64-unknown-linux-gnu updated - rustc 1.86.0 (05f9846f8 2025-03-31) (from rustc 1.85.1 (4eb161250 2025-03-15))
  nightly-x86_64-unknown-linux-gnu updated - rustc 1.88.0-nightly (2fa8b11f0 2025-04-06) (from rustc 1.88.0-nightly (1799887bb 2025-03-29))

Basically followed the readme generated from the Aya template

## Prerequisites

1. stable rust toolchains: `rustup toolchain install stable`
1. nightly rust toolchains: `rustup toolchain install nightly --component rust-src`
1. (if cross-compiling) rustup target: `rustup target add ${ARCH}-unknown-linux-musl`
1. (if cross-compiling) LLVM: (e.g.) `brew install llvm` (on macOS)
1. (if cross-compiling) C toolchain: (e.g.) [`brew install filosottile/musl-cross/musl-cross`](https://github.com/FiloSottile/homebrew-musl-cross) (on macOS)
1. bpf-linker: `cargo install bpf-linker` (`--no-default-features` on macOS)

If that's the case I am now also wondering why my install of bpf-linker fails in my Docker container if I don't use --no-default-features

I am basically trying to just make the build happen in a docker container, though using alpine.

As for the nightly part, from my understanding is that you need to have nightly available for the Aya eBPF (#250)
Though I think that for the most part I am using the rust stable
with the exception when I attempted to use the image FROM rustlang/rust:nightly-alpine but both return the same error. There it was nighly the whole way

[tamird]

bpf-linker is not independent of rust, regardless of features. The LLVM IR produced by rustc is consumed by bpf-linker, so the LLVM versions must be compatible, which usually means they must be the same major version.

I'm going to close this for now, please let us know if you continue to have issues.