Rust application much slower when built with rules_rust than with Cargo

[bitemyapp]

I'm going to try to assemble an MRE but it's going to take me a little time because I'm working inside of a proprietary monorepo. Let me describe what I'm seeing:

When building the same Rust codebase with both Cargo and Bazel on Apple Silicon (ARM64) and x86_64 Linux, the Bazel-built binary shows approximately 60% worse performance in benchmarks compared to the Cargo-built version, despite seemingly identical optimization settings.

Environment

Platform: macOS (ARM64) and Linux (x86_64)
Rust version: nightly-2025-02-07
rules_rust version: 0.60.0
Both builds use static linking
Both builds have identical environment variables

What I've Tried

• Verified LTO is properly enabled in both builds (both thin and fat LTO, with and without LTO=manual)
• linker-plugin-lto only worked on Linux, barely helped (~1.5%)
• Confirmed optimizations are correctly set (opt-level=3, codegen-units=1)
• Tried adding -Ctarget-cpu=native and ARM-specific target features on both platforms.
• Verified the same code paths are being executed (same performance profile allocation percentages)
• I used --subcommands and RUSTC_LOG=rustc_codegen_ssa::back::lto=info to confirm LTO was firing, it was on both platforms.
• Manually setting rustc_flags on the binary target, library targets, etc. a la https://github.com/bazelbuild/examples/tree/main/rust-examples/03-comp-opt
• The only thing that has significantly helped performance (-10%) has been to set this:

build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cllvm-args=-import-instr-limit=5000
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cllvm-args=-inline-threshold=10000

But even with that it's quite a bit slower than the Cargo version.

Primary hypothesis: Dependency resolution method differences between Cargo and Bazel

My best guess is that the way the Bazel build links the rust_library targets defeats cross-crate inlining, monomorphization and optimization during LTO. My project's performance is extremely sensitive to inlining. Caveat: the most performance sensitive/critical libraries are both extern in Bazel and Cargo.

Secondary hypotheses:

• Different LLVM optimization pass ordering between Cargo and Bazel builds
• Dependency build order affecting optimization outcomes (I don't rate this highly)

Reproduction Steps

• Build the same binary with both Cargo and Bazel using release configurations
• Run benchmarks on both binaries (both outside of the sandbox)
• Observe approximately 60% worse performance in the Bazel-built binary

The benchmark is rust_binary building a benches/ file that uses the Criterion benchmark harness intenrally.

I think I need a way to force Bazel to use --extern-style dependency resolution rather than -L library paths, similar to how Cargo resolves dependencies in workspace mode, to get proper monomorphization and LTO behavior. I cannot use Bazel for this Rust project if this requires rewriting all the crate dependencies as one big source tree.

Frankly I'm a little baffled because I know major projects are using Bazel for their builds and I don't know how I'm running into a perf issue like this that I haven't been able to see mentioned anywhere on the web. And I looked very hard for answers. I've been working on this for the last few days.

Here's a sample .bazelrc I used in my testing:

Linux:

build:release --action_env=RUSTC_LOG=rustc_codegen_ssa::back::lto=info
build:release --compilation_mode=opt
build:release --@rules_rust//rust/settings:lto=fat
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off

build:release --@rules_rust//rust/settings:extra_rustc_flag=-Clinker-plugin-lto
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cembed-bitcode=yes
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Clink-arg=-fuse-ld=lld

macOS:

build:release --action_env=RUSTC_LOG=rustc_codegen_ssa::back::lto=info
build:release --compilation_mode=opt
build:release --@rules_rust//rust/settings:lto=fat
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off

I am desperately hoping that I simply missed something obvious. My command for building the binary would typically be something like:

bazel build --config=release //crate:bench_name and then I'd run the benchmark binary in bazel-bin/ with the --bench argument.

One difference between the binaries I noticed was that the Bazel generated binary was ~8.8 MiB vs. the ~8.3 MiB of the Cargo built binary. I first noticed the performance disparity with a very vanilla bazel build -c opt generated binary.

toolchain setup from MODULE.bazel:

bazel_dep(name = "rules_rust", version = "0.60.0")

rust = use_extension("@rules_rust//rust:extensions.bzl", "rust")
rust.toolchain(
    edition = "2021",
    versions = ["nightly/2025-02-07"],
    dev_components = True,
)
use_repo(rust, "rust_toolchains")

register_toolchains("@rust_toolchains//:all")

[...]

crates.from_cargo(
    name = "crates",
    cargo_lockfile = "//:Cargo.lock",
    generate_binaries = True,
    manifests = [
       (Cargo.tomls of the workspace and constituent crates)
    ],
)

Then the rust_binary for the benchmark:

rust_binary(
    name = "my_benchmark",
    srcs = ["benches/my_benchmark.rs"],
    aliases = aliases(),
    edition = "2021",
    proc_macro_deps = all_crate_deps(
        proc_macro = True,
        proc_macro_dev = True,
    ),
    deps = [
        ":lib",
        (other in-workspace dependencies)
    ] + all_crate_deps(
        normal = True,
        normal_dev = True,
    ),
)

There are two particular dependencies which are exposed as rust_librarys in the Bazel build which are very inlining sensitive, either one not getting inlined properly would explain the overall benchmark being slower without any obvious hot-spots. Comparing the params and rustc invocations from Cargo don't show any obvious differences. e.g.

From Bazel, benchmark program params in bazel-out:

--extern=pkg-a=bazel-out/darwin_arm64-opt/bin/pkg-a/libpkg-a-262915972.rlib
-Ldependency=bazel-out/darwin_arm64-opt/bin/pkg-a
--extern=pkg-b=bazel-out/darwin_arm64-opt/bin/pkg-b/libpkg-b-2509656758.rlib
-Ldependency=bazel-out/darwin_arm64-opt/bin/pkg-b

From Cargo:

-L
dependency=/Users/callen/.cargo/cache/release/deps
--extern
pkg-a=/Users/callen/.cargo/cache/release/deps/libpkg-a-27b8eb42159f5c37.rlib
--extern
pkg-b=/Users/callen/.cargo/cache/release/deps/libpkg-b-f26c14e53d574db5.rlib

Then buried in Cargo's deps:

/Users/callen/.cargo/cache/release/deps/libpkg-b-f26c14e53d574db5.rmeta: (…source listing…)

/Users/callen/.cargo/cache/release/deps/libpkg-b-f26c14e53d574db5.rlib: (…source listing…)

/Users/callen/.cargo/cache/release/deps/pkg-b-f26c14e53d574db5.d: (…source listing…)

I've been using nm -C and nm -gUC with awk to generate symbol table counts and they're strongly similar. The main pattern I'm seeing is additional symbols getting included in the Bazel binary seemingly because dead-code elimination isn't firing for the library artifacts properly.

Turning LTO off for both builds makes the Cargo and Bazel benchmarks about the same.

One difference I just noticed:

Bazel rustc for building the final benchmark binary:

--emit=link=bazel-out/darwin_arm64-opt/bin/crate/benchmark
--emit=dep-info

Cargo, for the same:

--emit=dep-info,link

Update: after some research, I think the =path argument to emit=link is just setting the output path, shouldn't make a difference.

It looks like the rlibs have both object code and bitcode in both Bazel and Cargo.

The rust_library definitions look like this:

rust_library(
    name = "pkg-b",
    srcs = glob(["src/**/*.rs"]),
    crate_features = [
        "default",
    ],
    edition = "2021",
    proc_macro_deps = [
        "//pkg-b/macros",
    ] + all_crate_deps(proc_macro = True),
    deps = [
        "//pkg-b/vendored-crate-1",
        "//pkg-b/vendored-crate-2",
        "//pkg-b/vendored-crate-3",
        "//pkg-b/vendored-crate-4",
    ] + all_crate_deps(),
)

rust_library(
    name = "pkg-a",
    srcs = glob(["src/**/*.rs"]),
    aliases = aliases(),
    crate_features = [
        "default",
        "bazel_build",
    ],
    edition = "2021",
    proc_macro_deps = [
        "//pkg-b/macros",
    ] + all_crate_deps(proc_macro = True),
    deps = [
        "//pkg-c",
        "//pkg-b/vendored-crate-2",
        "//pkg-b",
    ] + all_crate_deps(),
)

New update: enabling build:release --@rules_rust//rust/settings:lto=thin worked locally on Linux and macOS (still same perf issue tho) but barfed on the proc macro crates in CI/CD with a linker error that is indicative of "tried improperly to LTO a proc macro crate".

Wondering if the CC and CXX compilers, and every other environment variable are the same between cargo and bazel. I think you may be compiling your sys dependencies differently.

Update: A Reddit user suggested a possibility, here's my reply:

I think you may be compiling your sys dependencies differently.

Could be, but all the perf impact is the pure Rust crates that live inside the Cargo workspace. There's very very little -sys in the dep tree and none of it is in the critical path. If any of it had been, it would've shown up when I profiled the optimized builds with debug symbols.

Update, follow-up:

    build:release --compilation_mode=opt
    build:release --@rules_rust//rust/settings:lto=thin
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebuginfo=0
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cstrip=debuginfo
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cforce-frame-pointers=no

This made the benchmark 3.5x as slow as the original benchmark with Cargo.

[lgarbarini]

Not a maintainer, but curious if you've used a tool like elf_diff to see what types of differences exist.

[bitemyapp]

Not a maintainer, but curious if you've used a tool like elf_diff to see what types of differences exist.

I've been sorting, diffing, comparing the symbol tables with nm and the section sizes with bloaty but I'll give that a shot, thank you!

[lgarbarini]

Not a maintainer, but curious if you've used a tool like elf_diff to see what types of differences exist.

I've been sorting, diffing, comparing the symbol tables with nm and the section sizes with bloaty but I'll give that a shot, thank you!

Got it, I'd be curious to see some examples of short functions where there's a diff (something innocuous in terms of confidentiality).

[bitemyapp]

@lgarbarini I haven't had a chance to try that tool yet but it does seem like the slowdown is evenly spread across two particular crates that really need to be fast and get inlined/LTO'd a lot. I don't know what elf_diff would show me that nm, samply, and bloaty haven't already, I've been comparing the symbol tables.

[lgarbarini]

@lgarbarini I haven't had a chance to try that tool yet but it does seem like the slowdown is evenly spread across two particular crates that really need to be fast and get inlined/LTO'd a lot. I don't know what elf_diff would show me that nm, samply, and bloaty haven't already, I've been comparing the symbol tables.

I think my message was unclear, completely agree that objdump, nm, samply, and bloaty have you covered, no need for elf_diff if you're doing it manually.

If the manual diff is exclusively inlined functions then I'll poke around some more, but if the diff contains other changes to other instructions outside of inlining, I would be curious to see a few excerpted sections from objdump.

[bitemyapp]

Oh what I wouldn't give for this tool to be written in Rust instead of Python.

[bitemyapp]

While we're waiting for elf_diff to finish, here's how I've been comparing the symbols:

#!/usr/bin/env bash

export CARGO_BINARY=/home/callen/.cargo/cache/release/deps/benchmark-c0594d7ce94f306a
export BAZEL_BINARY=(...path to the bazel-bin/ benchmark binary)

nm -gUC $CARGO_BINARY | awk -F'::' '/ [Tt] / { print $2 }' | sort | uniq -c | sort -nr
nm -gUC $BAZEL_BINARY | awk -F'::' '/ [Tt] / { print $2 }' | sort | uniq -c | sort -nr

nm -C $CARGO_BINARY | grep pkg-b | wc -l
nm -C $BAZEL_BINARY | grep pkg-b | wc -l

nm -C $CARGO_BINARY | grep pkg-a | wc -l
nm -C $BAZEL_BINARY | grep pkg-a | wc -l

nm -gUC $CARGO_BINARY | grep pkg-b | wc -l
nm -gUC $BAZEL_BINARY | grep pkg-b | wc -l

nm -gUC $CARGO_BINARY | grep pkg-a | wc -l
nm -gUC $BAZEL_BINARY | grep pkg-a | wc -l

nm -gUC $CARGO_BINARY | grep sync | wc -l
nm -gUC $BAZEL_BINARY | grep sync | wc -l

# nm -gUC $CARGO_BINARY | grep sync | sort > /tmp/cargo_sync.txt
# nm -gUC $BAZEL_BINARY | grep sync | sort > /tmp/bazel_sync.txt
# diff -u /tmp/cargo_sync.txt /tmp/bazel_sync.txt

# nm -gUC $CARGO_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sort > /tmp/cargo_sync.txt
# nm -gUC $BAZEL_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sort > /tmp/bazel_sync.txt
# diff -u /tmp/cargo_sync.txt /tmp/bazel_sync.txt

nm -gUC $CARGO_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo_sync.txt
nm -gUC $BAZEL_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel_sync.txt
diff -u /tmp/cargo_sync.txt /tmp/bazel_sync.txt

nm -gUC $CARGO_BINARY | grep pkg-b | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo.txt
nm -gUC $BAZEL_BINARY | grep pkg-b | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel.txt
diff -u /tmp/cargo.txt /tmp/bazel.txt

nm -gUC $CARGO_BINARY | grep pkg-a | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo.txt
nm -gUC $BAZEL_BINARY | grep pkg-a | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel.txt
diff -u /tmp/cargo.txt /tmp/bazel.txt

nm -gUC $CARGO_BINARY | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo.txt
nm -gUC $BAZEL_BINARY | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel.txt
diff -u /tmp/cargo.txt /tmp/bazel.txt

The differences just aren't significant at least from what I can see from nm.

345
343
296
295
121
121
193
192
88
89

Is an example of the output for the middle commands in between the diffs and the uniq counts.

[bitemyapp]

Someone in another thread raised it so I'll say it explicitly here: I have profiled this over and over. The slowdown is spread across the entire call graph of the 1 or 2 main crates that impact performance, there are no hot-spots, just deeply rooted parts of the call graph that get wider as a whole. I've been using samply to profile and it seems to be capturing everything but I'm running with cachegrind right now to try to get a more complete profile. If there was a hotspot I'd have ideas for what to try other than fiddling with .bazelrc over and over across the whole build.

[marvin-hansen]

Wow, this is quite something. Just want to follow along to see what comes up in the investigation.
Also, please share a repro to give the community a chance to help in a meaningful way.

[marvin-hansen]

Also, have you verified that the resulting binaries are actually statically linked?

I have reported god knows how many issues related to static linking with rules_rust and ultimately given up.
However, I believe it was fixed a while ago. If you statically link with MUSL and your resulting binary is actually static, your culprit might be the MUSL default memory allocator especially under heavy concurrent workload. I saw a 10X drop in that case. Switching to MiMalloc recovered the 10x perf drop and then added some tiny boost on top of it. See this repo: https://github.com/marvin-hansen/mimalloc

[bitemyapp]

@marvin-hansen There's only so much I can say about this which is related to why an MRE is so hard, but a few things:

• I don't use musl because I'm paranoid
• Our allocators are glibc malloc and mmap for an arena allocator, the critical path of the benchmark doesn't call malloc at all and actually aborts the process if it tries to. I generally use jemalloc for daemonized servers but malloc hasn't been a designated for this project yet. I try to wait until I have metrics/profiles before I change things from the platform defaults.

My main update at this point is that I'm no longer able to reproduce the original Cargo baseline benchmark performance on Linux…with Cargo. Even if I backtrack to older commits. I can still repro it on Mac + Cargo.

It's a macro-benchmark that involves a very thick stack of technology, I can't really make an MRE that meaningfully captures this. I will scrutinize my micro-benchmarks and see if any of them are evincing the same slow-down factor. If such a thing exists, then I will extract an MRE, but there's always a chance that whatever fixes the micro-benchmark doesn't fix anything for the big boy.

[bitemyapp]

@lgarbarini I looked at the elf_diff results this morning, I saw some false deleted/added where it was the same symbol with the same asm read-out getting flagged for unknown reasons. Nothing in particular stood out but I haven't had time to scrutinize it carefully. I'm going to focus on an MRE and shrinking the repro because there's just too much stuff happening in this benchmark and I need to isolate variables.

[bitemyapp]

Yeah I'm still having this perf issue with the Bazel-built version of my program. I managed to get some unrelated issues fixed and got back to the original baseline performance for the Cargo and Bazel versions, same situation as before. As near as I can tell it's like the usual LTO and LLVM optimizations aren't happening. An example is that we have an empty inline asm! hack as a branch hint, I'm pretty sure it's not getting optimized out in the Bazel version but I haven't confirmed yet. We rely on LLVM to do that. Another thing I'll note here is that the problem libm had which @tgross35 was able to fix with #[inline] on a generic function isn't likely applicable here. We have non-generic functions with #[inline] or more commonly #[inline(always)] on them.

libm thing: rust-lang/libm#545

Update: I replaced the asm! hack with core::hint::cold_path and both builds got faster by the same coefficient.

Update2: I can only repro this on macOS, not Linux. This is increasingly looking like something weird with macOS or ld64, but I'm not sure.

[bitemyapp]

Nope, it's repro'ing with Cargo inside of Docker + OrbStack + Ubuntu 24.10 + aarch64 on my M3 Pro Mac.

Suspiciously faster execution:

• Cargo + Apple Silicon macOS (~6-7 seconds benchmark time)
• Cargo + Apple Silicon Ubuntu 24.10 + Docker/OrbStack on M3 Pro Mac (~6-7s)
• Cargo + Google Axion (aarch64) Ubuntu 24.10 (~9.8s)

Slower execution:

• Manual rustc build + run (verbatim copy of the Cargo build in a script), Apple Silicon, macOS (~12-13s)
• cargo bench --no-run then running the binary manually (~12-13s)
• cargo run --bin my_benchmark (made a [[bin]] target for it in Cargo.toml) (~12-13s)
• Bazel + Google Axion (aarch64) Ubuntu 24.10 (19.0s)
• Bazel + Apple Silicon macOS (~12-13s)
• Bazel + Apple Silicon Ubuntu 24.10 + Docker/OrbStack on M3 Pro Mac (~12-13s)
• Cargo + x64 Linux (~12-13s)
• Bazel + x64 Linux (~12-13s)

My local test bench for Linux is a 9800X3D + RTX 5090 desktop that I built recently, the benchmark is single-threaded and the single-threaded performance of the M3 Pro and 9800X3D are generally very close which is convenient for my purposes.

The commonality here so far seems to be cargo + aarch64 rather than macOS/Mac toolchain.

Update: there seems to be something specifically about letting cargo bench run the binary. Running the binary manually whether cargo bench, a verbatim rustc facsimile of the cargo bench -vv rustc command, or Bazel built it yields the slower result. I dumped the ENV from the benchmark program and there weren't any interesting differences. Only thing that flagged up was DYLD_FALLBACK_LIBRARY_PATH and I tried setting the script harness' DYLD_FALLBACK_LIBRARY_PATH for my manual rustc facsimile of cargo bench to the same value and it wasn't any faster. In addition, this benchmark is 100% Rust so far as I'm aware with zero C/dylib/*-sys code involved. I'm double-checking on that to increase my certainty.

[bitemyapp]

Optimization flag perf impact:

Docker, OrbStack, M3 Mac, Bazel:

(these are sequential performance steps/changes)

• lto=thin -> lto=off: none
• codegen-units=1 -> (opt default, 16 I think): ~4.5% slower
• opt-level=3 -> opt-level=2: 1.2% slower

Docker, OrbStack, M3 Mac, Cargo:

(sequential again)

• lto=thin -> lto=off: 2.5% slower
• codegen-units=1 -> (release default, 16?): 14.4% slower
• opt-level=3 -> opt-level=2: oddly, it actually improves by ~1.2% on opt-level=2 unlike all the other instances.

lto=fat has only made a small difference unless I start messing with the LLVM inlining limits.

When I disabled codegen-units=1 for Cargo I disabled it for both the direct release profile but also for the package dependencies under profile.release.package."*". Ditto opt-level.