jemalloc-sys fails to build with musl on Ubuntu 16.04

[BurntSushi]

I'm not sure what the error means, but the relevant portion is here:

configure: WARNING: using cross tools not prefixed with host triplet
ar: `u' modifier ignored since `D' is the default (see `U')
/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(sysconf.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make: *** [lib/libjemalloc.so.2] Error 1
make: *** Waiting for unfinished jobs....
ar: `u' modifier ignored since `D' is the default (see `U')
thread 'main' panicked at 'command did not execute successfully: "make" "srcroot=../jemalloc/" "-j" "2"
expected success, got: exit code: 2', /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.3.0/build.rs:370:9

You can see the full build failure here: https://travis-ci.org/BurntSushi/ripgrep/jobs/524197138

musl version is 1.1.9.

Any ideas on what's wrong? I can't reproduce this on my local system. (But I'm using musl 1.1.21 locally.)

[BurntSushi]

cc @alexcrichton Is this related to rust-lang/rust#59411?

[BurntSushi]

See also: BurntSushi/ripgrep#1268

[gnzlbg]

@BurntSushi can you test compiling with cross build --target=x86_64-unknown-linux-musl ? (you need to cargo install cross first). I just want to figure out whether this is a ripgrep specific issue, or whether it might have to do with your C cross-compilation toolchain. If cross build works, chances are your cross compilation toolchain might not be properly set (although it could also mean that there is an issue with the musl version being used, but we can figure that out as the next step).

[gnzlbg]

For comparison, our build jobs do not show the "configure: WARNING: using cross tools not prefixed with host triplet" warning: https://travis-ci.com/gnzlbg/jemallocator/jobs/195415085#L665

[BurntSushi]

I tried simply removing --features pcre2 from my CI build (which avoids compiling PCRE2), but that seems to have resulted in the same error: https://travis-ci.org/BurntSushi/ripgrep/jobs/524438703

Keeping PCRE2 disabled, I then tried using cross in the CI build, and that worked: https://travis-ci.org/BurntSushi/ripgrep/jobs/524440678 But it fails on i686-unknown-linux-musl: https://travis-ci.org/BurntSushi/ripgrep/jobs/524440677

And re-enabling PCRE2 with cross fails: https://travis-ci.org/BurntSushi/ripgrep/jobs/524447301

Here is the pcre2-sys build script: https://github.com/BurntSushi/rust-pcre2/blob/master/pcre2-sys/build.rs

So this seems like quite a thorny problem. There are multiple variables at play here:

• A normal cargo build with musl and with PCRE2 fails.
• A normal cargo build with musl but without PCRE2 fails. (Seemingly same error as above.)
• A cross build with musl but without PCRE2 works on x86_64.
• A cross build with musl but without PCRE2 fails on i686 with a distinct error from anything else, i.e., undefined reference to __ffsdi2'`. This appears to be this bug: Rust regression i686-unknown-linux-musl has started to fail jemalloc/jemalloc#1464
• A cross build with musl but with PCRE2 fails, but with a distinct error in the build script. It looks like calling git inside cross's docker container doesn't work because git isn't available.

I've never used cross before, but the normal Cargo builds worked with musl when Rust was bundling jemalloc.

[gnzlbg]

But it fails on i686-unknown-linux-musl

jemalloc does not support that target anymore, so if you want to use jemalloc there you'd have to patch jemalloc itself

[gnzlbg]

And re-enabling PCRE2 with cross fails: https://travis-ci.org/BurntSushi/ripgrep/jobs/524447301

From inspecting that, it errors at line 75 of the build.rs: https://github.com/BurntSushi/rust-pcre2/blob/master/pcre2-sys/build.rs#L75

Either git is not available inside cross (IIRC this should work - or was recently added), or the command might be trying to modify the source directory, which cross forbids, e.g., if you want to checkout a git repo, cross forces doing that into the OUT_DIR (e.g. target/.../) - the source directory is read-only while building.

[BurntSushi]

Yeah, I fixed that. You can see a passing build with cross + jemalloc + Ubuntu 16.04 + musl + x86_64 here: https://travis-ci.org/BurntSushi/ripgrep/jobs/524515554

So it sounds like cross is working OK. But the normal cargo build --target x86_64-unknown-linux-musl still fails.

[gnzlbg]

But the normal cargo build --target x86_64-unknown-linux-musl still fails.

So I suspect that your cross-compilation toolchain for C might not be set up appropriately for compiling jemalloc. I think it might be better to try compiling jemalloc from source first and see if that succeeds. Basically:

git clone https://github.com/jemalloc/jemalloc
cd jemalloc
autoconf
./configure
make -j3 
make test -j3

But the autoconf / makefile environment variables (CC, CXX, TARGET, HOST, ... ) need to be set properly for that to work. If that works, but cargo build --target=... does not, then we have a problem in the build.rs. If that does not work, then maybe something in the environment / cross-compilation toolchain might be making the build of jemalloc fail.

[BurntSushi]

Now that I have builds working again with cross, I'm not sure if or when I'll be debugging jemalloc compilation unfortunately. I'm not particularly experienced with configuring cross compilation toolchains, and I don't have an easy way to test an Ubuntu 16.04 box at the moment, so it would probably be a lot of work for me. If I get some spare time I'll see about looking into it.

I expect this is likely to bite people who are producing static binaries with musl using Travis CI if and when they notice that musl's allocator is dog slow and want to switch back to jemalloc.

[gnzlbg]

cross exists to save you from having to know about this, so if that works for you I would call it a day and worry more about it when you hit "the next issue".

Emitting better diagnostic when somebody attempts to build the i686 musl target is being tracked here in case you want to follow that: #106