Sudden failure to link statically on Arch Linux

[sanmai-NL]

It's always worked before. Turning back to compiling Rust after some period of other activities, and found out my binaries don't work anymore.

cargo new --bin bla

     Created binary (application) `bla` project

cd /tmp/bla/

env RUSTFLAGS="-C target-feature=+crt-static" cargo build --target x86_64-unknown-linux-musl

   Compiling bla v0.1.0 (file:///tmp/bla)
    Finished dev [unoptimized + debuginfo] target(s) in 0.35 secs

Actual

file ./target/x86_64-unknown-linux-musl/debug/bla

./target/x86_64-unknown-linux-musl/debug/bla: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8c6e0c77c39c031c76d95379bfe2593165b13a70, with debug_info, not stripped

Expected

A description of an executable statically linked to musl libc.

Meta

rustc 1.21.0-nightly (1d2a6df38 2017-08-03)

OS

Arch Linux with musl package version 1.1.16-2.

[sanmai-NL]

Downgrading musl package to version 1.1.16-1 does not help. Any other options for testing?

[aidanhs]

Can you paste the output of ldd on the binary as well?

[sanmai-NL]

env -u LD_PRELOAD ldd -v ./target/x86_64-unknown-linux-musl/debug/bla

        statically linked

That seems fine then. The file output is normally correct, however, see e.g. to the bottom of this blog post.

My issue came up when I was trying to run the statically linked executables under Alpine Linux, as before. See this weird situation:

sudo env UNIFIED_CGROUP_HIERARCHY=no \
    rkt run --insecure-options=image docker://alpine:3.5 \
        --volume=bla,kind=host,source=/tmp/bla/target/x86_64-unknown-linux-musl/debug/ \
        --mount volume=bla,target=/bla/ \
        --exec=/bin/sh -- -c 'ldd -- /bla/bla; stat /bla/bla; /bla/bla; stat /lib64/ld-linux-x86-64.so.2'

[114845.186083] alpine[5]:      /lib64/ld-linux-x86-64.so.2 (0x55a8e0d15000)
[114845.186864] alpine[5]:   File: /bla/bla
[114845.187294] alpine[5]:   Size: 3912560      Blocks: 7648       IO Block: 4096   regular file
[114845.187433] alpine[5]: Device: 26h/38d      Inode: 3618974     Links: 2
[114845.187514] alpine[5]: Access: (0755/-rwxr-xr-x)  Uid: ( 1000/ UNKNOWN)   Gid: ( 1000/ UNKNOWN)
[114845.187583] alpine[5]: Access: 2017-08-04 20:28:41.000000000
[114845.187651] alpine[5]: Modify: 2017-08-04 18:23:00.000000000
[114845.187721] alpine[5]: Change: 2017-08-04 20:19:38.000000000
[114845.187793] alpine[5]: /bin/sh: /bla/bla: not found
[114845.187963] alpine[5]: stat: can't stat '/lib64/ld-linux-x86-64.so.2': No such file or directory

Seems the interpreter path isn't correct.

[aidanhs]

I've just tried to follow your steps on the latest nightly and I can't reproduce this - file reports the binary as statically linked. I have a vague recollection that Arch does some custom packaging of rust - can you confirm whether you're using the binary from rustup, or from arch itself?

Your choice of the version of musl package shouldn't make a difference (unless it's related to the custom packaging I mention) because Rust bundles the musl libc.a with the musl target.

[sanmai-NL]

Heh, it appears I had rustup installed as an Arch Linux package and via the common installer. Also, cargo now is included in the rustup Arch Linux package. I rectified the situation to only use the Arch Linux packaged rustup. I definitely use the x86_64-unknown-linux-musl target with the nightly-x86_64-unknown-linux-gnu toolchain. Nevertheless, the statically linked binaries still aren't being produced properly.

[sfackler]

I can reproduce this as well with rustup-managed 1.19 on Arch, but not on Debian Jessie, where the file type is

target/x86_64-unknown-linux-musl/debug/foo: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, not stripped

It seems like it'd probably have to do with the way Arch's musl toolchain is configured?

[sfackler]

Though it looks like you haven't told rustc to use musl-gcc as the linker, since that makes the resulting binary use /lib/ld-musl-x86_64.so.1 as its interpreter rather than /lib64/ld-linux-x86-64.so.2.

[aidanhs]

@sfackler unless I've misunderstood the intent, linking a static binary with the musl target should not require a custom linker unless this is an oddity of arch. In fact, the host musl toolchain should be totally unused (unless depending on a crate that invokes musl-gcc as part of a build script).

[sfackler]

Ah! There is a difference between Debian and Arch's musl-gcc wrappers - Debian's pulls in spec files that disable PIE in addition to the normal MUSL one.

This custom linker setup produces fully static binaries on Arch:

test-gcc

#!/bin/sh                                                                                                                                                                                                                                              
exec "${REALGCC:-gcc}" "$@" -specs "$HOME/no-pie-compile.specs" -specs "$HOME/no-pie-link.specs"

no-pie-link.specs

*self_spec:                                                                                                                                                                                                                                            
+ %{!shared:%{!r:%{!fPIE:%{!pie:-fno-PIE -no-pie}}}}

*self_spec:                                                                                                                                                                                                                                            
+ %{!r:%{!fpie:%{!fPIE:%{!fpic:%{!fPIC:%{!fno-pic:-fno-PIE}}}}}}

It looks like we may want to disable PIE for MUSL targets.

[sfackler]

We already appear to not enable PIE for MUSL targets, but it looks like we should actually explicitly be disabling PIE rather than relying on the linker default.

rust/src/librustc_back/target/linux_musl_base.rs

Line 68 in cd72f2e

base.position_independent_executables = false;

[sfackler]

It seems like #40113 may resolve this.

[Mark-Simulacrum]

Nominating for prioritization, though it looks like this is potentially already fixed? Confirmation would be good.

[michaelwoerister]

Discussed in the compiler team meeting. P-high, we want to keep an eye on this.

[alexcrichton]

#40113 seems close to landing, just a final bug or two to iron out.

@sfackler I'm curious though shouldn't we land #43693 regardless? It seems like a "more correct" patch regardless of what happens to the musl target? We may just stop exercising it once #40113 lands.

[sfackler]

I haven't been following #40113 super closely, but it seemed like it'd end up generating statically linked PIE executables, which seems better than statically linked non-PIE executables, right?

[alexcrichton]

Looking at this again, this is tagged as a regression but we don't know what caused it. @sanmai-NL do you have a way for me to reproduce locally? Failing that, do you know when this stopped working? If so, you wouldn't happen to know the relevant change, would you?

@sfackler oh sure yeah I don't even remember at this point why musl is non-PIE. If we have future targets that are not PIE (for whatever reason) though it seems like they could benefit from #43693

[sanmai-NL]

@alexcrichton:
This shell code is supposed to reproduce it ... but doesn't, so far! Any regression is probably because of Arch Linux changes.

mkdir /tmp/bla/ || rm -fr /tmp/bla/ &&
sudo env UNIFIED_CGROUP_HIERARCHY=no \
    rkt run --insecure-options=image 'docker://stephank/archlinux:makepkg' \
        --net=host \
        --dns=host \
        --user=0 \
        --working-dir='/bla/' \
        --volume=bla,kind=host,source=/tmp/bla/ \
        --mount volume=bla,target=/bla/ \
        --exec=/bin/sh -- -c '
        set -x
        yes | pacman -Sy rustup musl pax-utils &&
        rustup install nightly-x86_64-unknown-linux-gnu &&
        rustup default nightly-x86_64-unknown-linux-gnu &&
        rustup target add x86_64-unknown-linux-musl &&
        cargo init --bin --name bla --vcs none . &&
        env RUSTFLAGS="-C target-feature=+crt-static" \
            cargo build --target x86_64-unknown-linux-musl &&
        file target/x86_64-unknown-linux-musl/debug/bla
        scanelf -ain target/x86_64-unknown-linux-musl/debug/bla
        target/x86_64-unknown-linux-musl/debug/bla'
        
sudo env UNIFIED_CGROUP_HIERARCHY=no \
    rkt run --insecure-options=image 'docker://alpine' \
        --working-dir='/bla/' \
        --net=host \
        --dns=host \
        --volume=bla,kind=host,source=/tmp/bla/ \
        --mount volume=bla,target=/bla/ \
        --exec=/bin/sh -- -c '
        set -x
        apk add --update file
        target/x86_64-unknown-linux-musl/debug/bla
        ldd target/x86_64-unknown-linux-musl/debug/bla
        file target/x86_64-unknown-linux-musl/debug/bla'

On my own system the issue is still present though. Apparently the regression is between the age of the Docker image and the date of this GitHub issue. Apparently that is two months.

[alexcrichton]

@sfackler due to my most recent comment I believe we will end up needing #43693 regardless.

[alexcrichton]

@sfackler now that #40113 has landed would you be interested in trying to re-land #43693?

[aidanhs]

#44070 didn't land in the latest nightly, but I compiled it myself.

on my host ubuntu:16.04:

# echo 'fn main() {}' > x.rs
# ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc --target x86_64-unknown-linux-musl x.rs
# file x
x: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=45654470ff0026421c064536a65c073dbfceb840, not stripped
# ldd x
        not a dynamic executable

i.e. still works fine

then continuing in the same terminal with arch:

# docker run -it --rm -v $(pwd):/myrust -w /myrust base/archlinux bash
[root@6aa90f62adb2 myrust]# pacman -Sy curl file gcc
[...]
[root@6aa90f62adb2 myrust]# curl https://sh.rustup.rs -sSf | sh
[...]
[root@6aa90f62adb2 myrust]# source $HOME/.cargo/env
[root@6aa90f62adb2 myrust]# rustup target add x86_64-unknown-linux-musl
[...]
[root@6aa90f62adb2 myrust]# rustc --target x86_64-unknown-linux-musl x.rs
[root@6aa90f62adb2 myrust]# file x
x: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=5992690cf70188d0192de6841a4c9399b37d5cc7, with debug_info, not stripped
[root@6aa90f62adb2 myrust]# ldd x
        statically linked
[root@6aa90f62adb2 myrust]# ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc --target x86_64-unknown-linux-musl x.rs
[root@6aa90f62adb2 myrust]# file x
x: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=633f517e0c5a4fea53cf0033446edf1af079d64c, with debug_info, not stripped
[root@6aa90f62adb2 myrust]# ldd x
        not a dynamic executable

i.e. I verified I could reproduce with stable rust, but the recompiled rust fixes the issue.

I'm going to close this since all the pieces are are in place and you should be able to take advantage in the next nightly. Ping me or open an issue if you have any other issues - thanks for reporting!