Miri unhelpfully fails to run Xargo on MacOS since nightly-2021-12-05

[saethlin]

Totally fresh setup and a recent MacOS. I'm reporting this because I'm currently having a conversation with a user on the community Discord, and I bisected to a working nightly (on my system at leasts) for them.

The last working nightly I could find by bisecting is 2021-12-04.

   Compiling compiler_builtins v0.1.55
    Checking core v0.0.0 (/Users/ben/.rustup/toolchains/nightly-2021-12-05-x86_64-apple-darwin/lib/rustlib/src/rust/library/core)
   Compiling libc v0.2.108
   Compiling cc v1.0.69
   Compiling memchr v2.4.1
   Compiling std v0.0.0 (/Users/ben/.rustup/toolchains/nightly-2021-12-05-x86_64-apple-darwin/lib/rustlib/src/rust/library/std)
error: could not compile `core`

Caused by:
  process didn't exit successfully: `/Users/ben/.rustup/toolchains/nightly-2021-12-05-x86_64-apple-darwin/bin/cargo-miri --crate-name core --edition=2018 /Users/ben/.rustup/toolchains/nightly-2021-12-05-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata -C opt-level=3 -C embed-bitcode=no -C metadata=44fd6ae2a7589a68 -C extra-filename=-44fd6ae2a7589a68 --out-dir /var/folders/s4/47bpcs6d63s111tmlndmx8fr0000gn/T/xargo.s7fT2PJxjaXE/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/var/folders/s4/47bpcs6d63s111tmlndmx8fr0000gn/T/xargo.s7fT2PJxjaXE/target/x86_64-apple-darwin/release/deps -L dependency=/var/folders/s4/47bpcs6d63s111tmlndmx8fr0000gn/T/xargo.s7fT2PJxjaXE/target/release/deps -Cdebug-assertions=off -Coverflow-checks=on --sysroot /Users/ben/Library/Caches/org.rust-lang.miri/HOST -Z force-unstable-if-unmarked` (exit status: 255)
warning: build failed, waiting for other jobs to finish...
error: build failed
error: `"/Users/ben/.rustup/toolchains/nightly-2021-12-05-x86_64-apple-darwin/bin/cargo" "check" "--release" "--manifest-path" "/var/folders/s4/47bpcs6d63s111tmlndmx8fr0000gn/T/xargo.s7fT2PJxjaXE/Cargo.toml" "--target" "x86_64-apple-darwin" "-p" "std"` failed with exit code: Some(101)
fatal error: failed to run xargo

It would be very nice to have a better error message. Is cargo-miri swallowing an error from xargo?

[KSBilodeau]

As of 2021-12-05, miri fails to build core if 2021-12-04 is not installed with the following error:

error: could not compile `core`

Caused by:
  process didn't exit successfully: `/Users/ksbilodeau/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/cargo-miri --crate-name core --edition=2018 /Users/ksbilodeau/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,future-incompat --crate-type lib --emit=dep-info,metadata -C opt-level=3 -C embed-bitcode=no -C metadata=ace66a2df2cc3ca8 -C extra-filename=-ace66a2df2cc3ca8 --out-dir /var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.z6aDGQc5PMmh/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.z6aDGQc5PMmh/target/x86_64-apple-darwin/release/deps -L dependency=/var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.z6aDGQc5PMmh/target/release/deps -Cdebug-assertions=off -Coverflow-checks=on --sysroot /Users/ksbilodeau/Library/Caches/org.rust-lang.miri/HOST -Z force-unstable-if-unmarked` (exit status: 255)
warning: build failed, waiting for other jobs to finish...
error: build failed
error: `"/Users/ksbilodeau/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/cargo" "check" "--release" "--manifest-path" "/var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.z6aDGQc5PMmh/Cargo.toml" "--target" "x86_64-apple-darwin" "-p" "std"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace
fatal error: failed to run xargo

if 2021-12-04 is installed and 2021-12-05+ is used, then proc_macro fails to compile with core compiling:

error: could not compile `proc_macro`

Caused by:
  process didn't exit successfully: `/Users/ksbilodeau/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/cargo-miri --crate-name proc_macro --edition=2018 /Users/ksbilodeau/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/src/rust/library/proc_macro/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,future-incompat --crate-type lib --emit=dep-info,metadata -C opt-level=3 -C embed-bitcode=no -C metadata=d54c47827132156a -C extra-filename=-d54c47827132156a --out-dir /var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.96UdjaQCav71/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.96UdjaQCav71/target/x86_64-apple-darwin/release/deps -L dependency=/var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.96UdjaQCav71/target/release/deps --extern std=/var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.96UdjaQCav71/target/x86_64-apple-darwin/release/deps/libstd-ffb98edf9742841a.rmeta -Cdebug-assertions=off -Coverflow-checks=on --sysroot /Users/ksbilodeau/Library/Caches/org.rust-lang.miri/HOST -Z force-unstable-if-unmarked` (exit status: 255)
warning: build failed, waiting for other jobs to finish...
error: build failed
error: `"/Users/ksbilodeau/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/cargo" "check" "--release" "--manifest-path" "/var/folders/gq/4ccrkb_120dg91cptjyljz740000gn/T/xargo.96UdjaQCav71/Cargo.toml" "--target" "x86_64-apple-darwin" "-p" "test"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace
fatal error: failed to run xargo

[RalfJung]

That is odd, since CI is still working (including on macOS) with later versions of rustc. And at least on Linux, later versions from rustup also definitely work.

cargo-miri should forward xargo stdout and stderr (otherwise you wouldn't even see things like "Compiling std v0.0.0").

if 2021-12-04 is installed and 2021-12-05+ is used

Yeah, mixing different versions will not work.

[RalfJung]

Does it work for you to clone the Miri source, rustup override set nightly[-date] in its directory, and then ./miri test vec? You'll need some extra rustup components: rust-src, rustc-dev, llvm-tools.

Which are the rustc commits associated with the 2021-12-04 and 2021-12-05 nightlies?

[RalfJung]

Which are the rustc commits associated with the 2021-12-04 and 2021-12-05 nightlies?

That would be 532d2b14c and efec54529. That's this commit range. On first sight I cannot really see anything here that could explain a regression.

[RalfJung]

This seems to be reproducible on the miri-test-libstd CI if I switch it to macOS.

But I can't really investigate the problem that way so I hope someone with access to a macOS machine can take a closer look.

[saethlin]

@RalfJung What information could I get for you that would be useful? I'm a casual MacOS user at best. I tried to dtruss the failing command and I couldn't find any errors that looked out of place (all the ones I found were linkers searching for libraries and the like).

[saethlin]

If I just cargo miri run a second time without cleaning the miri cache or the local build... the build completes. Sometimes I need to do this three times, and I get a build error again with no error message on proc_macro. Maybe this information helps?

[KSBilodeau]

@RalfJung I have some very important news. Using the Mac console, I was able to capture the miri crashes and determine the root cause of the crash.

Application Specific Information:
miri(77300,0x700002cc6000) malloc: *** error for object 0x6000070c1700: pointer being realloc'd was not allocated
abort() called

However, this seems to not be localized to miri. Today I also discovered that Clippy fails to run, and the crash log reports the same error

Application Specific Information:
clippy-driver(20172,0x7000098d8000) malloc: *** error for object 0x600002e23b00: pointer being realloc'd was not allocated
abort() called

Hence, I am afraid this issue may be much widespread than first anticipated. Would you like the full crash logs?

[RalfJung]

If I just cargo miri run a second time without cleaning the miri cache or the local build... the build completes. Sometimes I need to do this three times, and I get a build error again with no error message on proc_macro. Maybe this information helps?

Wow, what the heck...

I have some very important news. Using the Mac console, I was able to capture the miri crashes and determine the root cause of the crash.

Oh, it's a segfault? I am doubting more and more that this is caused by anything Miri does. In particular since you are seeing the same problem with clippy. So this is probably a rustc issue.

It would be good to get a backtrace from the crash, but I don't know how to do that on macOS.

[KSBilodeau]

If I just cargo miri run a second time without cleaning the miri cache or the local build... the build completes. Sometimes I need to do this three times, and I get a build error again with no error message on proc_macro. Maybe this information helps?

Wow, what the heck...

I have some very important news. Using the Mac console, I was able to capture the miri crashes and determine the root cause of the crash.

Oh, it's a segfault? I am doubting more and more that this is caused by anything Miri does. In particular since you are seeing the same problem with clippy. So this is probably a rustc issue.

It would be good to get a backtrace from the crash, but I don't know how to do that on macOS.

miri.txt

Does this by chance contain what you need?

[oli-obk]

Does this by chance contain what you need?

Yes! Looks like this happens during the thread join. I remember a recent macos codegen bug around threads. I am on mobile right now, but check the rustc issues for recent macos crashes

[RalfJung]

I can't find any such issue. The closest I got is rust-lang/rust#91124 but it does not mention thread joining.

[KSBilodeau]

@RalfJung having provided the logs to the rust repo as well, I believe this issue may be related to rust-lang/rust#92173 at least the part in which miri fails to run even after it compiles core and proc_macro. Still have no idea what is causing the first issue unfortunately.

[RalfJung]

Looks like this is now being tracked at rust-lang/rust#92185, but I'll keep this open so that Miri users can find it more easily.

[saethlin]

This seems to have stopped working breaking (there are no crashes anymore, English is hard) since Command Line Tools for Xcode 13.3. That was released a few months ago now.

It's not entirely clear to me if there was a fix or Apple made some unrelated change that stopped this from happening. But in any case, this isn't a problem at the moment.

[RalfJung]

This seems to have stopped working since Command Line Tools for Xcode 13.3

Stopped working or stopped breaking?

[saethlin]

Stopped breaking. 😂 I'll edit.