Compile time+memory regression between 1.49.0 and 1.50.0

Our compile times and memory footprint regressed substantially between Rust 1.49.0 and 1.50.0.

This regression is a hard blocker for us to be able to upgrade our project from 1.49.0. As this is a severe issue for us, please let us know if there's anything we can do to provide more diagnostics, test changes, etc.

This may be related to other recent issues that describe similar behavior:

* https://github.com/rust-lang/rust/issues/82406
* https://github.com/rust-lang/rust/issues/83911

### Code

This regression is observed on recent versions of the [Linkerd proxy](https://github.com/linkerd/linkerd2-proxy/tree/bba24dcdeda40105615db8d33e3fa04f980cf128). It's known that the proxy can manifest large type signatures, so builds [disable debug symbols by default](https://github.com/linkerd/linkerd2-proxy/blob/bba24dcdeda40105615db8d33e3fa04f980cf128/Cargo.toml#L57-L62)

This regression is not obvious with other, smaller projects that we maintain, so I'm unable to provide a smaller repro.

### Version it worked on

With Rust 1.49.0, the binary compiles in a little over two minutes, using a little over 1GB of memory:

`cargo clean && /usr/bin/time -v cargo +1.49.0 build -p linkerd2-proxy`
```
...
    Finished dev [unoptimized] target(s) in 2m 12s
        Command being timed: "cargo +1.49.0 build -p linkerd2-proxy"
        User time (seconds): 326.00
        System time (seconds): 14.55
        Percent of CPU this job got: 256%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:12.78
...
        Maximum resident set size (kbytes): 1308088
...
        Exit status: 0
```

### Version with regression

Using Rust 1.50.0, rustc runs for about 40 minutes before exhausting the system's memory:

`cargo clean && /usr/bin/time -v cargo +1.50.0 build -p linkerd2-proxy`
```
  process didn't exit successfully: `rustc --crate-name linkerd2_proxy --edition=2018 linkerd2-proxy/src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no --cfg 'feature="default"' --cfg 'feature="multicore"' --cfg 'feature="num_cpus"' -C metadata=c2caa01c24c285f7 -C extra-filename=-c2caa01c24c285f7 --out-dir /home/ver/b/linkerd2-proxy/target/debug/deps -C incremental=/home/ver/b/linkerd2-proxy/target/debug/incremental -L dependency=/home/ver/b/linkerd2-proxy/target/debug/deps --extern futures=/home/ver/b/linkerd2-proxy/target/debug/deps/libfutures-01f6240a9e4201d3.rlib --extern linkerd_app=/home/ver/b/linkerd2-proxy/target/debug/deps/liblinkerd_app-dd18fcabeef913ae.rlib --extern linkerd_signal=/home/ver/b/linkerd2-proxy/target/debug/deps/liblinkerd_signal-14263c92fcb0fc0b.rlib --extern num_cpus=/home/ver/b/linkerd2-proxy/target/debug/deps/libnum_cpus-74ee144d51bc047e.rlib --extern tokio=/home/ver/b/linkerd2-proxy/target/debug/deps/libtokio-de277ade05efd960.rlib --extern tracing=/home/ver/b/linkerd2-proxy/target/debug/deps/libtracing-03fd6826b817dabe.rlib -L native=/home/ver/b/linkerd2-proxy/target/debug/build/ring-a8b802a4aa425398/out` (signal: 9, SIGKILL: kill)
Command exited with non-zero status 101
	Command being timed: "cargo +1.50.0 build -p linkerd2-proxy"
	User time (seconds): 2986.18
	System time (seconds): 101.73
	Percent of CPU this job got: 133%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 38:29.28
...
	Maximum resident set size (kbytes): 62953524
...
	Exit status: 101
```

`rustc --version --verbose`:
```
rustc 1.50.0 (cb75ad5db 2021-02-10)
binary: rustc
commit-hash: cb75ad5db02783e8b0222fee363c5f63f7e2cf5b
commit-date: 2021-02-10
host: x86_64-unknown-linux-gnu
release: 1.50.0
```

We see similar behavior with more recent versions of Rust as well, including 1.51.0 and nightly (`cargo 1.53.0-nightly (0ed318d18 2021-04-23)`) as well.

cc @hawkw, who can provide some more details. I believe we've tested with `lto=off` without any changes in behavior.

@rustbot modify labels: +regression-from-stable-to-stable -regression-untriaged


	let (codegen_units, _) = tcx.sess.time("partition_and_assert_distinct_symbols", \|\| {
	sync::join(
	\|\| {
	&*tcx.arena.alloc_from_iter(partition(
	tcx,
	&mut items.iter().cloned(),
	tcx.sess.codegen_units(),
	&inlining_map,
	))
	},
	\|\| assert_symbols_are_distinct(tcx, items.iter()),
	)
	});

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Compile time+memory regression between 1.49.0 and 1.50.0 #84873

Code

Version it worked on

Version with regression

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Participants

Compile time+memory regression between 1.49.0 and 1.50.0 #84873

Description

Code

Version it worked on

Version with regression

Activity

jyn514 commented on May 3, 2021

jyn514 commented on May 3, 2021

olix0r commented on May 3, 2021

olix0r commented on May 3, 2021

jyn514 commented on May 3, 2021

13 remaining items

audunska commented on May 14, 2021

audunska commented on Aug 6, 2021

apiraino commented on Aug 6, 2021

olix0r commented on Aug 9, 2021

pnkfelix commented on Nov 18, 2022

lqd commented on Nov 19, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Participants

Issue actions