Introduce -Zembed-metadata to allow omitting full metadata from rlibs and dylibs

[Kobzol]

This is a continuation of #120855 (I was mentored by @bjorn3 to move it forward). Most of the original code was written by bjorn3, I tried to clean it up a bit and add some documentation and tests.

This PR introduces a new unstable compiler flag called -Zembed-metadata=[no|yes], with the default being yes (see #57076 for context). When set to no, rustc will only store a small metadata stub inside rlibs/dylibs instead of the full metadata, to keep their size smaller. It should be used in combination with --emit=metadata, so that the users of such a compiled library can still read the metadata from the corresponding .rmeta file. This comment shows an example of binary/artifact size wins that can be achieved using this approach.

Contrary to #120855, this PR only introduces the new flag, along with a couple of run-make tests and documentation, but does not yet use it in bootstrap to actually compile rustc. I plan to do that as a follow-up step (along with integration in Cargo, which should ideally just always pass this flag to reduce the size of target directories).

Fixes #23366
Closes #29511
Fixes #57076

Another attempt of #93945 and #120855.

r? @petrochenkov

[Kobzol]

(The test failure is the one situation not yet implemented, which is described in the PR description - --extern plus .rmeta in a different directory than the rlib/dylib).

[petrochenkov]

I found it quite surprising that when --extern is passed, the -L search logic is completely skipped

That's by design, --extern with paths disables all the directory search, makes the build system own the input files, and makes rustc just do what the build system tells it.
I'd actually consider this the primary mode of work for rustc, and all the directory search is a sort of legacy (well, except -L dependency=PATH).

--extern supports passing multiple paths for the same crate, to specify all its rmeta, rlib and dylib versions.
So in case of split metadata I'd also expect cargo (or another build system) to pass path to the split rmeta explicitly, and rustc to not do any heuristics.

In #120855 the heuristics were a temporary measure to enable split metadata in bootstrap, while cargo still doesn't support it.
It should be fine to add them in your follow up PR implementing the bootstrap changes as well, but in the long run I don't think they should exist.

[Kobzol]

Interesting. So if I understood it correctly, you would like to remove all heuristics from rustc (so no lookup of .rmeta in known library search dirs and no ".rmeta in the same directory as .rlib" heuristic) and let Cargo pass essentially both .rlib and .rmeta for every dependency through --extern? I guess that's cleaner in a sense.

On the other hand, it would essentially double the length of the rustc command line, because for every library that you depend on, Cargo would have to pass both .rmeta and .rlib. Do you think that wouldn't be an issue? I suppose that it would only happen for the leaf linked artifact, because the intermediate libraries seem to only receive .rmeta files from Cargo, while the final artifact receives .rlib.

[petrochenkov]

So if I understood it correctly, you would like to remove all heuristics from rustc...

Yes.

On the other hand, it would essentially double the length of the rustc command line, because for every library that you depend on, Cargo would have to pass both .rmeta and .rlib.

My impression was that it is already the case with pipelined builds, but apparently it's indeed only .rmetas for dependencies and only .rlibs for the root crate.
I think it's fine if it happens only for direct dependencies of binary crates, in any case we can reconsider heuristics if there are any complaints about this.

[rustbot]

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

This PR modifies run-make tests.

cc @jieyouxu

[Kobzol]

Ok, rebased and removed the heuristics. I added an error message when the full metadata is not found, but I'm not sure if we should emit the error eagerly (left a comment in code). Could you please take a look at fe29fec? Thank you!

[Kobzol]

rust-lang/compiler-team#851 was accepted, so I think that we can go forward with this.

@petrochenkov What do you think about renaming the flag to -Zembed-metadata=[no|yes], with yes being the default, to be consistent with -Cembed-bitcode=[no|yes], which is a bit similar in spirit?

[petrochenkov]

What do you think about renaming the flag to -Zembed-metadata=[no|yes], with yes being the default, to be consistent with -Cembed-bitcode=[no|yes], which is a bit similar in spirit?

Seems ok to me.

[Kobzol]

Ok, changed it to -Zembed-metadata=[no|yes].

@rustbot ready

[Kobzol]

Created a tracking issue and fixed tests.

[petrochenkov]

r=me with the metadata version updated.
@rustbot author

[rustbot]

rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead.

cc @rust-lang/rust-analyzer

[Kobzol]

@bors r=petrochenkov rollup=never

Marked as rollup=never, because this modifies metadata version.

[bors]

📌 Commit f0efb97 has been approved by petrochenkov

It is now in the queue for this repository.

[bors]

⌛ Testing commit f0efb97 with merge 8c35f4a...

[bors]

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 8c35f4a to master...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing ed20157 (parent) -> 8c35f4a (this PR)

Test differences

Show 43 test diffs

Stage 1

• errors::verify_metadata_crate_location_unknown_type_68: pass -> [missing] (J0)
• errors::verify_metadata_crate_location_unknown_type_69: [missing] -> pass (J0)
• errors::verify_metadata_dl_error_63: pass -> [missing] (J0)
• errors::verify_metadata_dl_error_64: [missing] -> pass (J0)
• errors::verify_metadata_found_staticlib_66: pass -> [missing] (J0)
• errors::verify_metadata_found_staticlib_67: [missing] -> pass (J0)
• errors::verify_metadata_full_metadata_not_found_61: [missing] -> pass (J0)
• errors::verify_metadata_import_name_type_form_71: pass -> [missing] (J0)
• errors::verify_metadata_import_name_type_form_72: [missing] -> pass (J0)
• errors::verify_metadata_import_name_type_raw_74: pass -> [missing] (J0)
• errors::verify_metadata_import_name_type_raw_75: [missing] -> pass (J0)
• errors::verify_metadata_import_name_type_x86_72: pass -> [missing] (J0)
• errors::verify_metadata_import_name_type_x86_73: [missing] -> pass (J0)
• errors::verify_metadata_incompatible_rustc_67: pass -> [missing] (J0)
• errors::verify_metadata_incompatible_rustc_68: [missing] -> pass (J0)
• errors::verify_metadata_incompatible_target_modifiers_76: pass -> [missing] (J0)
• errors::verify_metadata_incompatible_target_modifiers_77: [missing] -> pass (J0)
• errors::verify_metadata_incompatible_target_modifiers_l_missed_77: pass -> [missing] (J0)
• errors::verify_metadata_incompatible_target_modifiers_l_missed_78: [missing] -> pass (J0)
• errors::verify_metadata_incompatible_target_modifiers_r_missed_78: pass -> [missing] (J0)
• errors::verify_metadata_incompatible_target_modifiers_r_missed_79: [missing] -> pass (J0)
• errors::verify_metadata_lib_filename_form_69: pass -> [missing] (J0)
• errors::verify_metadata_lib_filename_form_70: [missing] -> pass (J0)
• errors::verify_metadata_multiple_import_name_type_70: pass -> [missing] (J0)
• errors::verify_metadata_multiple_import_name_type_71: [missing] -> pass (J0)
• errors::verify_metadata_newer_crate_version_64: pass -> [missing] (J0)
• errors::verify_metadata_newer_crate_version_65: [missing] -> pass (J0)
• errors::verify_metadata_no_crate_with_triple_65: pass -> [missing] (J0)
• errors::verify_metadata_no_crate_with_triple_66: [missing] -> pass (J0)
• errors::verify_metadata_stable_crate_id_collision_62: pass -> [missing] (J0)
• errors::verify_metadata_stable_crate_id_collision_63: [missing] -> pass (J0)
• errors::verify_metadata_symbol_conflicts_current_61: pass -> [missing] (J0)
• errors::verify_metadata_symbol_conflicts_current_62: [missing] -> pass (J0)
• errors::verify_metadata_unknown_import_name_type_73: pass -> [missing] (J0)
• errors::verify_metadata_unknown_import_name_type_74: [missing] -> pass (J0)
• errors::verify_metadata_unknown_target_modifier_unsafe_allowed_79: pass -> [missing] (J0)
• errors::verify_metadata_unknown_target_modifier_unsafe_allowed_80: [missing] -> pass (J0)
• errors::verify_metadata_wasm_c_abi_75: pass -> [missing] (J0)
• errors::verify_metadata_wasm_c_abi_76: [missing] -> pass (J0)
• [run-make] tests/run-make/embed-metadata: [missing] -> pass (J1)

Stage 2

• [run-make] tests/run-make/embed-metadata: [missing] -> pass (J2)

Additionally, 2 doctest diffs were found. These are ignored, as they are noisy.

Job group index

• J0: aarch64-apple, aarch64-gnu, i686-gnu-2, i686-gnu-nopt-2, i686-mingw-2, i686-mingw-3, i686-msvc-2, x86_64-apple-1, x86_64-gnu, x86_64-gnu-llvm-18-3, x86_64-gnu-llvm-19-3, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-2, x86_64-msvc-2
• J1: x86_64-gnu-llvm-18-3, x86_64-gnu-llvm-19-3
• J2: aarch64-apple, aarch64-gnu, aarch64-gnu-debug, arm-android, armhf-gnu, dist-i586-gnu-i586-i686-musl, dist-various-1, i686-gnu-1, i686-gnu-nopt-1, i686-msvc-1, test-various, x86_64-apple-1, x86_64-gnu, x86_64-gnu-debug, x86_64-gnu-llvm-18-1, x86_64-gnu-llvm-18-2, x86_64-gnu-llvm-19-1, x86_64-gnu-llvm-19-2, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-1, x86_64-msvc-1

Job duration changes

1. aarch64-apple: 3668.0s -> 4135.9s (12.8%)
2. dist-x86_64-apple: 8228.8s -> 9021.5s (9.6%)
3. x86_64-gnu-aux: 6050.3s -> 6564.3s (8.5%)
4. x86_64-apple-2: 5146.4s -> 5562.5s (8.1%)
5. i686-gnu-2: 6154.4s -> 6543.8s (6.3%)
6. x86_64-mingw-1: 9132.0s -> 9660.3s (5.8%)
7. i686-msvc-2: 7121.3s -> 7515.9s (5.5%)
8. dist-x86_64-musl: 7165.8s -> 7498.9s (4.6%)
9. dist-x86_64-msvc: 6103.3s -> 6346.6s (4.0%)
10. mingw-check: 1251.6s -> 1295.7s (3.5%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (8c35f4a): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (secondary -1.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.7%	[-1.8%, -1.6%]	2
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary 3.9%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.9%	[3.9%, 3.9%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

Results (secondary 0.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 773.019s -> 774.818s (0.23%)
Artifact size: 365.92 MiB -> 365.93 MiB (0.00%)