Enforce in bootstrap that check must have stage at least 1 #143048

Kobzol · 2025-06-26T08:01:58Z

This PR is another step towards https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523586917, this time dealing with x check.

It enforces the invariant that:

We check std stage N with rustc stage N
We check everything else stage N with rustc stage N - 1

It creates a single function that prepares a proper build compiler for checking something, and also adds snapshot tests for various common check steps. Some obsolete code was also removed.

The default check stage also becomes 1, for all profiles. I tested manually that x check std with download-ci-rustc still works and doesn't build rustc locally.

Subsumes #139170.

r? @ghost

Kobzol · 2025-06-26T08:04:11Z

CC @jieyouxu (one of many heaps of bootstrap fixup commits I have in store 😆)

bors · 2025-06-26T10:22:00Z

☔ The latest upstream changes (presumably #141899) made this pull request unmergeable. Please resolve the merge conflicts.

Kobzol · 2025-06-26T14:31:27Z

Cross-compilation checks are now doing too much work, I need to take a look at how we can reduce that, because x check --stage1 with cross-compilation is no longer a thing.

bors · 2025-07-01T03:25:36Z

☔ The latest upstream changes (presumably #143254) made this pull request unmergeable. Please resolve the merge conflicts.

…fault check stage for it

rustbot · 2025-07-01T15:44:03Z

This PR modifies src/bootstrap/defaults.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

Kobzol · 2025-07-01T15:44:13Z

r? @jieyouxu

rustbot · 2025-07-01T15:44:16Z

jieyouxu is not on the review rotation at the moment.
They may take a while to respond.

rust-log-analyzer · 2025-07-01T16:02:17Z

The job mingw-check-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#16 3.096 Building wheels for collected packages: reuse
#16 3.098   Building wheel for reuse (pyproject.toml): started
#16 3.311   Building wheel for reuse (pyproject.toml): finished with status 'done'
#16 3.312   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132719 sha256=d2a2565e7037ad3883fb9337653f2e25bbb588534fbef3697286cbc26d1bf634
#16 3.312   Stored in directory: /tmp/pip-ephem-wheel-cache-4cv2rae2/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#16 3.314 Successfully built reuse
#16 3.315 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#16 3.711 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#16 3.712 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#16 DONE 3.8s

bors · 2025-07-01T22:49:32Z

☔ The latest upstream changes (presumably #143287) made this pull request unmergeable. Please resolve the merge conflicts.

jieyouxu

I like this a lot in general, it does feel much more self-consistent and coherent.

I left some questions/discussions for things that I wanted to double-check, but overall looks good!

jieyouxu · 2025-07-02T05:36:49Z

src/bootstrap/src/core/build_steps/check.rs

-        // We skip populating the sysroot in non-zero stage because that'll lead
-        // to rlib/rmeta conflicts if std gets built during this session.
-        if compiler.stage == 0 {
-            let libdir = builder.sysroot_target_libdir(compiler, target);
-            let hostdir = builder.sysroot_target_libdir(compiler, compiler.host);
-            add_to_sysroot(builder, &libdir, &hostdir, &stamp);
-        }


Remark: we really do have a lot of hacks for stage 0 std...

jieyouxu · 2025-07-02T05:44:55Z

src/bootstrap/src/core/builder/mod.rs

+        self.stage.or(self
+            .built_by
+            .map(|compiler| if self.name == "std" { compiler.stage } else { compiler.stage + 1 }))


Discussion: this makes me feel a bit uneasy, but I guess the logic here is: anything that's not the standard library should be considered a non-library artifact, and should be considered a product of the stage n (compiler, standard library) pair (and so gets numbered as the next stage)? I.e. reading back https://hackmd.io/QXj6LqntQTKPgm3uBuE-aA again, the core model is

stage corresponds to what gets built, not what is used to build it

(The following is me trying to make sure I understand this)

Let's call the (compiler, library) pair¹²
$$\mathbf{CLP}_{\mathrm{stage}} := \left\langle \text{compiler}_{\mathrm{stage}}, \text{library}_{\mathrm{stage}} \right\rangle$$
where the separation between stages is delineated by which $\mathrm{compiler}$ gets built. Then, by design, because the standard library is "special" and tightly coupled with the compiler, we fix the immediate standard library built by $\mathrm{compiler}_{\mathrm{stage}}$ to be of the same $\mathrm{stage}$, i.e. $\mathrm{library}_{\mathrm{stage}}$.

With the stage 0 redesign, we now have a nice straightforward "derivation chain" ($\longrightarrow$; artifacts can be used to build new artifacts) property where
$$\left\langle \text{compiler}_{\mathrm{stage}}, \text{library}_{\mathrm{stage}} \right\rangle \longrightarrow \left\langle \text{compiler}_{\mathrm{stage + 1}}, \text{library}_{\mathrm{stage + 1}} \right\rangle$$
actually holds even for $\mathrm{stage} = 0$.

However, for:

bootstrap tools
Tools that are unconditionally built by the stage 0 $\mathbf{CLP}$, i.e. always depends on $\mathbf{CLP}_{0}$. Their stage numbering will be called $\mathrm{BootstrapTool}_{1}$, at least in the short-term.

host rustc tools
Tools that depend on rustc_private and so on $\mathrm{compiler}_{\mathrm{stage}}$. This means that they will be numbered as $\mathrm{HostRustcTool}_{\mathrm{stage} + 1}$.

std tools
Tools that depend on the whole $\mathbf{CLP}_{\mathrm{stage}}$, i.e. both $\mathrm{compiler}_{\mathrm{stage}}$, and $\mathrm{library}_{\mathrm{stage}}$. Similiarly, this means that they will be numbered as $\mathrm{StdTool}_{\mathrm{stage} + 1}$.

They then follow the general "what gets built" model (?).

Footnotes

I can't come up with a better name, it's not exactly a toolchain, neither is it a "complete" sysroot I think ↩

Not to be confused with the Chilean Peso. ↩

src/bootstrap/src/core/builder/tests.rs

jieyouxu · 2025-07-02T07:14:11Z

src/bootstrap/src/core/builder/tests.rs

+            ctx.config("check")
+                .path("library")
+                .render_steps(), @r"
+        [build] llvm <host>
+        [build] rustc 0 <host> -> rustc 1 <host>
+        [check] rustc 1 <host> -> std 1 <host>


Discussion (not for this PR): I do wonder how we would encode / fix the download-rustc logic here. I can see why the previous pretend-to-be- $\mathrm{CLP}_{2}$ scheme is very confusing.

jieyouxu · 2025-07-02T07:16:30Z

src/bootstrap/src/core/builder/tests.rs

+        [build] llvm <host>
+        [build] rustc 0 <host> -> rustc 1 <host>
+        [build] rustc 1 <host> -> std 1 <host>
+        [check] compiletest <host>


Remark: yes, this looks right 👍

jieyouxu · 2025-07-02T07:54:55Z

src/bootstrap/src/core/build_steps/check.rs

-            &compiler.host,
-            &target,
-        );
-        run_cargo(builder, cargo, builder.config.free_args.clone(), &stamp, vec![], true, false);


Question: I don't remember, does the check macro support free args? I don't think this is actually equivalent, specifically the free_args part. Looking at src/tools/coverage-dump, it supports a --demangle flag:

rust/src/tools/coverage-dump/src/main.rs

Lines 15 to 17 in f51c987

if &args[1..] == &["--demangle"] {

return demangle();

}

Although, as the check step, this does not matter, right?

jieyouxu · 2025-07-02T07:56:39Z

src/bootstrap/src/core/build_steps/check.rs

+            // A small number of tools rely on in-tree standard
+            // library crates (e.g. compiletest needs libtest).


Nit: compiletest needs internal library feature from in-tree std (which could be different from stage 0 std), but no longer libtest.

jieyouxu · 2025-07-02T08:00:08Z

src/bootstrap/src/core/build_steps/check.rs

            builder.std(build_compiler, host);
-            builder.std(build_compiler, target);
+            builder.ensure(Std::new(build_compiler, target));


Question: wait, why is this different? For $\mathrm{stage} \ge 1$, isn't this equivalent? I.e.

rust/src/bootstrap/src/core/builder/mod.rs

Lines 1393 to 1409 in f51c987

if compiler.stage == 0 {

if target != compiler.host {

panic!(

r"It is not possible to build the standard library for `{target}` using the stage0 compiler.

You have to build a stage1 compiler for `{}` first, and then use it to build a standard library for `{target}`.

",

compiler.host

)

}

// We still need to link the prebuilt standard library into the ephemeral stage0 sysroot

self.ensure(StdLink::from_std(Std::new(compiler, target), compiler));

} else {

// This step both compiles the std and links it into the compiler's sysroot.

// Yes, it's quite magical and side-effecty.. would be nice to refactor later.

self.ensure(Std::new(compiler, target));

}

At this point, wouldn't we have already rejected invoking ./x check flow with $\mathrm{stage} = 0$?

jieyouxu · 2025-07-02T08:02:19Z

src/bootstrap/src/core/builder/tests.rs

+                .targets(&[TEST_TRIPLE_1])
+                .hosts(&[TEST_TRIPLE_1])


Nit (tiny, just for consistency): can you swap targets() and hosts() around? Usually the ordering is hosts then targets right :D

jieyouxu · 2025-07-02T08:05:01Z

src/ci/docker/host-x86_64/mingw-check-1/Dockerfile

@@ -48,7 +48,7 @@ ENV SCRIPT \
           python3 ../x.py build src/tools/build-manifest && \
           python3 ../x.py test --stage 0 src/tools/compiletest && \
           python3 ../x.py check compiletest --set build.compiletest-use-stage0-libtest=true && \
-           python3 ../x.py check --stage 1 --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu && \
+           python3 ../x.py check --stage 2 --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu && \


Suggestion: actually, can you add a comment for this specifically that we're exercising cross-compile here, so --stage 1 isn't sufficient?

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 26, 2025

Kobzol force-pushed the bootstrap-check-stage-1 branch from f06e41e to aed5f23 Compare June 26, 2025 08:03

This comment has been minimized.

Sign in to view

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 26, 2025

Kobzol force-pushed the bootstrap-check-stage-1 branch from aed5f23 to a36ee24 Compare June 26, 2025 14:29

This comment has been minimized.

Sign in to view

Kobzol mentioned this pull request Jul 1, 2025

Cannot build rustc from master branch on Windows 10 #142754

Open

Kobzol and others added 15 commits July 1, 2025 17:24

Make default check stage be 1, and error out on checking with stage 0

2a9017d

Remove custom_stage override from check::Std and make 1 be the de…

3cbe55f

…fault check stage for it

Automatically derive stage in step metadata where possible

9dd3f3f

Add snapshot tests for checking compiler, library and rustc tools

e894e7a

Fix check compiletest with in-tree libstd and add tests for it

1df394b

Make build compiler explicit in check::Rustc and check::Std

2f440a4

Fixup codegen checking

f0d0ba8

Fixup rust-analyzer checking

c4f45a2

Fixup tool checking

e1c018d

Implement CoverageDump checking through the tool_check_step macro

089da89

Unify selection of build compiler for checking in a single function

9c9b1c3

Add change tracker entry

3b62e08

Fix CI

273e86f

Add test for check cross-compilation

04a3369

Fix check in mingw-check-1

22fb548

Kobzol force-pushed the bootstrap-check-stage-1 branch from a36ee24 to 22fb548 Compare July 1, 2025 15:43

Kobzol marked this pull request as ready for review July 1, 2025 15:44

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 1, 2025

rustbot assigned jieyouxu Jul 1, 2025

jieyouxu removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 2, 2025

Kobzol mentioned this pull request Jul 2, 2025

Add bootstrap check snapshot tests #143316

Open

jieyouxu reviewed Jul 2, 2025

View reviewed changes

		// A small number of tools rely on in-tree standard
		// library crates (e.g. compiletest needs libtest).

	if compiler.stage == 0 {
	if target != compiler.host {
	panic!(
	r"It is not possible to build the standard library for `{target}` using the stage0 compiler.
	You have to build a stage1 compiler for `{}` first, and then use it to build a standard library for `{target}`.
	",
	compiler.host
	)
	}

	// We still need to link the prebuilt standard library into the ephemeral stage0 sysroot
	self.ensure(StdLink::from_std(Std::new(compiler, target), compiler));
	} else {
	// This step both compiles the std and links it into the compiler's sysroot.
	// Yes, it's quite magical and side-effecty.. would be nice to refactor later.
	self.ensure(Std::new(compiler, target));
	}

Enforce in bootstrap that check must have stage at least 1 #143048

Are you sure you want to change the base?

Enforce in bootstrap that check must have stage at least 1 #143048

Conversation

Kobzol commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Jun 26, 2025

Uh oh!

This comment has been minimized.

bors commented Jun 26, 2025

Uh oh!

Kobzol commented Jun 26, 2025

Uh oh!

This comment has been minimized.

bors commented Jul 1, 2025

Uh oh!

rustbot commented Jul 1, 2025

Uh oh!

Kobzol commented Jul 1, 2025

Uh oh!

rustbot commented Jul 1, 2025

Uh oh!

rust-log-analyzer commented Jul 1, 2025

Uh oh!

bors commented Jul 1, 2025

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kobzol commented Jun 26, 2025 •

edited

Loading

jieyouxu Jul 2, 2025 •

edited

Loading

jieyouxu Jul 2, 2025 •

edited

Loading