compiletest: Add an experimental new executor to replace libtest #139660

Zalathar · 2025-04-11T06:55:51Z

This PR adds a new "executor" to compiletest for running the list of collected tests, to eventually replace the current dependency on unstable libtest internals.

The new executor is currently inactive by default. It must be activated explicitly by passing -n or --new-executor to compiletest, e.g. ./x test ui -- -n.

(After some amount of wider manual testing, the new executor will hopefully be made the default, and the libtest dependency can be removed. Contributors should not notice any change.)

The new executor is a stripped-down rewrite of the subset of libtest needed by compiletest.

Supported functionality

Specifying the number of concurrent tests with RUST_TEST_THREADS
Filtering and skipping tests by name (substring or exact-match)
Forcibly running ignored tests with --ignored
Optional fail-fast with --fail-fast
JSON output, compatible with bootstrap's parser for libtest output
Running each test in its own thread
Short backtraces that ignore the executor itself
Slow test detection, with a hard-coded timeout of 60 seconds
Capturing stdout/stderr, via #![feature(internal_output_capture)]
Suppressing output capture with --no-capture

Unsupported functionality

Non-JSON output, as this is handled by bootstrap instead
Separate code path for concurrency=1, as the concurrent path should handle this case naturally
Fallback to running tests synchronously if new threads can't be spawned
Special handling of hosts that don't support basic functionality like threads or timers
- Our ability to test targets should be unaffected
Graceful handling of some edge cases that could occur in arbitrary user-written unit tests, but would represent bugs in compiletest
Due to the current need for output capture, the new executor is still not entirely written in stable Rust

r? jieyouxu

rustbot · 2025-04-11T06:55:58Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

jieyouxu · 2025-04-11T10:11:59Z

pro gamer move

I'll look at this tmrw/Sunday.

petrochenkov · 2025-04-11T11:46:03Z

and the libtest dependency can be removed

Is there some history behind this change?
Why is it so important to remove the libtest dependency?

jieyouxu · 2025-04-11T12:32:21Z

and the libtest dependency can be removed

Is there some history behind this change? Why is it so important to remove the libtest dependency?

This is in the context of the stage 0 std redesign (#119899). Currently, compiletest depends on in-tree libtest (unless compiletest-use-stage0-libtest = true is specified). For compiler workflows, this means that compiletest will be rebuilt when iterating on compiler changes which feels quite bad with the stage 0 std redesign otherwise (see #t-infra/bootstrap > Review thread: stage 0 redesign PR @ 💬 discussions).

If we no longer depend on libtest, then we won't need this compiletest-use-stage0-libtest = true workaround/hack.

Zalathar · 2025-04-11T12:34:56Z

Depending on libtest internals is a big maintenance headache for compiletest. We already have a bunch of annoying hacks in compiletest to work around limitations that libtest imposes. And changing libtest to better meet compiletest's needs is very risky, because libtest is the backbone of testing across most of the wider Rust ecosystem.

What has made this change even more desirable is the upcoming stage 0 redesign (#119899). Currently compiletest relies on an in-tree libtest built with the stage 0 compiler. But after the redesign, stage 0 builds of the standard library will no longer be supported, so compiletest will have to rely on either a stage 1 build of libtest (resulting in unacceptable workflow regressions for compiler contributors), or the pre-built stage 0 libtest (which can go out of sync with the in-tree version, causing various bootstrapping problems).

This replacement still needs unstable library features (for now), but internal_output_capture is a much smaller unstable surface area. And in the future it should be possible to work towards removing the need for output capture in compiletest, making it 100% compatible with stable Rust. That removes a lot of the maintenance concerns associated with wanting to build it with the stage 0 compiler.

Zalathar · 2025-04-12T03:54:38Z

Tweaked some comments (diff); no functional change.

jieyouxu

Thanks for working on this! The new executor implementation here is very nice to follow, and having more control over how compiletest actually executes the tests is very much appreciated!

I left two nits, but otherwise the changes look good to me.

jieyouxu · 2025-04-14T02:50:11Z

src/tools/compiletest/src/executor.rs

+
+/// Applies command-line arguments for filtering/skipping tests by name.
+///
+/// Adapted from `filter_tests` in libtest.


Suggestion: can you leave me a note here to "reconsider test filtering behavior in new executor"? One gripe I had with libtest's test filtering was that after compiletest passes to libtest the canonicalized test name (based on a relative path), the default filter logic is a naive substring match which can often produce some surprising running more tests than expected behavior. A benefit of rolling our own executor here is to allow us to redesign the filter logic (as a possibility).

jieyouxu · 2025-04-14T02:55:25Z

src/tools/compiletest/src/executor.rs

+///
+/// Copied from `get_concurrency` in libtest.
+fn get_concurrency() -> usize {
+    if let Ok(value) = env::var("RUST_TEST_THREADS") {


Discussion: ~~Since the only intended consumer of compiletest's cli is bootstrap nowadays(rustdoc-gui-test uses compiletest programmatically), should this instead be a cli flag instead of an env var?~~ Although I guess even if it was an cli flag, we need also need to process RUST_TEST_THREADS anyway.

jieyouxu · 2025-04-14T02:57:43Z

@rustbot author

rustbot · 2025-04-14T02:57:47Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

bors · 2025-04-14T11:24:49Z

☔ The latest upstream changes (presumably #139781) made this pull request unmergeable. Please resolve the merge conflicts.

The new executor can be enabled by passing `--new-executor` or `-n` to compiletest. For example: `./x test ui -- -n`

Zalathar · 2025-04-15T03:01:52Z

Rebased to fix trivial conflicts, then added some FIXMEs to filter_tests and get_concurrency to address feedback (diff).

@rustbot ready

jieyouxu

Thanks

jieyouxu · 2025-04-15T03:44:58Z

@bors r+ rollup

bors · 2025-04-15T03:45:01Z

📌 Commit e3d6813 has been approved by jieyouxu

It is now in the queue for this repository.

Rollup of 17 pull requests Successful merges: - rust-lang#138374 (Enable contracts for const functions) - rust-lang#138380 (ci: add runners for vanilla LLVM 20) - rust-lang#138393 (Allow const patterns of matches to contain pattern types) - rust-lang#139517 (std: sys: process: uefi: Use NULL stdin by default) - rust-lang#139554 (std: add Output::exit_ok) - rust-lang#139660 (compiletest: Add an experimental new executor to replace libtest) - rust-lang#139669 (Overhaul `AssocItem`) - rust-lang#139671 (Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}) - rust-lang#139750 (std/thread: Use default stack size from menuconfig for NuttX) - rust-lang#139772 (Remove `hir::Map`) - rust-lang#139785 (Let CStrings be either 1 or 2 byte aligned.) - rust-lang#139789 (do not unnecessarily leak auto traits in item bounds) - rust-lang#139791 (drop global where-bounds before merging candidates) - rust-lang#139798 (normalize: prefer `ParamEnv` over `AliasBound` candidates) - rust-lang#139822 (Fix: Map EOPNOTSUPP to ErrorKind::Unsupported on Unix) - rust-lang#139833 (Fix some HIR pretty-printing problems) - rust-lang#139836 (Basic tests of MPMC receiver cloning) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#139660 - Zalathar:new-executor, r=jieyouxu compiletest: Add an experimental new executor to replace libtest This PR adds a new "executor" to compiletest for running the list of collected tests, to eventually replace the current dependency on unstable libtest internals. The new executor is currently inactive by default. It must be activated explicitly by passing `-n` or `--new-executor` to compiletest, e.g. `./x test ui -- -n`. (After some amount of wider manual testing, the new executor will hopefully be made the default, and the libtest dependency can be removed. Contributors should not notice any change.) The new executor is a stripped-down rewrite of the subset of libtest needed by compiletest. # Supported functionality - Specifying the number of concurrent tests with `RUST_TEST_THREADS` - Filtering and skipping tests by name (substring or exact-match) - Forcibly running ignored tests with `--ignored` - Optional fail-fast with `--fail-fast` - JSON output, compatible with bootstrap's parser for libtest output - Running each test in its own thread - Short backtraces that ignore the executor itself - Slow test detection, with a hard-coded timeout of 60 seconds - Capturing stdout/stderr, via `#![feature(internal_output_capture)]` - Suppressing output capture with `--no-capture` # Unsupported functionality - Non-JSON output, as this is handled by bootstrap instead - Separate code path for concurrency=1, as the concurrent path should handle this case naturally - Fallback to running tests synchronously if new threads can't be spawned - Special handling of hosts that don't support basic functionality like threads or timers - Our ability to test *targets* should be unaffected - Graceful handling of some edge cases that could occur in arbitrary user-written unit tests, but would represent bugs in compiletest - Due to the current need for output capture, the new executor is still not entirely written in stable Rust --- r? jieyouxu

Kobzol · 2025-04-18T08:58:20Z

src/tools/compiletest/src/executor/deadline.rs

+    fn remove_tests_past_deadline(&mut self) -> Vec<DeadlineEntry<'a>> {
+        let now = Instant::now();
+        let mut timed_out = vec![];
+        while let Some(deadline_entry) = pop_front_if(&mut self.queue, |entry| now < entry.deadline)


Shouldn't this be now >= entry.deadline?

Good catch, you're right. I'll make a fix.

Rollup of 17 pull requests Successful merges: - rust-lang#138374 (Enable contracts for const functions) - rust-lang#138380 (ci: add runners for vanilla LLVM 20) - rust-lang#138393 (Allow const patterns of matches to contain pattern types) - rust-lang#139517 (std: sys: process: uefi: Use NULL stdin by default) - rust-lang#139554 (std: add Output::exit_ok) - rust-lang#139660 (compiletest: Add an experimental new executor to replace libtest) - rust-lang#139669 (Overhaul `AssocItem`) - rust-lang#139671 (Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}) - rust-lang#139750 (std/thread: Use default stack size from menuconfig for NuttX) - rust-lang#139772 (Remove `hir::Map`) - rust-lang#139785 (Let CStrings be either 1 or 2 byte aligned.) - rust-lang#139789 (do not unnecessarily leak auto traits in item bounds) - rust-lang#139791 (drop global where-bounds before merging candidates) - rust-lang#139798 (normalize: prefer `ParamEnv` over `AliasBound` candidates) - rust-lang#139822 (Fix: Map EOPNOTSUPP to ErrorKind::Unsupported on Unix) - rust-lang#139833 (Fix some HIR pretty-printing problems) - rust-lang#139836 (Basic tests of MPMC receiver cloning) r? `@ghost` `@rustbot` modify labels: rollup

compiletest: Fix deadline bugs in new executor The experimental new executor for compiletest (rust-lang#139660) was found to have two major bugs in deadline handling for detecting slow tests: - The comparison between `now` and test deadlines was reversed, causing no timeouts to ever be recognised. - After fixing that bug, it was found that the existing code would issue timeouts for any test that had started more than 60 seconds ago, even if the test had finished long before its deadline was reached. This PR fixes those bugs. (The new executor is not yet enabled by default, so this PR has no immediate effect on contributors.) --- I noted in rust-lang#139998 (comment) that I hoped to have some unit tests to accompany these fixes. Unfortunately that turned out to be infeasible, because `DeadlineQueue` is tightly coupled to concrete `mpsc::Receiver` APIs (in addition to `Instant::now`), and trying to mock all of those would make the code much more complicated. I did, however, add a few assertions that would have caught the failure to remove tests from the queue after their deadline. r? jieyouxu

Rollup merge of rust-lang#140031 - Zalathar:deadline, r=jieyouxu compiletest: Fix deadline bugs in new executor The experimental new executor for compiletest (rust-lang#139660) was found to have two major bugs in deadline handling for detecting slow tests: - The comparison between `now` and test deadlines was reversed, causing no timeouts to ever be recognised. - After fixing that bug, it was found that the existing code would issue timeouts for any test that had started more than 60 seconds ago, even if the test had finished long before its deadline was reached. This PR fixes those bugs. (The new executor is not yet enabled by default, so this PR has no immediate effect on contributors.) --- I noted in rust-lang#139998 (comment) that I hoped to have some unit tests to accompany these fixes. Unfortunately that turned out to be infeasible, because `DeadlineQueue` is tightly coupled to concrete `mpsc::Receiver` APIs (in addition to `Instant::now`), and trying to mock all of those would make the code much more complicated. I did, however, add a few assertions that would have caught the failure to remove tests from the queue after their deadline. r? jieyouxu

compiletest: Use the new non-libtest executor by default The new executor was implemented in rust-lang#139660, but required a manual opt-in. This PR activates the new executor by default, but leaves the old libtest-based executor in place (temporarily) to make reverting easier if something unexpectedly goes horribly wrong. Currently the new executor can be explicitly disabled by passing the `-N` flag to compiletest (e.g. `./x test ui -- -N`), but eventually that flag will be removed, alongside the removal of the libtest dependency. The flag is mostly there to make manual comparative testing easier if something does go wrong. As before, there *should* be no user-visible difference between the old executor and the new executor. --- I didn't get much of a response to my [call for testing thread on Zulip](https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/Call.20for.20testing.3A.20New.20test.20executor.20for.20compiletest/with/512452105), and the reports I did get (along with my own usage) indicate that there aren't any problems. So I think it's reasonable to move forward with making this the default, in the hopes of being able to remove the libtest dependency relatively soon. When the libtest dependency is removed, it should be reasonable to build compiletest against pre-built stage0 std by default, even after the stage0 redesign. (Though we should probably have at least one CI job using in-tree stage1 std instead, to guard against the possibility of the `#![feature(internal_output_capture)]` API actually changing.)

Zalathar added the A-compiletest Area: The compiletest test runner label Apr 11, 2025

rustbot assigned jieyouxu Apr 11, 2025

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Apr 11, 2025

This comment has been minimized.

Sign in to view

Zalathar force-pushed the new-executor branch 2 times, most recently from 62d2626 to 3f2fa86 Compare April 11, 2025 08:28

Zalathar changed the title ~~compiletest: Add an experimental "new" executor to replace libtest~~ compiletest: Add an experimental new executor to replace libtest Apr 11, 2025

Zalathar force-pushed the new-executor branch from 3f2fa86 to 7ebb8a6 Compare April 12, 2025 03:54

jieyouxu approved these changes Apr 14, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 14, 2025

Zalathar added 2 commits April 15, 2025 12:36

Use a constant for unstable features needed by compiletest

4c23295

compiletest: Extract libtest-specific executor code to a submodule

6fda3e5

Zalathar force-pushed the new-executor branch from 7ebb8a6 to 0730ef2 Compare April 15, 2025 02:45

compiletest: Add an experimental new executor to replace libtest

e3d6813

The new executor can be enabled by passing `--new-executor` or `-n` to compiletest. For example: `./x test ui -- -n`

Zalathar force-pushed the new-executor branch from 0730ef2 to e3d6813 Compare April 15, 2025 03:00

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 15, 2025

jieyouxu approved these changes Apr 15, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 15, 2025

Zalathar mentioned this pull request Apr 15, 2025

Rollup of 17 pull requests #139845

Merged

bors merged commit efcf4f9 into rust-lang:master Apr 15, 2025
6 checks passed

rustbot added this to the 1.88.0 milestone Apr 15, 2025

Zalathar deleted the new-executor branch April 15, 2025 11:14

Zalathar mentioned this pull request Apr 18, 2025

compiletest: Use the new non-libtest executor by default #139998

Merged

Kobzol reviewed Apr 18, 2025

View reviewed changes

Zalathar mentioned this pull request Apr 19, 2025

compiletest: Fix deadline bugs in new executor #140031

Merged

Zalathar mentioned this pull request Apr 22, 2025

Add libtest JSON emitter oli-obk/ui_test#315

Open

jieyouxu mentioned this pull request Apr 23, 2025

Tracking issue for new compiletest executor #140192

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

compiletest: Add an experimental new executor to replace libtest #139660

compiletest: Add an experimental new executor to replace libtest #139660

Zalathar commented Apr 11, 2025 •

edited

Loading

rustbot commented Apr 11, 2025

This comment has been minimized.

jieyouxu commented Apr 11, 2025

petrochenkov commented Apr 11, 2025 •

edited

Loading

jieyouxu commented Apr 11, 2025 •

edited

Loading

Zalathar commented Apr 11, 2025 •

edited

Loading

Zalathar commented Apr 12, 2025

jieyouxu left a comment •

edited

Loading

jieyouxu Apr 14, 2025

jieyouxu Apr 14, 2025

jieyouxu commented Apr 14, 2025

rustbot commented Apr 14, 2025

bors commented Apr 14, 2025

Zalathar commented Apr 15, 2025

jieyouxu left a comment

jieyouxu commented Apr 15, 2025

bors commented Apr 15, 2025

Kobzol Apr 18, 2025 •

edited

Loading

Zalathar Apr 18, 2025

compiletest: Add an experimental new executor to replace libtest #139660

compiletest: Add an experimental new executor to replace libtest #139660

Conversation

Zalathar commented Apr 11, 2025 • edited Loading

Supported functionality

Unsupported functionality

rustbot commented Apr 11, 2025

This comment has been minimized.

jieyouxu commented Apr 11, 2025

petrochenkov commented Apr 11, 2025 • edited Loading

jieyouxu commented Apr 11, 2025 • edited Loading

Zalathar commented Apr 11, 2025 • edited Loading

Zalathar commented Apr 12, 2025

jieyouxu left a comment • edited Loading

Choose a reason for hiding this comment

jieyouxu Apr 14, 2025

Choose a reason for hiding this comment

jieyouxu Apr 14, 2025

Choose a reason for hiding this comment

jieyouxu commented Apr 14, 2025

rustbot commented Apr 14, 2025

bors commented Apr 14, 2025

Zalathar commented Apr 15, 2025

jieyouxu left a comment

Choose a reason for hiding this comment

jieyouxu commented Apr 15, 2025

bors commented Apr 15, 2025

Kobzol Apr 18, 2025 • edited Loading

Choose a reason for hiding this comment

Zalathar Apr 18, 2025

Choose a reason for hiding this comment

Zalathar commented Apr 11, 2025 •

edited

Loading

petrochenkov commented Apr 11, 2025 •

edited

Loading

jieyouxu commented Apr 11, 2025 •

edited

Loading

Zalathar commented Apr 11, 2025 •

edited

Loading

jieyouxu left a comment •

edited

Loading

Kobzol Apr 18, 2025 •

edited

Loading