Skip to content

Commit d0fc6d2

Browse files
EnselicEnselic
committed
tests: Require run-fail ui tests to have an exit code (SIGABRT not ok)
Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-crash` for those tests. And some tests behave differently on different targets. Introduce and use `run-fail-or-crash` for those tests. Some sanitizer tests use this for example, because on some platforms sanitizers will exit with code 1 in the face of memory errors, while on others they exit with abort in the face of memory errors.
1 parent 35f6036 commit d0fc6d2

File tree

74 files changed

+195
-98
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

74 files changed

+195
-98
lines changed

‎src/doc/rustc-dev-guide/src/tests/directives.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,10 @@ expectations](ui.md#controlling-passfail-expectations).
7575
| `check-fail` | Building (no codegen) should fail | `ui`, `crashes` | N/A |
7676
| `build-pass` | Building should pass | `ui`, `crashes`, `codegen`, `incremental` | N/A |
7777
| `build-fail` | Building should fail | `ui`, `crashes` | N/A |
78-
| `run-pass` | Running the test binary should pass | `ui`, `crashes`, `incremental` | N/A |
79-
| `run-fail` | Running the test binary should fail | `ui`, `crashes` | N/A |
78+
| `run-pass` | Program must exit with code `0` | `ui`, `crashes`, `incremental` | N/A |
79+
| `run-fail` | Program must exit with code `1..=127` | `ui`, `crashes` | N/A |
80+
| `run-crash` | Program must crash | `ui` | N/A |
81+
| `run-fail-or-crash` | Program must `run-fail` or `run-crash` | `ui` | N/A |
8082
| `ignore-pass` | Ignore `--pass` flag | `ui`, `crashes`, `codegen`, `incremental` | N/A |
8183
| `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental` | N/A |
8284
| `failure-status` | Check | `ui`, `crashes` | Any `u16` |

‎src/doc/rustc-dev-guide/src/tests/ui.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -448,7 +448,7 @@ even run the resulting program. Just add one of the following
448448
- `//@ build-pass` — compilation and linking should succeed but do
449449
not run the resulting binary.
450450
- `//@ run-pass` — compilation should succeed and running the resulting
451-
binary should also succeed.
451+
binary should make it exit with code 0 which indicates success.
452452
- Fail directives:
453453
- `//@ check-fail` — compilation should fail (the codegen phase is skipped).
454454
This is the default for UI tests.
@@ -457,10 +457,16 @@ even run the resulting program. Just add one of the following
457457
without the codegen phase, then a second time the full compile should
458458
fail.
459459
- `//@ run-fail` — compilation should succeed, but running the resulting
460-
binary should fail.
461-
462-
For `run-pass` and `run-fail` tests, by default the output of the program itself
463-
is not checked.
460+
binary should make it exit with a code in the range `1..=127` which
461+
indicates regular failure.
462+
- `//@ run-crash` — compilation should succeed, but running the resulting
463+
binary should fail with a crash (e.g. `SIGABRT` or `SIGSEGV`).
464+
- `//@ run-crash-or-crash` — compilation should succeed, but running the
465+
resulting binary should either fail with a code in the range `1..=127` or
466+
crash (e.g. `SIGABRT` or `SIGSEGV`).
467+
468+
For `run-pass`. `run-fail`, `run-crash` and `run-fail-or-crash` tests, by
469+
default the output of the program itself is not checked.
464470

465471
If you want to check the output of running the program, include the
466472
`check-run-results` directive. This will check for a `.run.stderr` and

‎src/tools/compiletest/src/common.rs

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,11 +110,37 @@ string_enum! {
110110
}
111111
}
112112

113+
string_enum! {
114+
#[derive(Clone, Copy, PartialEq, Debug, Hash)]
115+
pub enum RunResult {
116+
Pass => "run-pass",
117+
Fail => "run-fail",
118+
Crash => "run-crash",
119+
}
120+
}
121+
122+
#[derive(Copy, Clone, Debug, PartialEq, PartialOrd)]
123+
pub enum RunFailMode {
124+
/// Running the program must make it exit with a regular failure exit code
125+
/// in the range `1..=127`. If the program is terminated by e.g. a signal
126+
/// the test will fail.
127+
Fail,
128+
/// Running the program must result in a crash, e.g. by `SIGABRT` or
129+
/// `SIGSEGV` on Unix or on Windows by having an appropriate NTSTATUS high
130+
/// bit in the exit code.
131+
Crash,
132+
/// Running the program must either fail or crash. Useful for e.g. sanitizer
133+
/// tests since some sanitizer implementations exit the process with code 1
134+
/// to in the face of memory errors while others abort (crash) the process
135+
/// in the face of memory errors.
136+
FailOrCrash,
137+
}
138+
113139
#[derive(Copy, Clone, Debug, PartialEq, PartialOrd)]
114140
pub enum FailMode {
115141
Check,
116142
Build,
117-
Run,
143+
Run(RunFailMode),
118144
}
119145

120146
string_enum! {

‎src/tools/compiletest/src/directive-list.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,7 +241,9 @@ const KNOWN_DIRECTIVE_NAMES: &[&str] = &[
241241
"regex-error-pattern",
242242
"remap-src-base",
243243
"revisions",
244+
"run-crash",
244245
"run-fail",
246+
"run-fail-or-crash",
245247
"run-flags",
246248
"run-pass",
247249
"run-rustfix",

‎src/tools/compiletest/src/header.rs

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ use camino::{Utf8Path, Utf8PathBuf};
99
use semver::Version;
1010
use tracing::*;
1111

12-
use crate::common::{Config, Debugger, FailMode, Mode, PassMode};
12+
use crate::common::{Config, Debugger, FailMode, Mode, PassMode, RunFailMode};
1313
use crate::debuggers::{extract_cdb_version, extract_gdb_version};
1414
use crate::errors::ErrorKind;
1515
use crate::executor::{CollectedTestDesc, ShouldPanic};
@@ -655,7 +655,13 @@ impl TestProps {
655655
Some(FailMode::Build)
656656
} else if config.parse_name_directive(ln, "run-fail") {
657657
check_ui("run");
658-
Some(FailMode::Run)
658+
Some(FailMode::Run(RunFailMode::Fail))
659+
} else if config.parse_name_directive(ln, "run-crash") {
660+
check_ui("run");
661+
Some(FailMode::Run(RunFailMode::Crash))
662+
} else if config.parse_name_directive(ln, "run-fail-or-crash") {
663+
check_ui("run");
664+
Some(FailMode::Run(RunFailMode::FailOrCrash))
659665
} else {
660666
None
661667
};

‎src/tools/compiletest/src/runtest.rs

Lines changed: 23 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,10 @@ use tracing::*;
1717

1818
use crate::common::{
1919
Assembly, Codegen, CodegenUnits, CompareMode, Config, CoverageMap, CoverageRun, Crashes,
20-
DebugInfo, Debugger, FailMode, Incremental, MirOpt, PassMode, Pretty, RunMake, Rustdoc,
21-
RustdocJs, RustdocJson, TestPaths, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR, UI_RUN_STDOUT,
22-
UI_STDERR, UI_STDOUT, UI_SVG, UI_WINDOWS_SVG, Ui, expected_output_path, incremental_dir,
23-
output_base_dir, output_base_name, output_testname_unique,
20+
DebugInfo, Debugger, FailMode, Incremental, MirOpt, PassMode, Pretty, RunFailMode, RunMake,
21+
RunResult, Rustdoc, RustdocJs, RustdocJson, TestPaths, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR,
22+
UI_RUN_STDOUT, UI_STDERR, UI_STDOUT, UI_SVG, UI_WINDOWS_SVG, Ui, expected_output_path,
23+
incremental_dir, output_base_dir, output_base_name, output_testname_unique,
2424
};
2525
use crate::compute_diff::{DiffLine, make_diff, write_diff, write_filtered_diff};
2626
use crate::errors::{Error, ErrorKind, load_errors};
@@ -277,7 +277,11 @@ impl<'test> TestCx<'test> {
277277

278278
fn should_run(&self, pm: Option<PassMode>) -> WillExecute {
279279
let test_should_run = match self.config.mode {
280-
Ui if pm == Some(PassMode::Run) || self.props.fail_mode == Some(FailMode::Run) => true,
280+
Ui if pm == Some(PassMode::Run)
281+
|| matches!(self.props.fail_mode, Some(FailMode::Run(_))) =>
282+
{
283+
true
284+
}
281285
MirOpt if pm == Some(PassMode::Run) => true,
282286
Ui | MirOpt => false,
283287
mode => panic!("unimplemented for mode {:?}", mode),
@@ -2959,6 +2963,20 @@ impl ProcRes {
29592963
// compiletest, which is unnecessary noise.
29602964
std::panic::resume_unwind(Box::new(()));
29612965
}
2966+
2967+
pub fn run_result_and_code(&self) -> (RunResult, Option<i32>) {
2968+
let code = self.status.code();
2969+
(
2970+
if self.status.success() {
2971+
RunResult::Pass
2972+
} else if code.is_some_and(|c| c >= 1 && c <= 127) {
2973+
RunResult::Fail
2974+
} else {
2975+
RunResult::Crash
2976+
},
2977+
code,
2978+
)
2979+
}
29622980
}
29632981

29642982
#[derive(Debug)]

‎src/tools/compiletest/src/runtest/ui.rs

Lines changed: 43 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ use rustfix::{Filter, apply_suggestions, get_suggestions_from_json};
66
use tracing::debug;
77

88
use super::{
9-
AllowUnused, Emit, FailMode, LinkToAux, PassMode, TargetLocation, TestCx, TestOutput,
10-
Truncated, UI_FIXED, WillExecute,
9+
AllowUnused, Emit, FailMode, LinkToAux, PassMode, RunFailMode, RunResult, TargetLocation,
10+
TestCx, TestOutput, Truncated, UI_FIXED, WillExecute,
1111
};
1212
use crate::json;
1313

@@ -140,12 +140,49 @@ impl TestCx<'_> {
140140
&proc_res,
141141
);
142142
}
143+
let (run_result, code) = proc_res.run_result_and_code();
144+
// Help users understand why the test failed by including the actual
145+
// exit code and actual run result in the failure message.
146+
let pass_hint = format!("code={code:?} (test would pass as `{run_result}`)");
143147
if self.should_run_successfully(pm) {
144-
if !proc_res.status.success() {
145-
self.fatal_proc_rec("test run failed!", &proc_res);
148+
if run_result != RunResult::Pass {
149+
self.fatal_proc_rec(
150+
&format!("test did not exit with success! {pass_hint}"),
151+
&proc_res,
152+
);
146153
}
147-
} else if proc_res.status.success() {
148-
self.fatal_proc_rec("test run succeeded!", &proc_res);
154+
} else if self.props.fail_mode == Some(FailMode::Run(RunFailMode::Fail)) {
155+
// If the test is marked as `run-fail` but do not support
156+
// unwinding we allow it to crash, since a panic will trigger an
157+
// abort (crash) instead of unwind (exit with code 101).
158+
let crash_ok = !self.config.can_unwind();
159+
if run_result != RunResult::Fail && !(crash_ok && run_result == RunResult::Crash) {
160+
let extra_msg = if crash_ok {
161+
&format!(
162+
" or crash (crash allowed since `{}` does not support unwinding)",
163+
self.config.target
164+
)
165+
} else {
166+
""
167+
};
168+
self.fatal_proc_rec(
169+
&format!("test did not exit with failure{extra_msg}! {pass_hint}",),
170+
&proc_res,
171+
);
172+
}
173+
} else if self.props.fail_mode == Some(FailMode::Run(RunFailMode::Crash)) {
174+
if run_result != RunResult::Crash {
175+
self.fatal_proc_rec(&format!("test did not crash! {pass_hint}"), &proc_res);
176+
}
177+
} else if self.props.fail_mode == Some(FailMode::Run(RunFailMode::FailOrCrash)) {
178+
if run_result != RunResult::Fail && run_result != RunResult::Crash {
179+
self.fatal_proc_rec(
180+
&format!("test did not exit with failure or crash! {pass_hint}"),
181+
&proc_res,
182+
);
183+
}
184+
} else {
185+
unreachable!("run_ui_test() must not be called if the test should not run");
149186
}
150187

151188
self.get_output(&proc_res)

‎tests/ui/contracts/contract-attributes-generics.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@
55
//@ [unchk_pass] run-pass
66
//@ [chk_pass] run-pass
77
//
8-
//@ [chk_fail_pre] run-fail
9-
//@ [chk_fail_post] run-fail
10-
//@ [chk_const_fail] run-fail
8+
//@ [chk_fail_pre] run-crash
9+
//@ [chk_fail_post] run-crash
10+
//@ [chk_const_fail] run-crash
1111
//
1212
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1313
//

‎tests/ui/contracts/contract-attributes-nest.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
//@ [unchk_fail_post] run-pass
66
//@ [chk_pass] run-pass
77
//
8-
//@ [chk_fail_pre] run-fail
9-
//@ [chk_fail_post] run-fail
8+
//@ [chk_fail_pre] run-crash
9+
//@ [chk_fail_post] run-crash
1010
//
1111
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1212
//@ [unchk_fail_pre] compile-flags: -Zcontract-checks=no

‎tests/ui/contracts/contract-attributes-tail.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
//@ [unchk_fail_post] run-pass
66
//@ [chk_pass] run-pass
77
//
8-
//@ [chk_fail_pre] run-fail
9-
//@ [chk_fail_post] run-fail
8+
//@ [chk_fail_pre] run-crash
9+
//@ [chk_fail_post] run-crash
1010
//
1111
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1212
//@ [unchk_fail_pre] compile-flags: -Zcontract-checks=no

‎tests/ui/contracts/contract-captures-via-closure-copy.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ compile-flags: -Zcontract-checks=yes
33

44
#![feature(contracts)]

‎tests/ui/contracts/contract-const-fn.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@
88
//
99
//@ [all_pass] run-pass
1010
//
11-
//@ [runtime_fail_pre] run-fail
12-
//@ [runtime_fail_post] run-fail
11+
//@ [runtime_fail_pre] run-crash
12+
//@ [runtime_fail_post] run-crash
1313
//
1414
//@ [all_pass] compile-flags: -Zcontract-checks=yes
1515
//@ [runtime_fail_pre] compile-flags: -Zcontract-checks=yes

‎tests/ui/contracts/contracts-ensures-early-fn-exit.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
//
33
//@ [unchk_pass] run-pass
44
//@ [chk_pass] run-pass
5-
//@ [chk_fail_try] run-fail
6-
//@ [chk_fail_ret] run-fail
7-
//@ [chk_fail_yeet] run-fail
5+
//@ [chk_fail_try] run-crash
6+
//@ [chk_fail_ret] run-crash
7+
//@ [chk_fail_yeet] run-crash
88
//
99
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1010
//@ [chk_pass] compile-flags: -Zcontract-checks=yes

‎tests/ui/contracts/internal_machinery/contract-ast-extensions-nest.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
//@ [unchk_fail_post] run-pass
66
//@ [chk_pass] run-pass
77
//
8-
//@ [chk_fail_pre] run-fail
9-
//@ [chk_fail_post] run-fail
8+
//@ [chk_fail_pre] run-crash
9+
//@ [chk_fail_post] run-crash
1010
//
1111
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1212
//@ [unchk_fail_pre] compile-flags: -Zcontract-checks=no

‎tests/ui/contracts/internal_machinery/contract-ast-extensions-tail.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
//@ [unchk_fail_post] run-pass
66
//@ [chk_pass] run-pass
77
//
8-
//@ [chk_fail_pre] run-fail
9-
//@ [chk_fail_post] run-fail
8+
//@ [chk_fail_pre] run-crash
9+
//@ [chk_fail_post] run-crash
1010
//
1111
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1212
//@ [unchk_fail_pre] compile-flags: -Zcontract-checks=no

‎tests/ui/contracts/internal_machinery/contract-intrinsics.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
//@ [default] run-pass
44
//@ [unchk_pass] run-pass
55
//@ [chk_pass] run-pass
6-
//@ [chk_fail_requires] run-fail
7-
//@ [chk_fail_ensures] run-fail
6+
//@ [chk_fail_requires] run-crash
7+
//@ [chk_fail_ensures] run-crash
88
//
99
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1010
//@ [chk_pass] compile-flags: -Zcontract-checks=yes

‎tests/ui/contracts/internal_machinery/contract-lang-items.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
//@ [unchk_fail_post] run-pass
55
//@ [chk_pass] run-pass
66
//
7-
//@ [chk_fail_post] run-fail
7+
//@ [chk_fail_post] run-crash
88
//
99
//@ [unchk_pass] compile-flags: -Zcontract-checks=no
1010
//@ [unchk_fail_post] compile-flags: -Zcontract-checks=no

‎tests/ui/extern/extern-types-field-offset.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ check-run-results
33
//@ exec-env:RUST_BACKTRACE=0
44
//@ normalize-stderr: "(core/src/panicking\.rs):[0-9]+:[0-9]+" -> "$1:$$LINE:$$COL"

‎tests/ui/mir/alignment/borrow_misaligned_field_projection.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ ignore-i686-pc-windows-msvc: #112480
33
//@ compile-flags: -C debug-assertions
44
//@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is

‎tests/ui/mir/alignment/misaligned_borrow.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ ignore-i686-pc-windows-msvc: #112480
33
//@ compile-flags: -C debug-assertions
44
//@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is

‎tests/ui/mir/alignment/misaligned_lhs.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ ignore-i686-pc-windows-msvc: #112480
33
//@ compile-flags: -C debug-assertions
44
//@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is

‎tests/ui/mir/alignment/misaligned_mut_borrow.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ ignore-i686-pc-windows-msvc: #112480
33
//@ compile-flags: -C debug-assertions
44
//@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is

‎tests/ui/mir/alignment/misaligned_rhs.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ ignore-i686-pc-windows-msvc: #112480
33
//@ compile-flags: -C debug-assertions
44
//@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is

‎tests/ui/mir/alignment/two_pointers.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ ignore-i686-pc-windows-msvc: #112480
33
//@ compile-flags: -C debug-assertions
44
//@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is

‎tests/ui/mir/enum/convert_non_enum_break.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ compile-flags: -C debug-assertions
33
//@ error-pattern: trying to construct an enum from an invalid value 0x10000
44

‎tests/ui/mir/enum/convert_non_enum_niche_break.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ compile-flags: -C debug-assertions
33
//@ error-pattern: trying to construct an enum from an invalid value 0x5
44

‎tests/ui/mir/enum/niche_option_tuple_break.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//@ run-fail
1+
//@ run-crash
22
//@ compile-flags: -C debug-assertions
33
//@ error-pattern: trying to construct an enum from an invalid value 0x3
44

0 commit comments

Comments
 (0)