Skip to content

Commit 1a5df32

Browse files
committed
[DNM] Add documentation for hypothetical pre-timeout scripts
In the spirit of documentation driven development, this commit adds docs for the pre-timeout scripts proposed in #1906. I'll start prototyping an implementation soon. In the meantime, I figured we could start shaking out the major design questions by reviewing what's proposed in the documentation here.
1 parent eaa5634 commit 1a5df32

File tree

1 file changed

+54
-14
lines changed

1 file changed

+54
-14
lines changed

site/src/docs/configuration/setup-scripts.md renamed to site/src/docs/configuration/scripts.md

Lines changed: 54 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,25 +3,34 @@ icon: material/ray-start-arrow
33
status: experimental
44
---
55

6-
# Setup scripts
6+
# Scripts
77

88
<!-- md:version 0.9.59 -->
99

10-
!!! experimental "Experimental: This feature is not yet stable"
10+
!!! experimental "Experimental: Setup scripts are not yet stable"
1111

1212
- **Enable with:** Add `experimental = ["setup-scripts"]` to `.config/nextest.toml`
1313
- **Tracking issue:** [#978](https://github.com/nextest-rs/nextest/issues/978)
1414

15-
Nextest supports running _setup scripts_ before tests are run. Setup scripts can be scoped to
16-
particular tests via [filtersets](../filtersets/index.md).
15+
!!! experimental "Experimental: Pre-timeout scripts are not yet stable"
1716

18-
Setup scripts are configured in two parts: _defining scripts_, and _setting up rules_ for when they should be executed.
17+
- **Enable with:** Add `experimental = ["pre-timeout-scripts"]` to `.config/nextest.toml`
18+
- **Tracking issue:** [#978](https://github.com/nextest-rs/nextest/issues/978)
19+
20+
Nextest supports running _scripts_ when certain events occur during a test run. Scripts can be scoped to particular tests via [filtersets](../filtersets/index.md).
21+
22+
Nextest currently recognizes two types of scripts:
23+
24+
* _Setup scripts_, which are executed at the start of a test run.
25+
* _Pre-timeout scripts_, which are executed before nextest begins terminating a test that has exceeded its timeout.
26+
27+
Scripts are configured in two parts: _defining scripts_, and _setting up rules_ for when they should be executed.
1928

2029
## Defining scripts
2130

22-
Setup scripts are defined using the top-level `script` configuration. For example, to define a script named "my-script", which runs `my-script.sh`:
31+
Scripts are defined using the top-level `script` configuration. For example, to define a script named "my-script", which runs `my-script.sh`:
2332

24-
```toml title="Setup script definition in <code>.config/nextest.toml</code>"
33+
```toml title="Script definition in <code>.config/nextest.toml</code>"
2534
[script.my-script]
2635
command = 'my-script.sh'
2736
```
@@ -36,16 +45,16 @@ command = 'script.sh -c "Hello, world!"'
3645
command = ['script.sh', '-c', 'Hello, world!']
3746
```
3847

39-
Setup scripts can have the following configuration options attached to them:
48+
Scripts can have the following configuration options attached to them:
4049

41-
- **`slow-timeout`**: Mark a setup script [as slow](../features/slow-tests.md) or [terminate it](../features/slow-tests.md#terminating-tests-after-a-timeout), using the same configuration as for tests. By default, setup scripts are not marked as slow or terminated (this is different from the slow timeout for tests).
42-
- **`leak-timeout`**: Mark setup scripts [leaky](../features/leaky-tests.md) after a timeout, using the same configuration as for tests. By default, the leak timeout is 100ms.
50+
- **`slow-timeout`**: Mark a script [as slow](../features/slow-tests.md) or [terminate it](../features/slow-tests.md#terminating-tests-after-a-timeout), using the same configuration as for tests. By default, scripts are not marked as slow or terminated (this is different from the slow timeout for tests).
51+
- **`leak-timeout`**: Mark scripts [leaky](../features/leaky-tests.md) after a timeout, using the same configuration as for tests. By default, the leak timeout is 100ms.
4352
- **`capture-stdout`**: `true` if the script's standard output should be captured, `false` if not. By default, this is `false`.
4453
- **`capture-stderr`**: `true` if the script's standard error should be captured, `false` if not. By default, this is `false`.
4554

4655
### Example
4756

48-
```toml title="Advanced setup script definition"
57+
```toml title="Advanced script definition"
4958
[script.db-generate]
5059
command = 'cargo run -p db-generate'
5160
slow-timeout = { period = "60s", terminate-after = 2 }
@@ -56,7 +65,7 @@ capture-stderr = false
5665

5766
## Setting up rules
5867

59-
In configuration, you can create rules for when to use scripts on a per-profile basis. This is done via the `profile.<profile-name>.scripts` array. For example, you can set up a script that generates a database if tests from the `db-tests` package, or any packages that depend on it, are run.
68+
In configuration, you can create rules for when to use scripts on a per-profile basis. This is done via the `profile.<profile-name>.scripts` array. For example, you can configure a setup script that generates a database if tests from the `db-tests` package, or any packages that depend on it, are run.
6069

6170
```toml title="Basic rules"
6271
[[profile.default.scripts]]
@@ -66,7 +75,7 @@ setup = 'db-generate'
6675

6776
(This example uses the `rdeps` [filterset](../filtersets/index.md) predicate.)
6877

69-
Setup scripts can also filter based on platform, using the rules listed in [_Specifying platforms_](../configuration/specifying-platforms.md):
78+
Scripts can also filter based on platform, using the rules listed in [_Specifying platforms_](../configuration/specifying-platforms.md):
7079

7180
```toml title="Platform-specific rules"
7281
[[profile.default.scripts]]
@@ -82,13 +91,32 @@ filter = 'test(/^script_tests::/)'
8291
setup = ['script1', 'script2']
8392
```
8493

94+
Executing pre-timeout scripts follows the same pattern. For example, you can configure a pre-timeout script for every test that contains `slow` in its name.
95+
96+
```toml title="Basic pre-timeout rules"
97+
[[profile.default.scripts]]
98+
filter = 'test(slow)'
99+
pre-timeout = 'capture-backtrace'
100+
```
101+
102+
A single rule can specify any number of setup scripts and any number of pre-timeout scripts.
103+
104+
```toml title="Combination rules"
105+
[[profile.default.scripts]]
106+
filter = 'test(slow)'
107+
setup = ['setup-1', 'setup-2']
108+
pre-timeout = ['pre-timeout-1', 'pre-timeout-2']
109+
```
110+
85111
## Script execution
86112

113+
### Setup scripts
114+
87115
A given setup script _S_ is only executed if the current profile has at least one rule where the `filter` and `platform` predicates match the current execution environment, and the setup script _S_ is listed in `setup`.
88116

89117
Setup scripts are executed serially, in the order they are defined (_not_ the order they're specified in the rules). If any setup script exits with a non-zero exit code, the entire test run is terminated.
90118

91-
### Environment variables
119+
#### Environment variables
92120

93121
Setup scripts can define environment variables that will be exposed to tests that match the script. This is done by writing to the `$NEXTEST_ENV` environment variable from within the script.
94122

@@ -126,6 +154,18 @@ fn my_env_test() {
126154
}
127155
```
128156

157+
### Pre-timeout scripts
158+
159+
A given pre-timeout script _S_ is executed when the current profile has at least one rule where the `platform` predicates match the current execution environment, the script _S_ is listed in `pre-timeout`, and a test matching the `filter` has reached its configured timeout.
160+
161+
Pre-timeout scripts are executed serially, in the order they are defined (_not_ the order they're specified in the rules). If any pre-timeout script exits with a non-zero exit code, an error is logged but the test run continues.
162+
163+
Nextest sets the following environment variables when executing a pre-timeout script:
164+
165+
* **`NEXTEST_PRE_TIMEOUT_TEST_PID`**: the ID of the process running the test.
166+
* **`NEXTEST_PRE_TIMEOUT_TEST_NAME`**: the name of the running test.
167+
* **`NEXTEST_PRE_TIMEOUT_TEST_BINARY`**: the name of the binary running the test.
168+
129169
## Setup scripts in JUnit output
130170

131171
<!-- md:version 0.9.86 -->

0 commit comments

Comments
 (0)