You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[DNM] Add documentation for hypothetical pre-timeout scripts
In the spirit of documentation driven development, this commit adds docs
for the pre-timeout scripts proposed in #1906.
I'll start prototyping an implementation soon. In the meantime, I
figured we could start shaking out the major design questions by
reviewing what's proposed in the documentation here.
Nextest supports running _scripts_ when certain events occur during a test run. Scripts can be scoped to particular tests via [filtersets](../filtersets/index.md).
21
+
22
+
Nextest currently recognizes two types of scripts:
23
+
24
+
*_Setup scripts_, which are executed at the start of a test run.
25
+
*_Pre-timeout scripts_, which are executed before nextest begins terminating a test that has exceeded its timeout.
26
+
27
+
Scripts are configured in two parts: _defining scripts_, and _setting up rules_ for when they should be executed.
19
28
20
29
## Defining scripts
21
30
22
-
Setup scripts are defined using the top-level `script` configuration. For example, to define a script named "my-script", which runs `my-script.sh`:
31
+
Scripts are defined using the top-level `script` configuration. For example, to define a script named "my-script", which runs `my-script.sh`:
23
32
24
-
```toml title="Setup script definition in <code>.config/nextest.toml</code>"
33
+
```toml title="Script definition in <code>.config/nextest.toml</code>"
Setup scripts can have the following configuration options attached to them:
48
+
Scripts can have the following configuration options attached to them:
40
49
41
-
-**`slow-timeout`**: Mark a setup script [as slow](../features/slow-tests.md) or [terminate it](../features/slow-tests.md#terminating-tests-after-a-timeout), using the same configuration as for tests. By default, setup scripts are not marked as slow or terminated (this is different from the slow timeout for tests).
42
-
-**`leak-timeout`**: Mark setup scripts [leaky](../features/leaky-tests.md) after a timeout, using the same configuration as for tests. By default, the leak timeout is 100ms.
50
+
-**`slow-timeout`**: Mark a script [as slow](../features/slow-tests.md) or [terminate it](../features/slow-tests.md#terminating-tests-after-a-timeout), using the same configuration as for tests. By default, scripts are not marked as slow or terminated (this is different from the slow timeout for tests).
51
+
-**`leak-timeout`**: Mark scripts [leaky](../features/leaky-tests.md) after a timeout, using the same configuration as for tests. By default, the leak timeout is 100ms.
43
52
-**`capture-stdout`**: `true` if the script's standard output should be captured, `false` if not. By default, this is `false`.
44
53
-**`capture-stderr`**: `true` if the script's standard error should be captured, `false` if not. By default, this is `false`.
45
54
46
55
### Example
47
56
48
-
```toml title="Advanced setup script definition"
57
+
```toml title="Advanced script definition"
49
58
[script.db-generate]
50
59
command = 'cargo run -p db-generate'
51
60
slow-timeout = { period = "60s", terminate-after = 2 }
@@ -56,7 +65,7 @@ capture-stderr = false
56
65
57
66
## Setting up rules
58
67
59
-
In configuration, you can create rules for when to use scripts on a per-profile basis. This is done via the `profile.<profile-name>.scripts` array. For example, you can set up a script that generates a database if tests from the `db-tests` package, or any packages that depend on it, are run.
68
+
In configuration, you can create rules for when to use scripts on a per-profile basis. This is done via the `profile.<profile-name>.scripts` array. For example, you can configure a setup script that generates a database if tests from the `db-tests` package, or any packages that depend on it, are run.
60
69
61
70
```toml title="Basic rules"
62
71
[[profile.default.scripts]]
@@ -66,7 +75,7 @@ setup = 'db-generate'
66
75
67
76
(This example uses the `rdeps`[filterset](../filtersets/index.md) predicate.)
68
77
69
-
Setup scripts can also filter based on platform, using the rules listed in [_Specifying platforms_](../configuration/specifying-platforms.md):
78
+
Scripts can also filter based on platform, using the rules listed in [_Specifying platforms_](../configuration/specifying-platforms.md):
Executing pre-timeout scripts follows the same pattern. For example, you can configure a pre-timeout script for every test that contains `slow` in its name.
95
+
96
+
```toml title="Basic pre-timeout rules"
97
+
[[profile.default.scripts]]
98
+
filter = 'test(slow)'
99
+
pre-timeout = 'capture-backtrace'
100
+
```
101
+
102
+
A single rule can specify any number of setup scripts and any number of pre-timeout scripts.
103
+
104
+
```toml title="Combination rules"
105
+
[[profile.default.scripts]]
106
+
filter = 'test(slow)'
107
+
setup = ['setup-1', 'setup-2']
108
+
pre-timeout = ['pre-timeout-1', 'pre-timeout-2']
109
+
```
110
+
85
111
## Script execution
86
112
113
+
### Setup scripts
114
+
87
115
A given setup script _S_ is only executed if the current profile has at least one rule where the `filter` and `platform` predicates match the current execution environment, and the setup script _S_ is listed in `setup`.
88
116
89
117
Setup scripts are executed serially, in the order they are defined (_not_ the order they're specified in the rules). If any setup script exits with a non-zero exit code, the entire test run is terminated.
90
118
91
-
### Environment variables
119
+
####Environment variables
92
120
93
121
Setup scripts can define environment variables that will be exposed to tests that match the script. This is done by writing to the `$NEXTEST_ENV` environment variable from within the script.
94
122
@@ -126,6 +154,18 @@ fn my_env_test() {
126
154
}
127
155
```
128
156
157
+
### Pre-timeout scripts
158
+
159
+
A given pre-timeout script _S_ is executed when the current profile has at least one rule where the `platform` predicates match the current execution environment, the script _S_ is listed in `pre-timeout`, and a test matching the `filter` has reached its configured timeout.
160
+
161
+
Pre-timeout scripts are executed serially, in the order they are defined (_not_ the order they're specified in the rules). If any pre-timeout script exits with a non-zero exit code, an error is logged but the test run continues.
162
+
163
+
Nextest sets the following environment variables when executing a pre-timeout script:
164
+
165
+
***`NEXTEST_PRE_TIMEOUT_TEST_PID`**: the ID of the process running the test.
166
+
***`NEXTEST_PRE_TIMEOUT_TEST_NAME`**: the name of the running test.
167
+
***`NEXTEST_PRE_TIMEOUT_TEST_BINARY`**: the name of the binary running the test.
0 commit comments