2
2
3
3
[ ![ Coverage Status] ( https://coveralls.io/repos/github/tarantool/test-run/badge.svg )] ( https://coveralls.io/github/tarantool/test-run )
4
4
5
- ### Test Suite
6
-
7
- Bunch of tests, that lay down in the subfolder (recursively) with ` suite.ini `
8
- file. ` suite.ini ` is basic ini-file, that consists of one section ` default ` ,
9
- and a number of fields:
10
-
11
- * ` core `
12
- * ` description ` - Test Suite description
13
- * ` script ` - shebang file to start tarantool with
14
- * disables:
15
- * ` disabled ` - tests that must be skipped
16
- * ` release_disabled ` - tests that must be skipped when Tarantool has been
17
- builded with ` Release `
18
- * ` valgrind_disabled ` - tests that must be skipped when Valgrind is enabled
19
- * ` lua_libs ` - paths for lua files, that should be copied into the folder,
20
- where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua
21
- lua/2.lua`)
22
- * ` long_run ` - mark tests as long, enabled only with ` --long ` option (delimited
23
- with the space, e.g. ` long_run=t1.test.lua t2.test.lua ` )
24
- * ` config ` - test configuration file name
25
-
26
- Field ` core ` must be one of:
27
-
28
- * ` tarantool ` - Test-Suite for Functional Testing
29
- * ` app ` - Another functional Test-Suite
30
- * ` unittest ` - Unit-Testing Test Suite
31
-
32
- ### Test
33
-
34
- Each test consists of files ` *.test(.lua|.sql|.py)? ` , ` *.result ` , and may have
35
- skip condition file ` *.skipcond ` . On first run (without ` .result ` ) ` .result `
36
- is generated from output. Each run, in the beggining, ` .skipcond ` file is
37
- executed. In the local env there's object ` self ` , that's ` Test ` object. If test
38
- must be skipped - you must put ` self.skip = 1 ` in this file. Next,
39
- ` .test(.lua|.py)? ` is executed and file ` .reject ` is created, then ` .reject ` is
40
- compared with ` .result ` . If something differs, then 15 last string of this diff
41
- file are printed and ` .reject ` file is saving in the ` <vardir>/rejects/<suite> `
42
- subfolder given in options or set localy as ` var/rejects/<suite> ` by default.
43
- If not, then ` .reject ` file is deleted.
5
+ ## Test suite configuration
6
+
7
+ Test suite is a bunch of tests located in a subdirectory (recursively) with a ` suite.ini `
8
+ file. The ` suite.ini ` is a basic ini-file with one section ` [default] ` .
9
+
10
+ ``` ini
11
+ [default]
12
+ core = luatest
13
+ description = Example test suite using luatest
14
+ ; comment line
15
+ ...
16
+ ```
17
+
18
+ All list values in ` suite.ini ` are delimited by spaces.
19
+ Lines starting with ` ; ` are comment lines.
20
+
21
+ Below is a list of configuration values (fields) in the ` suite.ini ` file.
22
+
23
+ ### General configuration
24
+
25
+ * ` core ` — major testing dependency or method
26
+ Should have one of the following values:
27
+
28
+ * ` tarantool ` — test suite used for functional testing
29
+ * ` app ` — TAP-tests, another functional test suite
30
+ * ` unittest ` — unit testing test suite
31
+ * ` luatest ` — test suite using luatest library
32
+
33
+ * ` description ` — test suite description
34
+ * ` script ` (optional) — shebang file to start tarantool with
35
+ * ` config ` — name of a test configuration file
36
+ TODO: what's a test configuration file?
37
+
38
+ ### Skipping tests
39
+
40
+ A number of fields are used to disable (skip) certain tests:
41
+
42
+ * ` disabled ` — list of tests that should be skipped
43
+ * ` release_disabled ` — list of tests that should be skipped when Tarantool is
44
+ built with ` Release `
45
+ * ` valgrind_disabled ` — list of tests that should be skipped when Valgrind is enabled
46
+ * ` long_run ` — mark tests as long, such tests will run only with ` --long ` option.
47
+
48
+ ``` ini
49
+ long_run = t1.test.lua t2.test.lua
50
+ ```
51
+
52
+ ### Other parameters
53
+
54
+ * ` lua_libs ` — paths for lua files, that should be copied into the folder,
55
+ where server is started. For example:
56
+
57
+ ``` ini
58
+ lua_libs = lua/require_mod.lua lua/serializer_test.lua lua/process_timeout.lua
59
+ ```
60
+
61
+ * ` show_reproduce_content ` (optional, ` True ` by default) — TODO: what's this?
62
+ * ` is_parallel ` (optional, ` False ` by default) — whether the tests in the suite can run in parallel
63
+
64
+ * ` use_unix_sockets_iproto ` (optional, ` False ` by default) — use hard-coded unix sockets
65
+ * ` use_unix_sockets_iproto ` (optional, ` False ` by default) — use hard-coded unix sockets for iproto.
66
+ This parameter is specific to ` core = app ` .
67
+
68
+ ### Fragile tests
69
+
70
+ Tests which fail sometimes due to external reasons can be marked as fragile (flaky).
71
+ Test-run will retry each test if it fails.
72
+ It's recommended to provide a list of related
73
+ [ tarantool/tarantool] ( https://github.com/tarantool/tarantool ) issues
74
+ next to each fragile test.
75
+
76
+ ``` ini
77
+ fragile = {
78
+ " retries" : 10,
79
+ " tests" : {
80
+ " tarantoolctl.test.lua" : {
81
+ " issues" : [ " gh-5059" , " gh-5346" ],
82
+ },
83
+ {...},
84
+ {...},
85
+ }
86
+ }
87
+ ```
88
+
89
+ ## Steps of a test run
90
+
91
+ Each test consists of files ` *.test(.lua|.sql|.py)? ` , ` *.result ` ,
92
+ and an optional skip condition file ` *.skipcond ` .
93
+
94
+ In the beginning of each test run, the ` .skipcond ` file is executed as a Python script.
95
+ In the local Python environment there's a ` self ` object, which is an instance of the [ ` Test ` class] ( ./lib/test.py ) .
96
+ To skip the test under some condition, set ` self.skip = 1 ` .
97
+
98
+ Next, the test in ` .test(.lua|.py)? ` is executed.
99
+ If this is a first test run and there's no ` .result ` file yet, it is generated from the output.
100
+ If there is a ` .result ` file, output is written to a ` .reject ` file,
101
+ which is then compared to ` .result ` .
102
+
103
+ If there's a difference between ` .reject ` and ` .result ` , the following happens:
104
+
105
+ 1 . The last 15 lines of diff are printed to output.
106
+ 2 . The ` .reject ` file is saved in the ` <vardir>/rejects/<suite> `
107
+ subfolder given in options or set locally as ` var/rejects/<suite> ` by default.
108
+
109
+ If there is no difference between ` .reject ` and ` .result ` , the ` .reject ` file is deleted.
44
110
45
111
### Test configuration
46
112
47
- Test configuration file contains config for multiple run. For each test section
48
- system runs separated test and compares result with common ` .result ` file. For
49
- example we need to run one test for different db engines( "* " means default
113
+ Test configuration file contains configuration for multiple runs.
114
+ For each test section, system runs a separate test and compares the result to the common ` .result ` file.
115
+ For example, we need to run a test with different DB engines ( ` "*" ` means the default
50
116
configuration):
51
117
52
118
``` json
@@ -62,15 +128,15 @@ configuration):
62
128
}
63
129
```
64
130
65
- In test case we can get configuration from inspector:
131
+ In the test case we can get configuration from the inspector:
66
132
67
133
``` lua
68
134
engine = test_run :get_cfg (' engine' )
69
135
-- first run engine is 'memtx'
70
136
-- second run engine is 'sophia'
71
137
```
72
138
73
- "engine" value has a special meaning for * .test.sql files: if it is "memtx" or
139
+ "engine" value has a special meaning for ` *.test.sql ` files: if it is "memtx" or
74
140
"vinyl", then the corresponding default engine will be set before executing
75
141
commands from a test file. An engine is set with the following commands:
76
142
@@ -79,11 +145,11 @@ UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_defau
79
145
pragma sql_default_engine= ' memtx|vinyl'
80
146
```
81
147
82
- If the first fails, then the second will be executed. When both fails, fail the test.
148
+ If the first fails, then the second will be executed. When both fail, the test fails .
83
149
84
- #### Python
150
+ #### Python tests
85
151
86
- Files: ` <name>.test.py ` , ` <name>.result ` and ` <name>.skipcond ` (optionaly ).
152
+ Files: ` <name>.test.py ` , ` <name>.result ` and ` <name>.skipcond ` (optionally ).
87
153
88
154
Environment:
89
155
@@ -173,7 +239,7 @@ Tests interact only with `AdminConnection`. Supports some preprocessor functions
173
239
174
240
** Delimiter example:**
175
241
176
- ```
242
+ ``` lua
177
243
env = require (' test_run' )
178
244
test_run = env .new ()
179
245
box .schema .space .create (' temp' )
194
260
195
261
** Delimiter result:**
196
262
197
- ```
263
+ ``` yaml
198
264
env = require('test_run')
199
265
test_run = env.new()
200
266
box.schema.space.create('temp')
0 commit comments