Skip to content

Commit 1a1002f

Browse files
committed
Update and refine the readme
1 parent 1689f58 commit 1a1002f

File tree

1 file changed

+115
-49
lines changed

1 file changed

+115
-49
lines changed

README.md

Lines changed: 115 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -2,51 +2,117 @@
22

33
[![Coverage Status](https://coveralls.io/repos/github/tarantool/test-run/badge.svg)](https://coveralls.io/github/tarantool/test-run)
44

5-
### Test Suite
6-
7-
Bunch of tests, that lay down in the subfolder (recursively) with `suite.ini`
8-
file. `suite.ini` is basic ini-file, that consists of one section `default`,
9-
and a number of fields:
10-
11-
* `core`
12-
* `description` - Test Suite description
13-
* `script` - shebang file to start tarantool with
14-
* disables:
15-
* `disabled` - tests that must be skipped
16-
* `release_disabled` - tests that must be skipped when Tarantool has been
17-
builded with `Release`
18-
* `valgrind_disabled` - tests that must be skipped when Valgrind is enabled
19-
* `lua_libs` - paths for lua files, that should be copied into the folder,
20-
where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua
21-
lua/2.lua`)
22-
* `long_run` - mark tests as long, enabled only with `--long` option (delimited
23-
with the space, e.g. `long_run=t1.test.lua t2.test.lua`)
24-
* `config` - test configuration file name
25-
26-
Field `core` must be one of:
27-
28-
* `tarantool` - Test-Suite for Functional Testing
29-
* `app` - Another functional Test-Suite
30-
* `unittest` - Unit-Testing Test Suite
31-
32-
### Test
33-
34-
Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have
35-
skip condition file `*.skipcond`. On first run (without `.result`) `.result`
36-
is generated from output. Each run, in the beggining, `.skipcond` file is
37-
executed. In the local env there's object `self`, that's `Test` object. If test
38-
must be skipped - you must put `self.skip = 1` in this file. Next,
39-
`.test(.lua|.py)?` is executed and file `.reject` is created, then `.reject` is
40-
compared with `.result`. If something differs, then 15 last string of this diff
41-
file are printed and `.reject` file is saving in the `<vardir>/rejects/<suite>`
42-
subfolder given in options or set localy as `var/rejects/<suite>` by default.
43-
If not, then `.reject` file is deleted.
5+
## Test suite configuration
6+
7+
Test suite is a bunch of tests located in a subdirectory (recursively) with a `suite.ini`
8+
file. The `suite.ini` is a basic ini-file with one section `[default]`.
9+
10+
```ini
11+
[default]
12+
core = luatest
13+
description = Example test suite using luatest
14+
; comment line
15+
...
16+
```
17+
18+
All list values in `suite.ini` are delimited by spaces.
19+
Lines starting with `;` are comment lines.
20+
21+
Below is a list of configuration values (fields) in the `suite.ini` file.
22+
23+
### General configuration
24+
25+
* `core` — major testing dependency or method
26+
Should have one of the following values:
27+
28+
* `tarantool` — test suite used for functional testing
29+
* `app` — TAP-tests, another functional test suite
30+
* `unittest` — unit testing test suite
31+
* `luatest` — test suite using luatest library
32+
33+
* `description` — test suite description
34+
* `script` (optional) — shebang file to start tarantool with
35+
* `config` — name of a test configuration file
36+
TODO: what's a test configuration file?
37+
38+
### Skipping tests
39+
40+
A number of fields are used to disable (skip) certain tests:
41+
42+
* `disabled` — list of tests that should be skipped
43+
* `release_disabled` — list of tests that should be skipped when Tarantool is
44+
built with `Release`
45+
* `valgrind_disabled` — list of tests that should be skipped when Valgrind is enabled
46+
* `long_run` — mark tests as long, such tests will run only with `--long` option.
47+
48+
```ini
49+
long_run = t1.test.lua t2.test.lua
50+
```
51+
52+
### Other parameters
53+
54+
* `lua_libs` — paths for lua files, that should be copied into the folder,
55+
where server is started. For example:
56+
57+
```ini
58+
lua_libs = lua/require_mod.lua lua/serializer_test.lua lua/process_timeout.lua
59+
```
60+
61+
* `show_reproduce_content` (optional, `True` by default) — TODO: what's this?
62+
* `is_parallel` (optional, `False` by default) — whether the tests in the suite can run in parallel
63+
64+
* `use_unix_sockets_iproto` (optional, `False` by default) — use hard-coded unix sockets
65+
* `use_unix_sockets_iproto` (optional, `False` by default) — use hard-coded unix sockets for iproto.
66+
This parameter is specific to `core = app`.
67+
68+
### Fragile tests
69+
70+
Tests which fail sometimes due to external reasons can be marked as fragile (flaky).
71+
Test-run will retry each test if it fails.
72+
It's recommended to provide a list of related
73+
[tarantool/tarantool](https://github.com/tarantool/tarantool) issues
74+
next to each fragile test.
75+
76+
```ini
77+
fragile = {
78+
"retries": 10,
79+
"tests": {
80+
"tarantoolctl.test.lua": {
81+
"issues": [ "gh-5059", "gh-5346" ],
82+
},
83+
{...},
84+
{...},
85+
}
86+
}
87+
```
88+
89+
## Steps of a test run
90+
91+
Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`,
92+
and an optional skip condition file `*.skipcond`.
93+
94+
In the beginning of each test run, the `.skipcond` file is executed as a Python script.
95+
In the local Python environment there's a `self` object, which is an instance of the [`Test` class](./lib/test.py).
96+
To skip the test under some condition, set `self.skip = 1`.
97+
98+
Next, the test in `.test(.lua|.py)?` is executed.
99+
If this is a first test run and there's no `.result` file yet, it is generated from the output.
100+
If there is a `.result` file, output is written to a `.reject` file,
101+
which is then compared to `.result`.
102+
103+
If there's a difference between `.reject` and `.result`, the following happens:
104+
105+
1. The last 15 lines of diff are printed to output.
106+
2. The `.reject` file is saved in the `<vardir>/rejects/<suite>`
107+
subfolder given in options or set locally as `var/rejects/<suite>` by default.
108+
109+
If there is no difference between `.reject` and `.result`, the `.reject` file is deleted.
44110

45111
### Test configuration
46112

47-
Test configuration file contains config for multiple run. For each test section
48-
system runs separated test and compares result with common `.result` file. For
49-
example we need to run one test for different db engines("*" means default
113+
Test configuration file contains configuration for multiple runs.
114+
For each test section, system runs a separate test and compares the result to the common `.result` file.
115+
For example, we need to run a test with different DB engines (`"*"` means the default
50116
configuration):
51117

52118
```json
@@ -62,15 +128,15 @@ configuration):
62128
}
63129
```
64130

65-
In test case we can get configuration from inspector:
131+
In the test case we can get configuration from the inspector:
66132

67133
```lua
68134
engine = test_run:get_cfg('engine')
69135
-- first run engine is 'memtx'
70136
-- second run engine is 'sophia'
71137
```
72138

73-
"engine" value has a special meaning for *.test.sql files: if it is "memtx" or
139+
"engine" value has a special meaning for `*.test.sql` files: if it is "memtx" or
74140
"vinyl", then the corresponding default engine will be set before executing
75141
commands from a test file. An engine is set with the following commands:
76142

@@ -79,11 +145,11 @@ UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_defau
79145
pragma sql_default_engine='memtx|vinyl'
80146
```
81147

82-
If the first fails, then the second will be executed. When both fails, fail the test.
148+
If the first fails, then the second will be executed. When both fail, the test fails.
83149

84-
#### Python
150+
#### Python tests
85151

86-
Files: `<name>.test.py`, `<name>.result` and `<name>.skipcond`(optionaly).
152+
Files: `<name>.test.py`, `<name>.result` and `<name>.skipcond` (optionally).
87153

88154
Environment:
89155

@@ -173,7 +239,7 @@ Tests interact only with `AdminConnection`. Supports some preprocessor functions
173239

174240
**Delimiter example:**
175241

176-
```
242+
```lua
177243
env = require('test_run')
178244
test_run = env.new()
179245
box.schema.space.create('temp')
@@ -194,7 +260,7 @@ test
194260

195261
**Delimiter result:**
196262

197-
```
263+
```yaml
198264
env = require('test_run')
199265
test_run = env.new()
200266
box.schema.space.create('temp')

0 commit comments

Comments
 (0)