Skip to content

Update and refine the readme #323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Update and refine the readme #323

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f160970
Update and refine the readme
NickVolynkin Feb 14, 2022
10ba68e
update
NickVolynkin Mar 3, 2022
ef57de9
update 2
NickVolynkin Mar 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
276 changes: 215 additions & 61 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,75 +2,227 @@

[![Coverage Status](https://coveralls.io/repos/github/tarantool/test-run/badge.svg)](https://coveralls.io/github/tarantool/test-run)

### Test Suite

Bunch of tests, that lay down in the subfolder (recursively) with `suite.ini`
file. `suite.ini` is basic ini-file, that consists of one section `default`,
and a number of fields:

* `core`
* `description` - Test Suite description
* `script` - shebang file to start tarantool with
* disables:
* `disabled` - tests that must be skipped
* `release_disabled` - tests that must be skipped when Tarantool has been
builded with `Release`
* `valgrind_disabled` - tests that must be skipped when Valgrind is enabled
* `lua_libs` - paths for lua files, that should be copied into the folder,
where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua
lua/2.lua`)
* `long_run` - mark tests as long, enabled only with `--long` option (delimited
with the space, e.g. `long_run=t1.test.lua t2.test.lua`)
* `config` - test configuration file name

Field `core` must be one of:

* `tarantool` - Test-Suite for Functional Testing
* `app` - Another functional Test-Suite
* `unittest` - Unit-Testing Test Suite

### Test

Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have
skip condition file `*.skipcond`. On first run (without `.result`) `.result`
is generated from output. Each run, in the beggining, `.skipcond` file is
executed. In the local env there's object `self`, that's `Test` object. If test
must be skipped - you must put `self.skip = 1` in this file. Next,
`.test(.lua|.py)?` is executed and file `.reject` is created, then `.reject` is
compared with `.result`. If something differs, then 15 last string of this diff
file are printed and `.reject` file is saving in the `<vardir>/rejects/<suite>`
subfolder given in options or set localy as `var/rejects/<suite>` by default.
If not, then `.reject` file is deleted.

### Test configuration

Test configuration file contains config for multiple run. For each test section
system runs separated test and compares result with common `.result` file. For
example we need to run one test for different db engines("*" means default
configuration):

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
## Table of Contents

- [Test suite configuration file](#test-suite-configuration-file)
- [General configuration values](#general-configuration-values)
- [Disabling and skipping tests](#disabling-and-skipping-tests)
- [Other parameters](#other-parameters)
- [Fragile tests](#fragile-tests)
- [Test composition](#test-composition)
- [Test execution](#test-execution)
- [Test configuration](#test-configuration)
- [Writing various types of tests](#writing-various-types-of-tests)
- [Python tests](#python-tests)
- [Lua](#lua)
- [SQL](#sql)
- [Interaction with the test environment](#interaction-with-the-test-environment)
- [pretest_clean()](#pretest_clean)
- [Tags](#tags)
- [Projects that use test-run](#projects-that-use-test-run)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Test suite configuration file

Test suite is a bunch of tests located in a directory with a `suite.ini`
file. The `suite.ini` is a basic ini-file with one section `[default]`.

```ini
[default]
core = luatest
description = Example test suite using luatest
...
```

Below is a list of configuration values (fields) in the `suite.ini` file.

### General configuration values

* `core` — major type of tests in this suite.
Should have one of the following values:

* `tarantool` — a test that runs `tarantool` commands from a file,
and then compares output with a reference file.
Often called a "diff-test".
* `app` — a Lua script test.
Most of such tests produce
[TAP13 output](http://testanything.org/tap-version-13-specification.html)
using the built-in `tap` module.
Test-run validates such output without a reference file.
Some tests produce non-TAP13 output, which is compared to a reference file.
* `luatest` — test suite using the [luatest](https://github.com/tarantool/luatest/) library.
* `unittest` — an executable test file.

* `description`

* `script` — The file with Tarantool commands, used in `core = tarantool` tests.
It is used to start the default server using `tarantoolctl`.
The value should be a file in the same directory as `suite.ini`, like `box.lua` or `master.lua`.
This setting is mandatory for test suites with `core = tarantool`
and ignored in other types of tests.

* `config` — name of a test configuration file, for example, `engine.cfg`.
For details, see the [test configuration](#test-configuration) section below.

* `lua_libs` — paths to Lua files that should be copied to the directory,
where a server is started. For example:

```ini
lua_libs = lua/require_mod.lua lua/serializer_test.lua lua/process_timeout.lua
```

### Disabling and skipping tests

A number of fields are used to disable certain tests:

* `disabled` — list of tests that should be skipped under all conditions.

* `release_disabled` — list of tests that should only run when Tarantool is
built in the debug mode (with `-DCMAKE_BUILD_TYPE=Debug` and not `=RelWithDebInfo`).
Tests that use error injections, which are only available in debug builds,
should always be `release_disabled`.

* `valgrind_disabled` — list of tests that should be skipped when Valgrind is enabled

* `long_run` — mark tests as long. Such tests will run only with `--long` option.

```ini
long_run = t1.test.lua t2.test.lua
```

Note that there's also an option to conditionally skip a test with [.skipcond](#test-composition) files.

### Other parameters

* `is_parallel` (optional, `False` by default) — whether the tests in the suite can run in parallel

* `use_unix_sockets` (optional, `False` by default) — use UNIX sockets for console
* `use_unix_sockets_iproto` (optional, `False` by default) — use UNIX sockets for IProto.

* `show_reproduce_content` (optional, `True` by default) — when set to `True`,
show the contents of the reproduce file for each failed test.
(Implemented in [#113](https://github.com/tarantool/test-run/issues/113))

Reproduce files are not required for investigating results of
tests that run each case in a separate Tarantool instance,
which is the preferred way to run tests.
For such tests it makes sense to set `show_reproduce_content = False`.

### Fragile tests

Tests which fail sometimes due to external reasons can be marked as fragile (flaky).
Test-run will retry each test if it fails.
It's recommended to provide a list of related
[tarantool/tarantool](https://github.com/tarantool/tarantool) issues
next to each fragile test.

```ini
fragile = {
"retries": 10,
"tests": {
"tarantoolctl.test.lua": {
"issues": [ "gh-5059", "gh-5346" ],
},
{...},
{...},
}
}
```

## Test composition

Each test consists of the following files:

* Test file: `<name>.test.lua`, `<name>_test.lua`, `<name>.test.py`, or `<name>.test.sql`.
* Reference file (optional): `<name>.result` .
* Skip condition file (optional): `<name>.skipcond`.

Reference file contains saved test output.
It is required for tests with non-TAP13 output.
Running `test-run.py` with `--update-result` option will update
the `.result` files with new test output.

The optional skip condition file is a Python script.
It is used to skip a test on some conditions, typically on a certain OS.

In the local Python environment of a test run there's a `self` object,
which is an instance of the [`Test` class](./lib/test.py).
Set `self.skip = 1` to skip this test.
For example, to skip `sometest` on OpenBSD,
add the following `sometest.skipcond` file:

```python
import platform

# Disabled on OpenBSD due to fail #XXXX.
if platform.system() == 'OpenBSD':
self.skip = 1
```


## Test execution

Running a test begins with executing the `.skipcond` file.
If after execution `self.skip` equals `1`, test-run skips this test.

Next, the test file is executed and the output is written to a `.reject` file.
If the output is TAP13-compatible, test-run validates it.
Otherwise, test-run compares it with the `.result` file.
If there's a difference between `.reject` and `.result`, the test fails and
the last 15 lines of diff are printed to output.

Whenever a test fails, the `.reject` file is saved
and the path to this file is printed to output.

## Test configuration

Test configuration file contains configuration for multiple runs.
For each test section, system runs a separate test and compares the result to the common `.result` file.

For example, `my.test.lua` will run with two different sets of parameters:

```json
{
"my.test.lua": {
"first": {"a": 1, "b": 2},
"second": {"a": 1, "b": 3}
}
}
```

A common case is to run a test with different DB engines.
In the example below:

* `first.test.lua` will run only on the memtx engine;
* `second.test.lua` will run as is, without parameterizing;
* all other tests in the suite (`*`) will be parameterized to run on both memtx and vinyl engines.

```json
{
"first.test.lua": {
"memtx": {"engine": "memtx"}
},
"second.test.lua": {},
"*": {
"memtx": {"engine": "memtx"},
"sophia": {"engine": "sophia"}
"vinyl": {"engine": "vinyl"}
}
}
```

In test case we can get configuration from inspector:
In the test case we can get configuration from the inspector:

```lua
engine = test_run:get_cfg('engine')
-- first run engine is 'memtx'
-- second run engine is 'sophia'
-- second run engine is 'vinyl'
```

"engine" value has a special meaning for *.test.sql files: if it is "memtx" or
"engine" value has a special meaning for `*.test.sql` files: if it is "memtx" or
"vinyl", then the corresponding default engine will be set before executing
commands from a test file. An engine is set with the following commands:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding commands below: maybe it is too deep details for a user guide.


Expand All @@ -79,11 +231,13 @@ UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_defau
pragma sql_default_engine='memtx|vinyl'
```

If the first fails, then the second will be executed. When both fails, fail the test.
If the first fails, then the second will be executed. When both fail, the test fails.

## Writing various types of tests

#### Python
### Python tests

Files: `<name>.test.py`, `<name>.result` and `<name>.skipcond`(optionaly).
Files: `<name>.test.py`, `<name>.result` and `<name>.skipcond` (optionally).

Environment:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sql was changed to remote in 2015 (a9adaf5) and then renamed to iproto (6adba83). Also the BoxConnection class definitely changes its meaning since those times. The examples below are irrelevant.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While admin below still actual.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also suggest to re-verify that the example below still work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to touch this part of readme in this PR. Will review it later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How I'll differentiate parts, where problems are left deliberately and where it is unintentional? No, I think your point does not work here: your changes are not so dense.


Expand Down Expand Up @@ -166,14 +320,14 @@ box.info.lsn
...
```

#### Lua
### Lua
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

core = tarantool and core = app tests are really different. It worth to split its description.


Files: `<name>.test.lua`, `<name>.result` and `<name>.skipcond`(optionaly).
Tests interact only with `AdminConnection`. Supports some preprocessor functions (eg `delimiter`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests interact only with AdminConnection.

Don't know what it means, but it looks irrelevant now.


**Delimiter example:**

```
```lua
env = require('test_run')
test_run = env.new()
box.schema.space.create('temp')
Expand All @@ -194,7 +348,7 @@ test

**Delimiter result:**

```
```yaml
env = require('test_run')
test_run = env.new()
box.schema.space.create('temp')
Expand Down Expand Up @@ -255,14 +409,14 @@ function echo(...) \
end
```

#### SQL
### SQL

*.test.sql files are just SQL statements written line-by-line.

It is possible to mix SQL and Lua commands using `\set language lua` and `\set
language sql` commands.

##### Interaction with the test environment
## Interaction with the test environment

In lua test you can use `test_run` module to interact with the test
environment.
Expand Down Expand Up @@ -337,7 +491,7 @@ test_run:cmd('setopt delimiter ""');
join(test_run, 30)
```

### pretest_clean()
## pretest_clean()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's strange that a function name is here. Let's give the section a human readable title.


Nothing will be done before a Python test and for `core = unittest`
test suites.
Expand All @@ -353,7 +507,7 @@ The following files will be removed:
* `*.inprogress`
* `[0-9]*/`

### Tags
## Tags

Usage:

Expand Down Expand Up @@ -409,7 +563,7 @@ Unsupported features:
* Marking unit tests with tags.
* Multiline comments (use singleline ones for now).

### Used By
## Projects that use test-run

- [Tarantool](https://github.com/tarantool/tarantool) - in-memory database and application server
- [memcached](https://github.com/tarantool/memcached) - Memcached protocol 'wrapper' for Tarantool
Expand Down