Update and refine the readme

Author: NickVolynkin

README.md

@@ -2,51 +2,185 @@
 
 [![Coverage Status](https://coveralls.io/repos/github/tarantool/test-run/badge.svg)](https://coveralls.io/github/tarantool/test-run)
 
-### Test Suite
-
-Bunch of tests, that lay down in the subfolder (recursively) with `suite.ini`
-file. `suite.ini` is basic ini-file, that consists of one section `default`,
-and a number of fields:
-
-* `core`
-* `description` - Test Suite description
-* `script` - shebang file to start tarantool with
-* disables:
-    * `disabled` - tests that must be skipped
-    * `release_disabled` - tests that must be skipped when Tarantool has been
-      builded with `Release`
-    * `valgrind_disabled` - tests that must be skipped when Valgrind is enabled
-* `lua_libs` - paths for lua files, that should be copied into the folder,
-  where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua
-  lua/2.lua`)
-* `long_run` - mark tests as long, enabled only with `--long` option (delimited
-  with the space, e.g. `long_run=t1.test.lua t2.test.lua`)
-* `config` - test configuration file name
-
-Field `core` must be one of:
-
-* `tarantool` - Test-Suite for Functional Testing
-* `app` - Another functional Test-Suite
-* `unittest` - Unit-Testing Test Suite
-
-### Test
-
-Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have
-skip condition file `*.skipcond`.  On first run (without `.result`) `.result`
-is generated from output.  Each run, in the beggining, `.skipcond` file is
-executed. In the local env there's object `self`, that's `Test` object. If test
-must be skipped - you must put `self.skip = 1` in this file. Next,
-`.test(.lua|.py)?` is executed and file `.reject` is created, then `.reject` is
-compared with `.result`. If something differs, then 15 last string of this diff
-file are printed and `.reject` file is saving in the `<vardir>/rejects/<suite>`
-subfolder given in options or set localy as `var/rejects/<suite>` by default.
-If not, then `.reject` file is deleted.
-
-### Test configuration
-
-Test configuration file contains config for multiple run. For each test section
-system runs separated test and compares result with common `.result` file. For
-example we need to run one test for different db engines("*" means default
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+## Table of Contents
+
+- [Test suite configuration file](#test-suite-configuration-file)
+  - [General configuration values](#general-configuration-values)
+  - [Disabling and skipping tests](#disabling-and-skipping-tests)
+  - [Other parameters](#other-parameters)
+  - [Fragile tests](#fragile-tests)
+- [Test composition](#test-composition)
+- [Test execution](#test-execution)
+- [Test configuration](#test-configuration)
+- [Writing various types of tests](#writing-various-types-of-tests)
+  - [Python tests](#python-tests)
+  - [Lua](#lua)
+  - [SQL](#sql)
+- [Interaction with the test environment](#interaction-with-the-test-environment)
+- [pretest_clean()](#pretest_clean)
+- [Tags](#tags)
+- [Projects that use test-run](#projects-that-use-test-run)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Test suite configuration file
+
+Test suite is a bunch of tests located in a directory with a `suite.ini`
+file. The `suite.ini` is a basic ini-file with one section `[default]`.
+
+```ini
+[default]
+core = luatest
+description = Example test suite using luatest
+...
+```
+
+There's a syntax for single- and multiline lists. (TODO see ./sql-tap/suite.ini)
+For example:
+
+```ini
+single_line = first.test.lua second.test.lua third.test.lua
+multiline = first.test.lua ;
+            second.test.lua ;
+            third.test.lua ;
+            ...
+```
+
+Below is a list of configuration values (fields) in the `suite.ini` file.
+
+### General configuration values
+
+* `core` — major type of tests in this suite.
+  Should have one of the following values:
+
+  * `tarantool` — diff-test, a test that reads a file with commands, 
+    feeds it line by line to the tarantool console,
+    writes requests and responses to an output file,
+    and then compares that file with a reference file.
+  * `app` — a Lua script test.
+    Most of such tests produce
+    [TAP13 output](http://testanything.org/tap-version-13-specification.html)
+    using the built-in `tap` module.
+    Test-run validates such output without a reference file.
+    Some tests produce non-TAP13 output, which is compared to a reference file.
+  * `luatest` — test suite using the [luatest](https://github.com/tarantool/luatest/) library.
+  * `unittest` — an executable test file.
+
+* `description`
+
+* `script` — A file with Tarantool commands.
+  It is used to start the default server using `tarantoolctl`.
+  The value should be a file in the same directory as `suite.ini`, like `box.lua` or `master.lua`.
+  This setting is mandatory for suites with diff-tests (`core = tarantool`)
+  and ignored with other types of tests.
+  
+* `config` — name of a test configuration file, for example, `engine.cfg`.
+  For details, see the [test configuration](#test-configuration) section below.
+
+* `lua_libs` — paths to Lua files that should be copied to the directory,
+  where a server is started. For example:
+
+  ```ini
+  lua_libs = lua/require_mod.lua lua/serializer_test.lua lua/process_timeout.lua
+  ```
+
+### Disabling and skipping tests
+
+A number of fields are used to disable certain tests:
+
+* `disabled` — list of tests that should be skipped.
+
+* `release_disabled` — list of tests that should only run when Tarantool is
+   built in the debug mode (with `-DCMAKE_BUILD_TYPE=Debug` and not `=RelWithDebInfo`).
+   Tests that use error injections, which are only available in debug builds,
+   should always be `release_disabled`.
+
+* `valgrind_disabled` — list of tests that should be skipped when Valgrind is enabled
+
+* `long_run` — mark tests as long. Such tests will run only with `--long` option.
+
+  ```ini
+  long_run = t1.test.lua t2.test.lua
+  ```
+
+### Other parameters
+
+* `show_reproduce_content` (optional, `True` by default) — when set to `True`,
+  show the contents of the reproduce file for each failed test.
+  Reproduce files are not required for investigating results of
+  tests that run each case in a separate Tarantool instance.
+  For such tests it makes sense to set `show_reproduce_content = False`.
+  (Implemented in [#113](https://github.com/tarantool/test-run/issues/113))
+
+* `is_parallel` (optional, `False` by default) — whether the tests in the suite can run in parallel
+
+* `use_unix_sockets` (optional, `False` by default) — use hard-coded UNIX sockets
+* `use_unix_sockets_iproto` (optional, `False` by default) — use hard-coded UNIX sockets for IProto.
+
+### Fragile tests
+
+Tests which fail sometimes due to external reasons can be marked as fragile (flaky).
+Test-run will retry each test if it fails.
+It's recommended to provide a list of related
+[tarantool/tarantool](https://github.com/tarantool/tarantool) issues
+next to each fragile test.
+
+```ini
+fragile = {
+    "retries": 10,
+    "tests": {
+        "tarantoolctl.test.lua": {
+            "issues": [ "gh-5059", "gh-5346" ],
+        },
+        {...},
+        {...},
+    }
+}
+```
+
+## Test composition
+
+Each test consists of the following files:
+
+* Test file: `<name>.test.lua`, `<name>_test.lua`, `<name>.test.py`, or `<name>.test.sql`.
+* Reference file: `<name>.result`.
+* Skip condition file: `<name>.skipcond`.
+
+Reference file contains saved test output.
+It is required for tests with non-TAP13 output.
+Running `test-run.py` with `--update-result` option will update
+the `.result` files with new test output.
+
+The optional skip condition file is a Python script.
+In the local Python environment of a test run there's a `self` object,
+which is an instance of the [`Test` class](./lib/test.py).
+Set `self.skip = 1` to skip this test.
+
+
+## Test execution
+
+Running a test begins with executing the `.skipcond` file.
+If `self.skip` is set to `1`, test-run skips this test
+and reports TODO.
+
+Next, the test file is executed and the output is written to a `.reject` file.
+If the output is TAP13-compatible, test-run validates it.
+Otherwise, test-run compares it with the `.result` file.
+If there's a difference between `.reject` and `.result`, the test fails and
+the last 15 lines of diff are printed to output.
+
+Whenever a test fails, the `.reject` file is saved in the `<vardir>/rejects/<suite>`
+subdirectory given in options or set locally as `var/rejects/<suite>` by default.
+If the test is considered successful, the `.reject` file is deleted.
+
+## Test configuration
+
+Test configuration file contains configuration for multiple runs.
+For each test section, system runs a separate test and compares the result to the common `.result` file.
+For  example, we need to run a test with different DB engines (`"*"` means the default
 configuration):
 
 ```json
@@ -62,15 +196,15 @@ configuration):
 }
 ```
 
-In test case we can get configuration from inspector:
+In the test case we can get configuration from the inspector:
 
 ```lua
 engine = test_run:get_cfg('engine')
 -- first run engine is 'memtx'
 -- second run engine is 'sophia'
 ```
 
-"engine" value has a special meaning for *.test.sql files: if it is "memtx" or
+"engine" value has a special meaning for `*.test.sql` files: if it is "memtx" or
 "vinyl", then the corresponding default engine will be set before executing
 commands from a test file. An engine is set with the following commands:
 
@@ -79,11 +213,13 @@ UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_defau
 pragma sql_default_engine='memtx|vinyl'
 ```
 
-If the first fails, then the second will be executed. When both fails, fail the test.
+If the first fails, then the second will be executed. When both fail, the test fails.
+
+## Writing various types of tests
 
-#### Python
+### Python tests
 
-Files: `<name>.test.py`, `<name>.result` and `<name>.skipcond`(optionaly).
+Files: `<name>.test.py`, `<name>.result` and `<name>.skipcond` (optionally).
 
 Environment:
 
@@ -166,14 +302,14 @@ box.info.lsn
 ...
 ```
 
-#### Lua
+### Lua
 
 Files: `<name>.test.lua`, `<name>.result` and `<name>.skipcond`(optionaly).
 Tests interact only with `AdminConnection`. Supports some preprocessor functions (eg `delimiter`)
 
 **Delimiter example:**
 
-```
+```lua
 env = require('test_run')
 test_run = env.new()
 box.schema.space.create('temp')
@@ -194,7 +330,7 @@ test
 
 **Delimiter result:**
 
-```
+```yaml
 env = require('test_run')
 test_run = env.new()
 box.schema.space.create('temp')
@@ -255,14 +391,14 @@ function echo(...) \
 end
 ```
 
-#### SQL
+### SQL
 
 *.test.sql files are just SQL statements written line-by-line.
 
 It is possible to mix SQL and Lua commands using `\set language lua` and `\set
 language sql` commands.
 
-##### Interaction with the test environment
+## Interaction with the test environment
 
 In lua test you can use `test_run` module to interact with the test
 environment.
@@ -337,7 +473,7 @@ test_run:cmd('setopt delimiter ""');
 join(test_run, 30)
 ```
 
-### pretest_clean()
+## pretest_clean()
 
 Nothing will be done before a Python test and for `core = unittest`
 test suites.
@@ -353,7 +489,7 @@ The following files will be removed:
 * `*.inprogress`
 * `[0-9]*/`
 
-### Tags
+## Tags
 
 Usage:
 
@@ -409,7 +545,7 @@ Unsupported features:
 * Marking unit tests with tags.
 * Multiline comments (use singleline ones for now).
 
-### Used By
+## Projects that use test-run
 
 - [Tarantool](https://github.com/tarantool/tarantool) - in-memory database and application server
 - [memcached](https://github.com/tarantool/memcached) - Memcached protocol 'wrapper' for Tarantool