Full-spectrum integration testing system.

[grzkv]

Motivation

There are multiple bugs that are not detected by our current CI and unit tests. The stability bugs are especially frequent of those. Unit tests are not well-suited to test the parts of code that call into the OS and especially the ones using networking. An integration testing system is a much better candidate here.

Requirements

• Tests the following:
  • Stability and liveness of carbonapi while processing a set of requests.
  • Timeout test for a set of typical requests.
  • (in the future) Correctness test for a set of typical render and find requests.
  • (in the future) Test a metrics scrape.
• Can run automatically for each PR.
• Can run in any environment.
• Uses mock backends, which:
  • Are tens in number.
  • Are split into clusters.
  • Have replication.
• Uses neutral data. E.g. randomly generated data.

Design

System consists of three parts:

1. HTTP requests player, which will send a set of requests to carbonapi that are defined in advance.
2. carbonapi itself.
3. Set of mock backends, which simulate the real-life backends.

In the first version, it makes sense to run them on a single box (it can also be a container) or inside 3 separate containers via docker-compose. Each backend will occupy a port on a single machine.

HTTP player details

There are multiple available. We don't have to write our own. We need to try and choose one.

Mock backends

We will have to implement this part. Mock backends should support the main procol we use for communication with a real backend. They will play back preset responses for each request.

Workflow

The test will run something like this:

1. Bring up all the systems.
2. Send a set of requests to carbonapi.
3. Check if the responses were received.
4. (in the future) Check correctness of the responses.
5. (in the future) Try scraping metrics from carbonapi and check if they satisfy basic heuristics.
6. Check liveness of carbonapi.

[deniszh]

It makes sense to me.
In my opinion, we can start with the HTTP player part and correctness/scrape tests using the normal go-carbon backend, as we have in hurl tests, but without hurl. :)
Then we can spend time on the mock backend and create tests for stability and timeouts.