Resurrect integration tests

[Lorak-mmk]

Our integration tests are in a pretty sorry state, which isn't good considering we use this driver for pretty much all Scylla tests.
I'd like to improve this situation so I'm opening this issue to keep track of those efforts and the current state.
I'm describing the problems as I understand them, and I'm not an expert in this codebase. If anything here is incorrect feel free to correct me.

Problem 1: We don't run all the tests

Currently we are running standard and cqlengine tests. We should work on enabling long, upgrade and simulacron tests, and run them, but not in CI (parhaps before reach release).

TODO:

• Fix and enable the rest of tests/integration/standard tests.
• Fix and enable tests/integration/cqlengine tests.
• Fix tests/integration/long, and start running them before release
• Adjust tests/integration/upgrade so that it can test Scylla upgrades too, and run them before release.
• Remove tests that use DSE-specific functionality. I don't think we support such functionality, so it would we better to yank all DSE-specific code from the driver - to reduce complexity and avoid confusion (e.g. DSE Cloud vs Scylla Cloud). This includes removing tests/integration/cloud and most of tests/integration/advanced, along with associated code in the driver and documentation.
• I'm not sure what to do about tests/integration/simulacron and what even are those - more research needed. If they can be used with Scylla - great. If not, let's not delete them - we can use them with Cassandra only.

Problem 2: We only test with Scylla, and with 1 version of it.

This is pretty self explanatory. I think we should run integration tests with:
- Currently supported stable Scylla releases (right now: 5.1 and 5.0)
- Latest Cassandra stable release
- Optional: upcoming Scylla stable release.

Matrix tests (when we get them to work) take care of testing with various Scylla version. It would be breat to also add integration tests CI running on latest stable Cassandra.
TODO:

• Add Cassandra integration tests to CI

Problem 3: Tests are slow

Tests are run sequentially, and each one needs to setup a cluster - unless the previous one was using cluster with the same topology. This is quite slow, considering very long Scylla startup times. Afaik this is the limitation of tests/integration/__init__.py (unless there is some limitation I don't know about, like ccm not being able to run >1 cluster), that has a global var to hold a cluster - so it can only use one at a time.

Tests are now faster than in the past, thank to improvements in Scylla startup times.
The above about sub-optimal usage of clusters, resulting in more restarts than necessary, is still true, and accounts for big chunk of execution time.
What can we do about it? Migrate tests to pytest fully, so that we are able to use fixtures. Then, we can use them (and optionally pytest hooks) to guarantee optimal execution order and cluster utilization.
Apart from that, some tests are itself quite slow. Sometimes because of sleeps, sometimes because they to a lot of (perhaps not necessary) work.

TODD:

• Migrate test code from unittest to pytest
• Prepare test utils (fixtures etc) to make cluster usage by tests optimal
• Migrate tests to use the above
• Investigate tests execution time, and fix slow tests
• Bonus: refactor __init__.py in integration to be more comprehensible.

Problem 4: We only run those tests with asyncio reactor

That one I'm not sure is important or even correct.
I can see that there are more reactors supported by driver (and we do run some unit tests on some of those), yet we only use asyncio libev during integration tests. Shouldn't we run some tests with all the other ones?

We now run integration tests with libev, asyncio, and asyncore (for older Python version). We could start testing the rest of reactors too (eventlet, gevent, twisted).
This would result in more resources spent on integration testing - I think we can reduce amount of tested Python version to counteract this (maybe test latest, and oldest supported?).
We still run unit tests with only 1 reactor (I think) - we could tests more here as well.

Problem 5: A lot of unnecessary logs from ccm (I guess?).

Setup / teardown logs are massive - they cause my terminal to lag, and require some time to scroll trough. Most of the time they are also completely useless. They should be hidden by default and only displayed when some env variable is set.

I think this is no longer a problem (because I don't remember encountering it lately), but I'm not sure. Will update if I ever encounter it.

Problem 6: Multiple warnings

There are a lot of warnings printed when executing integration test. Not all of them are our fault, but we should fix those that we can, as they make it harder to read the results.

Problem 7: Skipped tests

A lot of tests are skipped - we should go through them, and make sure there are no unnecessary skips.

[Lorak-mmk]

Added problems 5 and 6

[Lorak-mmk]

First PR regarding those problems: #210

[fruch]

On note on the removal of some code and handling of warning, keep in mind it might complex a bit the syncing with upstream (even that upstream doesn't move at a very fast pace)

regarding testing with multiple reactors/backends, unfortunately they are not exactly equivalent, and you can run all the test ontop of all of them.
also the main one that is most battle proven from my POV is the libev one, it's the default once it's compiled, and that the one we use in dtest, when using any other, the behavior in some situation isn't the same, especially on scylla node restarts.

I don't even remember why asyncio is used in integration tests, but recently I switched it to libev a444188, exactly cause of sni_proxy tests that where using multiple ConncectionClasses

[fruch]

@Lorak-mmk @avelanarius, I've added you both to:
https://github.com/orgs/scylladb/teams/python-driver-maint

[nyh]

* Remove tests that use DSE-specific functionality. I don't think we support such functionality, so it would we better to yank all DSE-specific code from the driver - to reduce complexity and avoid confusion (e.g. DSE Cloud vs Scylla Cloud). This includes removing `tests/integration/cloud` and most of `tests/integration/advanced`, along with associated code in the driver and documentation.

If you plan to remove the DSE-specific functionality itself, then it makes sense to remove the tests. But if you leave the functionality, it would be a bad idea to remove the tests, as this can lead to these DSE-specific features in our driver to break without anyone noticing.

Do we plan to remove the DSE-specific functionality? This can hurt our driver's ability to become the "default" Cassandra driver, but do we have such a goal?

* I'm not sure what to do about `tests/integration/simulacron` and what even are those - more research needed. If they can be used with Scylla - great. If not, let's not delete them - we can use them with Cassandra only.

https://github.com/datastax/simulacron

It's meant to simulate various protocol corner cases without a real server (without Cassandra or Scylla).

Problem 2: We only test with Scylla, and with 1 version of it.

This is pretty self explanatory. I think we should run integration tests with:

* Currently supported stable Scylla releases (right now: 5.1 and 5.0)

* Latest Cassandra stable release

Right. Very important in my opinion.

* Optional: upcoming Scylla stable release.

My personal favorite approach: CI should run against a specific version of Scylla (e.g., 5.1), but when a developer runs the test, it runs against whatever Scylla version this developer might have lying around - or docker if there is none.

By the way, check out Scylla's test/cql-pytest/run and test/cql-pytest/run-cassandra for example scripts that run Scylla or Cassandra very quickly. Or, I guess, you can also use ccm...

Problem 3: Tests are slow

Tests are run sequentially, and each one needs to setup a cluster - unless the previous one was using cluster with the same topology. This is quite slow, considering very long Scylla startup times. Afaik this is the limitation of tests/integration/__init__.py (unless there is some limitation I don't know about, like ccm not being able to run >1 cluster), that has a global var to hold a cluster - so it can only use one at a time.

This is very important, and the main reason why developers like me avoid dtest like the plague ;-) You can use ideas from Scylla's test/cql-pytest for single-node tests or test/topology for multi-node tests, or come up with a simpler test framework. In particular "ccm" was a good idea, but an inconvenient and much-too-slow implementation.

[fruch]

* Remove tests that use DSE-specific functionality. I don't think we support such functionality, so it would we better to yank all DSE-specific code from the driver - to reduce complexity and avoid confusion (e.g. DSE Cloud vs Scylla Cloud). This includes removing `tests/integration/cloud` and most of `tests/integration/advanced`, along with associated code in the driver and documentation.

If you plan to remove the DSE-specific functionality itself, then it makes sense to remove the tests. But if you leave the functionality, it would be a bad idea to remove the tests, as this can lead to these DSE-specific features in our driver to break without anyone noticing.

Do we plan to remove the DSE-specific functionality? This can hurt our driver's ability to become the "default" Cassandra driver, but do we have such a goal?

Why would any of our drivers have this as a goal ?

From my POV removing that code, adds the the complexity to taking code from upstream, and if we are o.k. with making that process more painful, then yes yank all of that code.

* I'm not sure what to do about `tests/integration/simulacron` and what even are those - more research needed. If they can be used with Scylla - great. If not, let's not delete them - we can use them with Cassandra only.

https://github.com/datastax/simulacron

It's meant to simulate various protocol corner cases without a real server (without Cassandra or Scylla).

Problem 2: We only test with Scylla, and with 1 version of it.

This is pretty self explanatory. I think we should run integration tests with:

* Currently supported stable Scylla releases (right now: 5.1 and 5.0)

* Latest Cassandra stable release

Right. Very important in my opinion.

* Optional: upcoming Scylla stable release.

My personal favorite approach: CI should run against a specific version of Scylla (e.g., 5.1), but when a developer runs the test, it runs against whatever Scylla version this developer might have lying around - or docker if there is none.

By the way, check out Scylla's test/cql-pytest/run and test/cql-pytest/run-cassandra for example scripts that run Scylla or Cassandra very quickly. Or, I guess, you can also use ccm...

Tests are already using ccm

Problem 3: Tests are slow

Tests are run sequentially, and each one needs to setup a cluster - unless the previous one was using cluster with the same topology. This is quite slow, considering very long Scylla startup times. Afaik this is the limitation of tests/integration/__init__.py (unless there is some limitation I don't know about, like ccm not being able to run >1 cluster), that has a global var to hold a cluster - so it can only use one at a time.

This is very important, and the main reason why developers like me avoid dtest like the plague ;-) You can use ideas from Scylla's test/cql-pytest for single-node tests or test/topology for multi-node tests, or come up with a simpler test framework. In particular "ccm" was a good idea, but an inconvenient and much-too-slow implementation.

Nice to see you guys running into re implementation of almost the same.

Anyhow tests here are using ccm, and the problems they have are not related to ccm, ccm can easily run multiple clusters at the same time, and doesn't have any globals as described here. and can be as fast as scylla can boot.

[mykaul]

Any updates on this epic?

[Lorak-mmk]

There is an open PR: #219 - I need to fix a flaky test there before it can merged. I was on a leave last week, that's why I didn't do it.
After it's merged, most important things from section 1 will be fixed.

After that, I plan to fix section 2 - it should be quick and easy to do. The rest require more work, not sure when I'll get around to them.