Unstable CI tests

[vpratz]

I currently observe that tests sometimes fail, and then pass on repeat. Maybe we can use this issue to identify those cases and develop a strategy to make the tests more robust. I currently observe two failure modes that seem to be temporary (e.g. here):

• Jax on Windows sometimes produces Windows fatal exception: code 0x8001010d
• tests/test_two_moons/test_two_moons.py::test_fit[inference_network='coupling_flow'] produces too high values or nan

[paul-buerkner]

Are these tests still unstable after the latest fixes that @stefanradev93 introduced?

[stefanradev93]

Closing, as the tests seem to be stable now.

[vpratz]

Reopening this, as we still sometimes see tests/test_two_moons/test_two_moons.py::test_fit[inference_network='coupling_flow'] fail with too large values (but no nans). What is the best approach here? Can we safely increase the threshold for passing, or do we need a more powerful network/longer training to get more consistent results?

[LarsKue]

I'm afraid there's not much we can do beside revamping the test. For now, the metric tests are disabled because they are too flaky. In general, we need more thorough integration testing, not just on two moons, so I will make a separate issue for that. Closing this one.