Fixture not found in 8.0 that is found in 7.4.4

[larsoner]

In this run:

https://github.com/openmeeg/openmeeg/actions/runs/7700673043/job/20985097864?pr=631#step:6:8200

You can see the 3.7 wheel work during testing (which uses 7.4.4), but fail in 3.8 testing (which uses 8.0.0). If I use pytest --fixtures $TEST_PATH the fixture is there:

  ---------------- fixtures defined from openmeeg.tests.conftest ----------------
  data_path [session scope] -- ..\venv-test\lib\site-packages\openmeeg\tests\conftest.py:15
      no docstring available
  
  run_some_parallel [session scope] -- ..\venv-test\lib\site-packages\openmeeg\tests\conftest.py:27
      Run some stuff in parallel.

But when I run the tests with pytest --tb=short -ra -m 'not slow' -vv $TEST_PATH I get for all tests:

  ============================= test session starts =============================
  platform win32 -- Python 3.8.10, pytest-8.0.0, pluggy-1.4.0 -- C:\Users\runneradmin\AppData\Local\Temp\cibw-run-32upad1x\cp38-win_amd64\venv-test\Scripts\python.exe
  cachedir: .pytest_cache
  rootdir: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-32upad1x\cp38-win_amd64
  collecting ... collected 13 items / 2 deselected / 11 selected
  
  ..\venv-test\Lib\site-packages\openmeeg\tests\test_doc.py::test_doc <- venv-test\lib\site-packages\openmeeg\tests\test_doc.py ERROR [  9%]

...

  =================================== ERRORS ====================================
  _________________________ ERROR at setup of test_doc __________________________
  file C:\Users\runneradmin\AppData\Local\Temp\cibw-run-32upad1x\cp38-win_amd64\venv-test\lib\site-packages\openmeeg\tests\test_doc.py, line 6
    def test_doc():
  E       fixture 'run_some_parallel' not found
  >       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
  >       use 'pytest --fixtures [testpath]' for help on them.

Notably, this only happens on the Windows cibuildwheel run -- the Linux one in that same GHA is green. So the issue seems isolated to 8.0.0 on Windows. Nonetheless I am a bit mystified at why pytest --fixtures seems to see a different set of fixtures compared to 7.4.4 🤷

• a detailed description of the bug or problem you are having
• pytest and operating system versions
• output of pip list from the virtual environment you are using
• minimal example if possible

Not sure how to make this one mimimal but I could put some time into this on my Windows VM at some point if it's not clear enough what's going on.

[larsoner]

(Oh and it shouldn't be a 3.7 vs 3.8 thing because things were fine last week on these same runs!)

[bluetech]

It looks like the conftest is not getting loaded since the data_path also doesn't show up in the "available fixtures". This is probably related to annoying Windows path stuff.

I can probably reconstruct things from above but just to make things easier for me:

• What is $TEST_PATH in the example?
• What is the current directory from which pytest is invoked?
• Where is the pytest config file located (if any)?

---

Unrelated, but I'm curious: is there a reason you're doing this:

https://github.com/openmeeg/openmeeg/blob/331759d15006fbdf1ff13842baa435ae87c8ef19/wrapping/python/openmeeg/tests/conftest.py#L10

instead of making the fixture autouse=True?

[larsoner]

I can probably reconstruct things from above but just to make things easier for me:

From a new GHA run in a PR sandbox (where I could eventually test a fix branch if you want):

• $TEST_PATH is 'C:\Users\runneradmin\AppData\Local\Temp\cibw-run-iplgjl8i\cp39-win_amd64\venv-test\lib\site-packages\openmeeg'
• pwd (running in a bash script) is /c/Users/runneradmin/AppData/Local/Temp/cibw-run-iplgjl8i/cp39-win_amd64/test_cwd
• pytest config file is in C:\Users\runneradmin\AppData\Local\Temp\cibw-run-iplgjl8i\cp39-win_amd64\venv-test\lib\site-packages\openmeeg\tests\conftest.py, found by pytest --fixtures $TEST_PATH as ..\venv-test\lib\site-packages\openmeeg\tests\conftest.py

Unrelated, but I'm curious: is there a reason you're doing this:

Last year when I wrote this I didn't understand autouse=True. Thanks for the reminder, I'll remove it!

[bluetech]

@larsoner We've fixed some windows regressions, could you try again with latest main? I suspect it won't fix the issue but maybe it will.

[larsoner]

Looks like on commit b510adf just now the failure is still there 😞

[bluetech]

OK, so keeping this on the radar, thanks for checking.

[larsoner]

A couple more points of interest -- I tried to replicate this locally on a Windows machine and couldn't, despite creating a similar venv etc. I wonder if it's possible it's some permissions issue or something that's different about the GitHub runners and/or something that cibuildwheel does.

However, the problem is also (somewhat mysteriously) solved by using pytest --pyargs openmeeg rather than pytest <path-to-openmeeg-tests>.

[webknjaz]

@larsoner pro tip: whenever something is only reproducible on GHA, it's nice to SSH right into the job env using https://github.com/marketplace/actions/debugging-with-tmate.

Also, it's possible that env vars or something like that were messing with your setup or pytest defaults. I noticed, you have a combination of python -m pytest and just pytest invocations across the repo. The former auto-adds the current working dir to PYTHONPATH while the latter doesn't, for example.

I usually like to stuff most/all the settings + CLI args into a pytest.ini so that all the executions everywhere are as similar as possible. Plus, seeing that you're doing partial checkouts and similar things that aren't repeated locally (via tools like tox/nox) is another point of divergence. Personally, I prefer making an sdist with the Git repo in it (because it's a source of truth for the redistributors, for example) and unpacking that into jobs instead of cloning from Git. Then, testing becomes closer to what downstreams might do (even have an action for this: https://github.com/marketplace/actions/checkout-python-sdist).

[larsoner]

@larsoner pro tip: whenever something is only reproducible on GHA, it's nice to SSH right into the job env using https://github.com/marketplace/actions/debugging-with-tmate.

Amazing! I did not know about that. Will be a real time-saver in the future 👍

I'll see if I can run a remote git bisect, that would at least tell us what PR changed things! Wish me luck 🤞

Also, it's possible that env vars or something like that were messing with your setup or pytest defaults. I noticed, you have a combination of python -m pytest and just pytest invocations across the repo. The former auto-adds the current working dir to PYTHONPATH while the latter doesn't, for example.

At least in this case locally I used the same command-line invocation including venv creating and naming (other than using Users\larsoner as the root instead of Users\runneradmin). But who knows if I matched enough stuff the same way...

[larsoner]

Okay it seems to have to do with directory depth or filename length somehow:

$ cd /c
$ bash "D:\a\openmeeg\openmeeg/build_tools/cibw_test_command.sh" "D:\a\openmeeg\openmeeg"
...
+ pytest --tb=short -ra -m 'not slow' -vv 'C:\Users\runneradmin\AppData\Local\Temp\cibw-run-utgtdr49\cp310-win_amd64\venv-test\lib\site-packages\openmeeg'
=============================================================================== test session starts ===============================================================================
platform win32 -- Python 3.10.11, pytest-8.3.4, pluggy-1.5.0 -- C:\Users\runneradmin\AppData\Local\Temp\cibw-run-utgtdr49\cp310-win_amd64\venv-test\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-utgtdr49\cp310-win_amd64\venv-test\lib\site-packages\openmeeg
collected 13 items / 2 deselected / 11 selected                                                                                                                                    

Users\runneradmin\AppData\Local\Temp\cibw-run-utgtdr49\cp310-win_amd64\venv-test\lib\site-packages\openmeeg\tests\test_doc.py::test_doc PASSED                               [  9%]
...
Users\runneradmin\AppData\Local\Temp\cibw-run-utgtdr49\cp310-win_amd64\venv-test\lib\site-packages\openmeeg\tests\test_vector.py::test_vector PASSED                         [100%] 

but then going just one directory deeper:

$ cd Users
$ bash "D:\a\openmeeg\openmeeg/build_tools/cibw_test_command.sh" "D:\a\openmeeg\openmeeg"
...
file C:\Users\runneradmin\AppData\Local\Temp\cibw-run-utgtdr49\cp310-win_amd64\venv-test\lib\site-packages\openmeeg\tests\test_sensors.py, line 8
  def test_sensors(data_path):
E       fixture 'data_path' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

C:\Users\runneradmin\AppData\Local\Temp\cibw-run-utgtdr49\cp310-win_amd64\venv-test\lib\site-packages\openmeeg\tests\test_sensors.py:8
============================================================================= short test summary info ============================================================================= 
ERROR runneradmin/AppData/Local/Temp/cibw-run-utgtdr49/cp310-win_amd64/venv-test/Lib/site-packages/openmeeg/tests/test_geometry.py::test_make_geometry_head
...

According to git bisect the problem arose due to 385796b

I still can't figure out how to replicate it locally. I've tried different directory depths and filename lengths with no luck. And I need to stop working on this for a bit 🙂

BTW if anyone else wants to try this pip install openmeeg pytest should be enough to get started at least. You don't really need the data -- the tests will fail but will fail inside the decorator that looks for the data path, which is "failing successfully" (the error I'm reporting here is that the fixture isn't even executed/found, which is "failing badly").