Issues a warning when a function is not collected as a test case just because it uses @pytest.fixture (#12989)

[dongfangtianyu]

close #12989 .

• Include documentation when adding new features.
• Include new tests or update existing tests when applicable.
• Allow maintainers to push and squash when merging my commits.
• Create a new changelog file in the changelog folder

A function is declared as a test case using the convention of the test prefix in its name ， or as a fixture using the pytest.fixture decorator.

When both are used simultaneously, it is not possible to have both a test case and a fixture.

So a warning is issued to remind the user: no test case, only a fixture.

Example

import pytest

@pytest.fixture
def test_function():
    pass

(venv) C:\Users\administrator\pytest\demo>pytest
============================= test session starts ==============================
platform win32 -- Python 3.12.0, pytest-8.4.0.dev262+gf160963c4.d20241211, pluggy-1.5.0
rootdir: C:\Users\administrator\pytest\demo
configfile: pytest.ini
plugins: hypothesis-6.111.0, xdist-3.6.1
collected 0 items                                                               

=============================== warnings summary =============================== 
test_aa.py:1
  C:\Users\administrator\pytest\demo\test_aa.py:1: PytestCollectionWarning: ca
nnot collect test function 'test_function',because it used the '@pytest.fixture' than becomes a fixture (from: test_aa.py)
    import warnings

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================== 1 warning in 0.26s ==============================

[dongfangtianyu]

Probably should add an ini config that allows old projects to ignore this warning?

[jakkdl]

I'm probably in favor of introducing a warning, I'm not sure what fixture requires being named test_; and if so they can suppress the warning. But it might be a better fit for https://pypi.org/project/flake8-pytest-style/

[jakkdl]

having to add an entry to filterwarnings is not great. What's the reason we need to add it?

[jakkdl]

I don't think adding this section is net positive, if you think https://docs.pytest.org/en/stable/how-to/fixtures.html#how-to-fixtures doesn't explain well what a fixture is then the intro section should be improved.

[nicoddemus]

I agree.

[jakkdl]

Suggested change

	f"cannot collect test function {name!r},"
	f"because it used the '@pytest.fixture' than becomes a fixture "
	f"(from: {self.nodeid})"
	f"failed to collect test function {name!r},"
	f"because it uses '@pytest.fixture'. A function cannot simultaneously "
	f"be both a test and a fixture.\n"
	f"If you want to use this as a fixture, rename it to not start with "
	f'"test_" to suppress this warning\n'
	f"If you want to test the fixture, write a separate test that "
	f"uses it.\n"
	f"(from: {self.nodeid})"

Maybe something like that? Also I'm not sure we need both the name and the nodeid?

[jakkdl]

when we're warning with an f-string I think it's good if this checks the entire warning string.
I also think it's good form to use result.assert_outcomes(warnings=1) here to make sure we're not outputting multiple warnings or something.

[nicoddemus]

I agree.

[nicoddemus]

Suggested change

	Issues a warning when a function is not collected as a test case just because it uses :py:func:`pytest.fixture`.
	This helps beginners distinguish fixtures from tests; experienced users can ignore the warning via config.
	A warning is now issued when a fixture uses the same name convention as a test function, for example a fixture named `test_user`.
	This helps preventing beginners that expect anything named `test_` to be considered a test.

However I'm not convinced this warning is really necessary... besides being noisy for a common pattern (test_user is a good example of a fixture which provides a "test user", a "user for testing"), even if somebody confuses it is not the end of the world, because being a fixture, it will usually be called anyway by another test function (and if not then the user did not understand anything related to fixtures and functions and the warning won't help much anyway). It would make more sense if the situation was that the code were not being called at all (so silently being ignored), but this is not the case here.

I'm -0 on this:

• Noisy for a common and valid pattern.
• Even if a beginner confuses the two, the downside is not so severe because the fixture will be called anyway.

[jakkdl]

silently-being-ignored sounds very tricky to catch, especially once you factor in partial runs with -k etc. That's for codecov to catch, although beginners probably wouldn't have that configured.

A lint rule does sound more suitable for this

[jakkdl]

I've been convinced that pytest isn't the right place to catch this, so with three people voting against (if weakly) I'm closing this