imported TestCase does not run setUpClass on second encounter

[samdoolin]

the issue

I often write a base TestCase with some generic tests, then use this as a parent class for more derived TestCases.

When importing the base TestCase into other files pytest discovers it and runs it (as does unittest). However pytest does not process setUpClass on the base TestCase the second time that it is encountered.

an example

src/
    __init__.py
    base.py
    test_foo.py
    test_bar.py

# base.py

import unittest


class BaseTestCase(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        if cls is BaseTestCase:
            raise unittest.SkipTest("skipping base class")
        else:
            cls.foo = True

    def test_called(self):
        # should only reach here in subclass
        self.assertTrue(self.foo)

# test_foo.py

from .base import BaseTestCase


class TestFoo(BaseTestCase):
    pass

# test_bar.py

from .base import BaseTestCase


class TestBar(BaseTestCase):
    pass

expected behaviour

2 skips, 2 passes - like unittest

$ python -m unittest -v
skipped 'skipping base class'
test_called (src.test_bar.TestBar) ... ok
skipped 'skipping base class'
test_called (src.test_foo.TestFoo) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK (skipped=2)

actual behaviour

$ pytest -v src
...
src/test_bar.py::BaseTestCase::test_called SKIPPED                                                               [ 25%]
src/test_bar.py::TestBar::test_called PASSED                                                                     [ 50%]
src/test_foo.py::BaseTestCase::test_called FAILED                                                                [ 75%]
src/test_foo.py::TestFoo::test_called PASSED                                                                     [100%]

======================================================= FAILURES =======================================================
_______________________________________________ BaseTestCase.test_called _______________________________________________

self = <src.base.BaseTestCase testMethod=test_called>

    def test_called(self):
        # should only reach here in subclass
>       self.assertTrue(self.foo)
E       AttributeError: 'BaseTestCase' object has no attribute 'foo'

src/base.py:15: AttributeError
=============================================== short test summary info ================================================
FAILED src/test_foo.py::BaseTestCase::test_called - AttributeError: 'BaseTestCase' object has no attribute 'foo'
======================================== 1 failed, 2 passed, 1 skipped in 0.07s ========================================

The fail is because the second time that BaseTestCase is encountered setUpClass is not run (which would skip it).

short term fix

a quick fix is to use from . import base to avoid a reference to BaseTestCase

environment

$ python -V
Python 3.7.6
$ pip freeze
attrs==19.3.0
certifi==2020.4.5.1
importlib-metadata==1.6.0
more-itertools==8.2.0
packaging==20.1
pluggy==0.13.0
py==1.8.1
pyparsing==2.4.7
pytest==5.4.1
six==1.14.0
wcwidth==0.1.9
zipp==3.1.0

[symonk]

relates/duplicate of #7092

[samdoolin]

Yes, similar to #7092. I did not check the behaviour of tearDownClass.

[SalmonMode]

They are definitely related, but are directly conflicting requests. This one wants imported test classes to be treated as a entirely newly defined test classes, as if they had been defined in the module they were imported to. #7092 wants imported classes to be ignored by the test runner.

[samdoolin]

The pytest docs claim that setUpClass/tearDownClass are supported features of unittest, so I was expecting the same behaviour.

[samdoolin]

Ignoring the imported BaseTestCase would solve the problem, but diverge from the behaviour of unittest.

[symonk]

@bluetech can you comment on what you would expect the correct behaviour from pytest here to be?

[harishkrao]

Listening in on this conversation since I am working on #7092 and would like to understand the overall expected behavior around these two issues.

[pbatko]

Ignoring the imported BaseTestCase would solve the problem, but diverge from the behaviour of unittest.

If pytest must 100% replicate unittest's semantics then it's obvious what must be done.
If it doesn't then the simpler solution from user perspective is one from #7092. That is unless there is a valid use case of running imported test classes at import sites. But even the OP gave an example where it is prevented with an exception:

raise unittest.SkipTest("skipping base class")

[Zac-HD]

If pytest must 100% replicate unittest's semantics then it's obvious what must be done.

We want pytest to be a drop-in replacement for the unittest runner, so we'll need to replicate that behaviour.

Thanks to everyone who has contributed to the discussion of this problem!