Ability to provide a custom doctest checker

[mgedmin]

What's the problem this feature will solve?

I have a pre-existing test suite that uses doctests with a custom checker class that does some regex replacements to normalize filenames in doctest output, to make my test suite pass under Windows.

I'm having a hard time trying to switch to pytest as a test runner because I was unable to find a way of providing a custom doctest checker.

This is not the only project I've worked with that uses doctests and a regexp-based output normalizers to deal with platform (and sometimes Python version) differences; https://github.com/zopefoundation/ is full of these (although they don't currently have any plans of migrating to pytest).

Describe the solution you'd like

I think a workable interface would be to provide a fixture named doctest_checker:

import doctest
import pytest

class Checker(doctest.OutputChecker):
    def check_output(self, want, got, optionflags):
        want = re.sub(..., ..., want)
        return doctest.OutputChecker.check_output(self, want, got, optionflags)


@pytest.fixture(scope='session')
def doctest_checker():
    return Checker()

I've seen this pattern (overriding a named fixture) used to provide project-specific customizations for pytest test suites that use celery (https://docs.celeryq.dev/en/stable/userguide/testing.html#session-scope), but I'm not absolutely set on it. If it's unworkable, or if there's a better way, let's use that instead.

Alternative Solutions

I've tried looking at the pytest-doctestplus plugin, but I found no way of providing a custom doctest checker either.

Additional context

[tobiasdiez]

Agree, this would be nice!

As a workaround, you can specify you own DoctestModule that uses you own checker, see eg sagemath/sage#36981

[mgedmin]

I see a SageDoctestModule in conftest.py, which seems like a useful example. One thing worries me: that conftest.py imports a bunch of stuff from the private _pytest package, which seems fragile, unless I pin pytest version and do extra maintenance work.

[ev-br]

One possible solution is to reuse scipy-doctest package, which has a way to swap a Checker class:
https://github.com/scipy/scipy_doctest?tab=readme-ov-file#alternative-checkers