Doctests in __main__ are ignored

[jaraco]

Prior to pytest 7, pytest would exercise doctests in __main__. #8949 changed that expectation (and for good reason).

In my use-case, I always put a __name__ == '__main__' check in my entry points, including __main__ and sometimes supply functions in that module that I'd like to have doctested. There are several ways I rely on doctests:

• Test explicit behaviors in functions of the module.
• Even if there are no doctests at all, doctesting a module performs syntax validation on it, ensuring that invalid syntax will be caught by the test suite, that the module is safe to import (doesn't execute on import), and that it has its dependencies met.
• A doctest can perform certain important assertions on the module, such as the presence or absence of key attributes or functions.

Most if not all of the benefits described in In defense of doctests apply to __main__ as much as any other package module.

I've been working under the assumption that __main__ modules were doctested in all of my projects and only today learned that they're not. I disagree that these modules should be special-cased just because some projects are unwilling to make them import-safe. Surely, pytest would consider it invalid if a user were to complain about any other module in the package failing to import because it has undesirable import-time behaviors, e.g.

# pkg/foo.py
raise SystemExit(1)

Yes, it's true that __main__.py has some special semantics, but so also does __init__.py, but that's not justification for excluding __init__ from doctests.

I note that there's a related issue #11716, which proposes to give users more control to exclude things for doctests. If implemented, that approach could potentially satisfy the need for some projects to exclude __main__.

Even prior to addressing that issue, it would have been possible for users to exclude __main__ modules by adding it to their pytest_collect_ignore (or is it pytest_collectignore; I can never remember).

Can we reconsider supporting __main__ for doctests?

[RonnyPfannschmidt]

This should be a opt in, not everyone is implementing the dunder main check

So dunder main submodules are actually scripts

Personally I recommend to not implement any logic in them besides import and calling main

[webknjaz]

FWIW, I'm also of the opinion that __main__.py should only import things and act as an entry-point. Just like console_scripts. And the actual implementation would be in private modules.

[jaraco]

I'd still like pytest to catch a syntax error in the file:

if __name__ == '__main__'
    run()

or a coding error in the entry point:

if __name__ == __main__:
    run()

Even if all this file does is act as an entry point, it's worthwhile to examine it. Doctests provides this functionality.

Consider packages where the primary purpose is the command, it's a little awkward to ask users to create a separate module for a short run script:

# cli.py
import sys

def run(*args=sys.argv[1:]):
    print("hello world")

# __main__.py
from .cli import run

if __name__ == '__main__':
    run()

That's a lot of boilerplate for a hello world example and a layer of indirection I'd like not to enforce on every application. While I agree it's good practice, even best practice, to move most of the functionality out of the "script", it also often makes sense for there to be a non-trivial amount of logic in the script.

Consider, for example, the script jaraco.develop.add-github-secrets. Like __main__, it's used only as an entry point. Similar to __main__, it's not meant to be imported except as a script. Yet, there are still concerns in that script that I don't feel like exporting elsewhere; they're best associated with this particular script. Should pytest be ignoring these files too, because they could potentially contain unprotected execution? In my opinion, A test system shouldn't be imposing preferences on code organization.

I'd be satisfied by an opt-back-in if you really think that's necessary, but in my opinion, the least surprising approach is to be consistent about how modules are discovered and tested and have users opt-in to the exceptional behaviors, rather than have an inconsistency that a user can opt out of.