PEP 561: When should you NOT have a py.typed file?

[geofft]

I'm working on widespread typing adoption at my workplace, at least for libraries shared across teams. The typing ecosystem has matured a lot since last time people tried, especially with the adoption of PEP 561 and mypy being much better at working with dependencies. I'm finding that almost every one of our codebases should have a py.typed file, and this is leading me to wonder - when should codebases not have one?

PEP 561 says, "Package maintainers who wish to support type checking of their code MUST add a marker file named py.typed to their package supporting typing." But it doesn't really define what "support type checking" means, and I think in practice the py.typed file is more useful for downstream users of the package who wish to type check their code than for the package's own maintainer.

In particular, even if your package has no annotations at all, it seems like having a py.typed file is at least harmless and actually of positive value: mypy (for instance) will detect what variables and function signatures are in your package, type them as Any, and allow downstream users to make progress gradually typing their code. However, without a py.typed file, mypy will raise a "module is installed, but missing library stubs or py.typed marker" error. You can suppress this, but this is a more annoying user experience for the downstream user, and if I understand correctly it will type the entire module as Any, reducing the amount of static validation that mypy can do.

So I think that packages should have py.typed files unless you specifically know you should not have one, even if you aren't interested in adding typing annotations or type-checking your own code, and I think the reasons you should not have a py.typed file are:

1. The package uses annotation syntax, but uses them for something other than PEP-484-compatible type hints (as opposed to not having annotations at all), and so you don't want a type checker trying to process the annotations. Or, as a generalization, you have some type hints but they're so inaccurate that it's better for downstream users not to use them at all (e.g., your annotations predate the no-implicit-Optional rule and you haven't had the time to fix it yet).
2. There are well-maintained type stubs somewhere else (e.g. pandas has the separate pandas-stubs project), and you want to guide users to find them instead of inferring everything as Any.
   (However, since explicit -stubs packages have a higher priority than types in the package itself, it seems like having a py.typed file is harmless if you have some other way of communicating this.)
3. The package doesn't have any annotations, but also the code is doing something that a static type checker can't handle (compiled modules, tricks with __getattr__, etc.), and so trying to infer the contents of the package would return misleading information. In this case it's preferable to ship .pyi files and add a py.typed marker, but if you don't want to write types at all, you should leave out the py.typed file.

Is this correct?

In other words, should templates for new projects include a py.typed file by default (as requested in e.g., audreyfeldroy/cookiecutter-pypackage#680) and should basically all Python packages (both public and in-house) that don't fall into one of the cases above go add a py.typed file even if they don't have type annotations?

---

On a related note, what are the rules about the contents of the py.typed file? If I'm going to add py.typed files to thousands of internal libraries that don't have type annotations yet, I'd kind of like to include some explanation of why it's okay for that file to be there and why you probably shouldn't remove it. PEP 561 doesn't say much about it, except that partial stubs "MUST include partial\n in a py.typed file." I'm not sure if "include" means the full contents must be exactly partial\n or whether it simply must be a substring. Outside of stubs (in particular, in actual packages, which is most of what I'm interested in), should I feel free to put something in the py.typed file? Should I use comment characters to avoid it being parsed in the future? (If I'm reading mypy's currrent implementation right, it doesn't care about contents for normal packages, and for stubs it checks whether the entire file .decode().strip() == "partial".)

[srittau]

Just a quick note: I think the following quote is a bit misleading: "Package maintainers who wish to support type checking of their code MUST add a marker file named py.typed to their package supporting typing." This only applies when distributing libraries. The py.typed file is then used as a marker for code using this library: "Hey, this library is typed, so use the type annotations from the library functions you are using." When checking your own code, py.typed is not necessary. This also means that for "application code", I would not distribute py.typed, only for libraries where the public API is sufficiently annotated.

[geofft]

Yeah, I should clarify my use case a bit - we have a number of internal libraries that are used from multiple applications, e.g., service clients, auth helpers, data models, shared code across a team, etc. These are the packages I specifically want to add py.typed to.

I agree that py.typed is not necessary for an application (or more precisely a package that is never going to be used as a dependency by some other package), but it's also harmless, right? So I'm inclined to just say "add py.typed unless you know it's going to be a problem."

We do occasionally see application packages turn into libraries (e.g., a command-line tool that turns out to be useful as an in-process API). Or sometimes it turns out to be easier to write integration tests in a separate package for dependency management reasons, and I'd want to be able to type-check those tests.

[hmc-cs-mdrissi]

When checking your own code, py.typed is not necessary

If your own code is single non-namespace package yes. Otherwise it gets messy. If you have two packages (A and B) developed in same repository and B imports A and A doesn't have py.typed then type checker may treat A as untyped and not use it's annotations.

With single package if it's namespace package you can have similar issue. If you namespace library my_lib with subpackages my_lib.a and my_lib.b where my_lib.b imports my_lib.a and my_lib.a doesn't have py.typed then I have seen pyright treat my_lib.a as untyped library.

In practice it's easier to just always have py.typed vs think about these edge cases.