Closed
Description
At the moment, whenever there is change to typeshed, we test all stdlib and third-party packages in the same testing environment, using the --custom-typeshed-dir
so that all packages can see each other. There are several practical problems with that:
- We can't test the
requires
fields for correctness; all packages are always available. - We can't have packages that depend on external packages, see for example Allow non-types dependencies #5768, Remove cryptography (depends on #5952) #5618,
paramiko
: Change dependencies ontypes-cryptography
, allow using upstream type hints #5847, and the discussion in Allow non-types dependencies #5769 (all related tocryptography
). (Except if we'd just install all non-types dependencies for all package into our venv.) - It doesn't scale if we increase the number of third-party packages in typeshed.
- Ideas like per-package configuration is not possible at the moment, see for example mypy tests: Allow per distribution strictness options #1526.
My suggestion: Only run the tests for the third-party stubs that have actually changed, and run each test in a separate environment. This environment only has the requirements from METADATA.toml
installed. This fixes the problems above and also means:
- The stubs will not be able to use features from stdlib that are not yet in the latest type checker distributions.
- Changes that depend on changes in another third-party stub need to wait until that change has been released. (Happens every three hours, so it shouldn't be too much of a problem.)
- Running the tests for all stubs will be slower, since each package needs its own venv installation.