diff --git a/README.md b/README.md index a6bc846..65407c8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # About -This is a sample package that uses distutils to demonstrate the new metadata feature of [PEP 561](https://www.python.org/dev/peps/pep-0561/). +This is the main repository for tracking issues related to [PEP 561](https://www.python.org/dev/peps/pep-0561/). + +This repo also includes a copy of the PEP, and a sample typed package. # License diff --git a/pep-0561.rst b/pep-0561.rst new file mode 100644 index 0000000..ddfd83c --- /dev/null +++ b/pep-0561.rst @@ -0,0 +1,304 @@ +PEP: 561 +Title: Distributing and Packaging Type Information +Author: Ethan Smith +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 09-Sep-2017 +Python-Version: 3.7 +Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017, 12-Apr-2018 + + +Abstract +======== + +PEP 484 introduced type hinting to Python, with goals of making typing +gradual and easy to adopt. Currently, typing information must be distributed +manually. This PEP provides a standardized means to leverage existing tooling +to package and distribute type information with minimal work and an ordering +for type checkers to resolve modules and collect this information for type +checking. + + +Rationale +========= + +Currently, package authors wish to distribute code that has inline type +information. Additionally, maintainers would like to distribute stub files +to keep Python 2 compatibility while using newer annotation syntax. However, +there is no standard method to distribute packages with type information. +Also, if one wished to ship stub files privately the only method available +would be via setting ``MYPYPATH`` or the equivalent to manually point to +stubs. If the package can be released publicly, it can be added to +typeshed [1]_. However, this does not scale and becomes a burden on the +maintainers of typeshed. In addition, it ties bug fixes in stubs to releases +of the tool using typeshed. + +PEP 484 has a brief section on distributing typing information. In this +section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for +shipping stub files. However, manually adding a path to stub files for each +third party library does not scale. The simplest approach people have taken +is to add ``site-packages`` to their ``MYPYPATH``, but this causes type +checkers to fail on packages that are highly dynamic (e.g. sqlalchemy +and Django). + + +Definition of Terms +=================== + +The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are +to be interpreted as described in RFC 2119. + +"inline" - the types are part of the runtime code using PEP 526 and 3107 +syntax. + +"stubs" - files containing only type information, empty of runtime code. + +"Distributions" are the packaged files which are used to publish and distribute +a release. [3]_ + +"Module" a file containing Python runtime code or stubbed type information. + +"Package" a directory or directories that namespace Python modules. + + +Specification +============= + +There are several motivations and methods of supporting typing in a package. +This PEP recognizes three (3) types of packages that users of typing wish to +create: + +1. The package maintainer would like to add type information inline. + +2. The package maintainer would like to add type information via stubs. + +3. A third party or package maintainer would like to share stub files for + a package, but the maintainer does not want to include them in the source + of the package. + +This PEP aims to support these scenarios and make them simple to add to +packaging and deployment. + +The two major parts of this specification are the packaging specifications +and the resolution order for resolving module type information. The type +checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec +of PEP 484 [2]_. + +New third party stub libraries SHOULD distribute stubs via the third party +packaging methods proposed in this PEP in place of being added to typeshed. +Typeshed will remain in use, but if maintainers are found, third party stubs +in typeshed MAY be split into their own package. + + +Packaging Type Information +-------------------------- + +In order to make packaging and distributing type information as simple and +easy as possible, packaging and distribution is done through existing +frameworks. + +Package maintainers who wish to support type checking of their code MUST add +a marker file named ``py.typed`` to their package supporting typing. This marker applies +recursively: if a top-level package includes it, all its sub-packages MUST support +type checking as well. To have this file installed with the package, +maintainers can use existing packaging options such as ``package_data`` in +distutils, shown below. + +Distutils option example:: + + setup( + ..., + package_data = { + 'foopkg': ['py.typed'], + }, + ..., + ) + +For namespace packages, the ``py.typed`` file should be in the submodules of +the namespace, to avoid conflicts and for clarity. + +This PEP does not support distributing typing information as part of +module-only distributions. The code should be refactored into a package-based +distribution and indicate that the package supports typing as described +above. + +Stub Only Packages +'''''''''''''''''' + +For package maintainers wishing to ship stub files containing all of their +type information, it is preferred that the ``*.pyi`` stubs are alongside the +corresponding ``*.py`` files. However, the stubs can also be put in a separate +package and distributed separately. Third parties can also find this method +useful if they wish to distribute stub files. The name of the stub package +MUST follow the scheme ``foopkg-stubs`` for type stubs for the package named +``foopkg``. Note that for stub only packages adding a py.typed marker is not +needed since the name ``*-stubs`` is enough to indicate it is a source of typing +information. + +Third parties seeking to distribute stub files are encouraged to contact the +maintainer of the package about distribution alongside the package. If the +maintainer does not wish to maintain or package stub files or type information +inline, then a third party stub only package can be created. + +In addition, stub-only distributions SHOULD indicate which version(s) +of the runtime package are supported by indicating the runtime distribution's +version(s) through normal dependency data. For example, the +stub package ``flyingcircus-stubs`` can indicate the versions of the +runtime ``flyingcircus`` distribution it supports through ``install_requires`` +in distutils-based tools, or the equivalent in other packaging tools. Note that +in pip 9.0, if you update ``flyingcircus-stubs``, it will update +``flyingcircus``. In pip 9.0, you can use the +``--upgrade-strategy=only-if-needed`` flag. In pip 10.0 this is the default +behavior. + + +Type Checker Module Resolution Order +------------------------------------ + +The following is the order in which type checkers supporting this PEP SHOULD +resolve modules containing type information: + +1. User code - the files the type checker is running on. + +2. Stubs or Python source manually put in the beginning of the path. Type + checkers SHOULD provide this to allow the user complete control of which + stubs to use, and patch broken stubs/inline types from packages. + +3. Stub packages - these packages SHOULD supersede any installed inline + package. They can be found at ``foopkg-stubs`` for package ``foopkg``. + +4. Inline packages - if there is nothing overriding the installed + package, and it opts into type checking, inline types SHOULD be used. + +5. Typeshed (if used) - Provides the stdlib types and several third party + libraries. + +Type checkers that check a different Python version than the version they run +on MUST find the type information in the ``site-packages``/``dist-packages`` +of that Python version. This can be queried e.g. +``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended +that the type checker allow for the user to point to a particular Python +binary, in case it is not in the path. + + +Partial Stub Packages +--------------------- + +Many stub packages will only have part of the type interface for libraries +completed, especially initially. For the benefit of type checking and code +editors, packages can be "partial". This means modules not found in the stub +package SHOULD be searched for in the corresponding runtime package. + +Type checkers should merge the stub package and runtime package directories. +This can be thought of as the functional equivalent of copying the stub package +into the same directory as the corresponding runtime package and type checking +the combined directory structure. Thus type checkers MUST maintain the normal +resolution order of checking ``*.pyi`` before ``*.py`` files. + +Stub packages can opt into declaring themselves as partial by including +``partial\n`` in the package's ``py.typed`` file. + + +Implementation +============== + +The proposed scheme of indicating support for typing is completely backwards +compatible, and requires no modification to package tooling. A sample package +with inline types is available [typed_pkg]_, as well as a sample package +checker [pkg_checker]_ which reads the metadata of installed packages and +reports on their status as either not typed, inline typed, or a stub package. + +The mypy type checker has an implementation of PEP 561 searching which can be +read about in the mypy docs [4]_. + + +Acknowledgements +================ + +This PEP would not have been possible without the ideas, feedback, and support +of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Andrey +Vlasovskikh, Nathaniel Smith, and Guido van Rossum. + + +Version History +=============== + +* 2018-05-15 + * Add partial stub package spec. + +* 2018-04-09 + + * Add reference to mypy implementation + * Clarify stub package priority. + +* 2018-02-02 + + * Change stub only package suffix to be -stubs not _stubs. + * Note that py.typed is not needed for stub only packages. + * Add note about pip and upgrading stub packages. + +* 2017-11-12 + + * Rewritten to use existing tooling only + * No need to indicate kind of type information in metadata + * Name of marker file changed from ``.typeinfo`` to ``py.typed`` + +* 2017-11-10 + + * Specification re-written to use package metadata instead of distribution + metadata. + * Removed stub only packages and merged into third party packages spec. + * Removed suggestion for typecheckers to consider checking runtime versions + * Implementations updated to reflect PEP changes. + +* 2017-10-26 + + * Added implementation references. + * Added acknowledgements and version history. + +* 2017-10-06 + + * Rewritten to use .distinfo/METADATA over a distutils specific command. + * Clarify versioning of third party stub packages. + +* 2017-09-11 + + * Added information about current solutions and typeshed. + * Clarify rationale. + + +References +========== +.. [1] Typeshed (https://github.com/python/typeshed) + +.. [2] PEP 484, Storing and Distributing Stub Files + (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files) + +.. [3] PEP 426 definitions + (https://www.python.org/dev/peps/pep-0426/) + +.. [4] Example implementation in a type checker + (https://mypy.readthedocs.io/en/latest/installed_packages.html) + +.. [typed_pkg] Sample typed package + (https://github.com/ethanhs/sample-typed-package) + +.. [pkg_checker] Sample package checker + (https://github.com/ethanhs/check_typedpkg) + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: