Add PEP text and clarify purpose

README.md

@@ -1,6 +1,8 @@
 # About
 
-This is a sample package that uses distutils to demonstrate the new metadata feature of [PEP 561](https://www.python.org/dev/peps/pep-0561/).
+This is the main repository for tracking issues related to [PEP 561](https://www.python.org/dev/peps/pep-0561/).
+
+This repo also includes a copy of the PEP, and a sample typed package.
 
 # License
 

pep-0561.rst

@@ -0,0 +1,304 @@
+PEP: 561 
+Title: Distributing and Packaging Type Information
+Author: Ethan Smith <[email protected]>
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 09-Sep-2017
+Python-Version: 3.7
+Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017, 12-Apr-2018
+
+
+Abstract
+========
+
+PEP 484 introduced type hinting to Python, with goals of making typing
+gradual and easy to adopt. Currently, typing information must be distributed
+manually. This PEP provides a standardized means to leverage existing tooling
+to package and distribute type information with minimal work and an ordering
+for type checkers to resolve modules and collect this information for type
+checking.
+
+
+Rationale
+=========
+
+Currently, package authors wish to distribute code that has inline type
+information. Additionally, maintainers would like to distribute stub files
+to keep Python 2 compatibility while using newer annotation syntax. However,
+there is no standard method to distribute packages with type information.
+Also, if one wished to ship stub files privately the only method available
+would be via setting ``MYPYPATH`` or the equivalent to manually point to
+stubs. If the package can be released publicly, it can be added to 
+typeshed [1]_. However, this does not scale and becomes a burden on the
+maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
+of the tool using typeshed.
+
+PEP 484 has a brief section on distributing typing information. In this
+section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
+shipping stub files. However, manually adding a path to stub files for each
+third party library does not scale. The simplest approach people have taken
+is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
+checkers to fail on packages that are highly dynamic (e.g. sqlalchemy 
+and Django).
+
+
+Definition of Terms
+===================
+
+The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
+to be interpreted as described in RFC 2119.
+
+"inline" - the types are part of the runtime code using PEP 526 and 3107 
+syntax.
+
+"stubs" - files containing only type information, empty of runtime code.
+
+"Distributions" are the packaged files which are used to publish and distribute
+a release. [3]_
+
+"Module" a file containing Python runtime code or stubbed type information.
+
+"Package" a directory or directories that namespace Python modules.
+
+
+Specification
+=============
+
+There are several motivations and methods of supporting typing in a package.
+This PEP recognizes three (3) types of packages that users of typing wish to
+create:
+
+1. The package maintainer would like to add type information inline.
+
+2. The package maintainer would like to add type information via stubs.
+
+3. A third party or package maintainer would like to share stub files for
+   a package, but the maintainer does not want to include them in the source
+   of the package.
+
+This PEP aims to support these scenarios and make them simple to add to
+packaging and deployment.
+
+The two major parts of this specification are the packaging specifications
+and the resolution order for resolving module type information. The type
+checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec
+of PEP 484 [2]_.
+
+New third party stub libraries SHOULD distribute stubs via the third party
+packaging methods proposed in this PEP in place of being added to typeshed.
+Typeshed will remain in use, but if maintainers are found, third party stubs
+in typeshed MAY be split into their own package.
+
+
+Packaging Type Information
+--------------------------
+
+In order to make packaging and distributing type information as simple and
+easy as possible, packaging and distribution is done through existing
+frameworks.
+
+Package maintainers who wish to support type checking of their code MUST add
+a marker file named ``py.typed`` to their package supporting typing. This marker applies
+recursively: if a top-level package includes it, all its sub-packages MUST support
+type checking as well. To have this file installed with the package,
+maintainers can use existing packaging options such as ``package_data`` in
+distutils, shown below.
+
+Distutils option example::
+
+    setup(
+        ...,
+        package_data = {
+            'foopkg': ['py.typed'],
+        },
+        ...,
+        )
+
+For namespace packages, the ``py.typed`` file should be in the submodules of
+the namespace, to avoid conflicts and for clarity.
+
+This PEP does not support distributing typing information as part of
+module-only distributions. The code should be refactored into a package-based
+distribution and indicate that the package supports typing as described
+above.
+
+Stub Only Packages
+''''''''''''''''''
+
+For package maintainers wishing to ship stub files containing all of their
+type information, it is preferred that the ``*.pyi`` stubs are alongside the
+corresponding ``*.py`` files. However, the stubs can also be put in a separate
+package and distributed separately. Third parties can also find this method
+useful if they wish to distribute stub files. The name of the stub package
+MUST follow the scheme ``foopkg-stubs`` for type stubs for the package named
+``foopkg``. Note that for stub only packages adding a py.typed marker is not
+needed since the name ``*-stubs`` is enough to indicate it is a source of typing
+information.
+
+Third parties seeking to distribute stub files are encouraged to contact the
+maintainer of the package about distribution alongside the package. If the
+maintainer does not wish to maintain or package stub files or type information
+inline, then a third party stub only package can be created.
+
+In addition, stub-only distributions SHOULD indicate which version(s)
+of the runtime package are supported by indicating the runtime distribution's
+version(s) through normal dependency data. For example, the
+stub package ``flyingcircus-stubs`` can indicate the versions of the
+runtime ``flyingcircus`` distribution it supports through ``install_requires``
+in distutils-based tools, or the equivalent in other packaging tools. Note that
+in pip 9.0, if you update ``flyingcircus-stubs``, it will update
+``flyingcircus``. In pip 9.0, you can use the
+``--upgrade-strategy=only-if-needed`` flag. In pip 10.0 this is the default
+behavior.
+
+
+Type Checker Module Resolution Order
+------------------------------------
+
+The following is the order in which type checkers supporting this PEP SHOULD
+resolve modules containing type information:
+
+1. User code - the files the type checker is running on.
+
+2. Stubs or Python source manually put in the beginning of the path. Type
+   checkers SHOULD provide this to allow the user complete control of which
+   stubs to use, and patch broken stubs/inline types from packages.
+
+3. Stub packages - these packages SHOULD supersede any installed inline
+   package. They can be found at ``foopkg-stubs`` for package ``foopkg``.
+
+4. Inline packages - if there is nothing overriding the installed
+   package, and it opts into type checking, inline types SHOULD be used.
+
+5. Typeshed (if used) - Provides the stdlib types and several third party
+   libraries.
+
+Type checkers that check a different Python version than the version they run
+on MUST find the type information in the ``site-packages``/``dist-packages``
+of that Python version. This can be queried e.g.
+``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended
+that the type checker allow for the user to point to a particular Python
+binary, in case it is not in the path.
+
+
+Partial Stub Packages
+---------------------
+
+Many stub packages will only have part of the type interface for libraries
+completed, especially initially. For the benefit of type checking and code
+editors, packages can be "partial". This means modules not found in the stub
+package SHOULD be searched for in the corresponding runtime package.
+
+Type checkers should merge the stub package and runtime package directories.
+This can be thought of as the functional equivalent of copying the stub package
+into the same directory as the corresponding runtime package and type checking
+the combined directory structure. Thus type checkers MUST maintain the normal
+resolution order of checking ``*.pyi`` before ``*.py`` files. 
+
+Stub packages can opt into declaring themselves as partial by including
+``partial\n`` in the package's ``py.typed`` file.
+
+
+Implementation
+==============
+
+The proposed scheme of indicating support for typing is completely backwards
+compatible, and requires no modification to package tooling. A sample package
+with inline types is available [typed_pkg]_, as well as a sample package
+checker [pkg_checker]_ which reads the metadata of installed packages and
+reports on their status as either not typed, inline typed, or a stub package.
+
+The mypy type checker has an implementation of PEP 561 searching which can be
+read about in the mypy docs [4]_.
+
+
+Acknowledgements
+================
+
+This PEP would not have been possible without the ideas, feedback, and support
+of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Andrey
+Vlasovskikh, Nathaniel Smith, and Guido van Rossum.
+
+
+Version History
+===============
+
+* 2018-05-15
+    * Add partial stub package spec.
+
+* 2018-04-09
+
+    * Add reference to mypy implementation
+    * Clarify stub package priority.
+
+* 2018-02-02
+
+    * Change stub only package suffix to be -stubs not _stubs.
+    * Note that py.typed is not needed for stub only packages.
+    * Add note about pip and upgrading stub packages.
+
+* 2017-11-12
+
+    * Rewritten to use existing tooling only
+    * No need to indicate kind of type information in metadata
+    * Name of marker file changed from ``.typeinfo`` to ``py.typed``
+
+* 2017-11-10
+
+    * Specification re-written to use package metadata instead of distribution
+      metadata.
+    * Removed stub only packages and merged into third party packages spec.
+    * Removed suggestion for typecheckers to consider checking runtime versions
+    * Implementations updated to reflect PEP changes.
+
+* 2017-10-26
+
+    * Added implementation references.
+    * Added acknowledgements and version history.
+
+* 2017-10-06
+
+    * Rewritten to use .distinfo/METADATA over a distutils specific command.
+    * Clarify versioning of third party stub packages.
+
+* 2017-09-11
+
+    * Added information about current solutions and typeshed.
+    * Clarify rationale.
+
+
+References
+==========
+.. [1] Typeshed (https://github.com/python/typeshed)
+
+.. [2] PEP 484, Storing and Distributing Stub Files
+   (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)
+ 
+.. [3] PEP 426 definitions
+   (https://www.python.org/dev/peps/pep-0426/)
+
+.. [4] Example implementation in a type checker
+   (https://mypy.readthedocs.io/en/latest/installed_packages.html)
+
+.. [typed_pkg] Sample typed package
+   (https://github.com/ethanhs/sample-typed-package)
+
+.. [pkg_checker] Sample package checker
+   (https://github.com/ethanhs/check_typedpkg)
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+
+
+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End: