Increasing pip's & PyPI's metadata strictness

[brainwane]

(Followup to discussion during packaging minisummit at PyCon North America in May 2019.)

Conda has a metadata solver. Pip and PyPI already or will soon know cases where package metadata is incorrect; how firmly/strictly should metadata correctness be enforced?

There's general agreement that strictness should be increased; the question is: how quickly and to what extent?

Issues

1. Metadata (largely dependency) can be incorrect initially, or may become incorrect in the future as new packages are released. This can be in wheels or in the source distribution.
   a. Agreement among PyPI working group to better enforce manylinux (Donald Stufft, Dustin Ingram, EWDIII) -- see "Run auditwheel on new manylinux uploads, reject if it fails #5420" Run auditwheel on new manylinux uploads, reject if it fails pypi/warehouse#5420
   b. EWDIII - There is no technical barrier for PyPI updating its metadata. There is a non-starter on updating the package/changing the metadata.
   c. Also possibility for staged releases. This would allow composition and checks before release. (Nick + Ernest)
   d. Can metadata can be corrected by producing a new wheel or post-release? Likely not by uploading a wheel.
   e. Ability to yank packages (will not install for interval-based version specifications, but would still be available to install under exact/equality pinning) per PEP 592. New simple API change to support yanking (a la Ruby gems)
   f. Metadata should not be able to differentiate between artefact and PyPI
   • The artifact metadata is canonical; and, the metadata exposed by PyPI should never diverge
2. Non-standards-compliant wheels tagged manylinux-*
3. Can pip stop (if an install is going to be?) breaking existing requirements
   a. will/may require the solver

Action Items:

1. PyPI stricter on upload, write a rollout plan
   a. Chris Wilcox said "I am going to start on a validator in pypa/validator that can be leveraged at twine/setuptools/warehouse" -- has started it -- also see twine check should guard against things not accepted by PyPI like version format twine#430 twine check should guard against things not accepted by PyPI like version format endless timeouts #430
   b. Hard fail on invalid markup with explicit description type pypi/warehouse#3285 Warehouse to start hard failing package uploads on invalid markup with explicit description type
2. Determine the behaviour of wheel build numbers
3. Simple API for yanking; underway
4. Finish pip solver
5. Warning about lack of python_requires
   a. Or could the spec/setuptools be updated to fail on this
   b. Also, can we fail when there's missing author, author_email, URL. Currently warnings on setup. (chris wilcox)
   c. For packages where no restrictions on Python version are desired, a “python_requires==*” would be satisfactory
   d. also see WIP: Add metadata validation setuptools#1562
6. Could we explore banning old upload clients from PyPI?
   a. Yes; support needs to be added; an issue needs to be created
7. General soft-fail support [if you opt in to compliance, we’ll block your upload; otherwise we will send you warnings] -- requires Draft release feature on main archive to allow testing a release before it goes live pypi/warehouse#726

This is meant as as a tracking issue covering the various TODOs necessary to plumb this through the parts of the toolchain.

[brainwane]

I linked to this on Discourse to give people a heads-up.

Some questions:

Determine the behaviour of wheel build numbers

Who needs to do this? This will affect Twine & Warehouse in particular, right?

Warning about lack of python_requires

Would this be in setuptools? @pganssle @jaraco? @crwilcox I see your name in the notes here - anything you could add here would be great.

[pganssle]

Would this be in setuptools? @pganssle @jaraco? @crwilcox I see your name in the notes here - anything you could add here would be great.

We could do it in setuptools, but it's more important to do it in warehouse, see pypi/warehouse#3889, since ideally the end goal is that everyone is properly annotating what versions of Python they support (or at least making an affirmative choice to put in something like '*.*'. A warning in setuptools will probably help some people, but more likely it will be largely ignored if it's seen at all.

I think one problem with the "PyPI raises a warning" approach is that IIUC twine has no mechanism to display any such warning. @di and @dstufft would know better than me whether adding such a capability is desirable for this or other reasons.

[crwilcox]

I have a branch started on this and intend it to be a part of packaging so it can be used from multiple sources. I think starting by using it in warehouse would make sense.

[dstufft]

The upload API does not support a warning mechanism, only error. Arguably it's better to send a warning via email though? Or at least one could make the argument that it is. Lots of people publish in an automated fashion and won't ever see a command line warning anyways.

[pradyunsg]

Warnings from PyPI via email make a lot of sense to me.

It's definitely a better option than adding a warnings mechanism that twine then exposes to the user.

We should also use something (like packaging) which can also be used across projects to do the metadata validation.

[ssbarnea]

Not really related to packaging itself but I wonder if PYPI could start archiving obsolete/unmaintained packages from the main index. PYPI UX could be improved if we would have a curated index with currently maintained packages. At this moment even if you search for a "foo" package, you will get a very poor search result as the default listing (relevance) does not display the last release date.

Maybe we could use the metadata compliance as a way to filter the package indexes, motivating people to migrate?

Regarding twine upload warnings, I am even more drastic: default to error and allow a temporary bypass option. Sadly 99% of people not even read the warnings, so make them error. The only trick is to include a link to ticket where people can see how to fix it and also comment.

[dholth]

I've released a couple of wheels with a build number. Every layer removed from the source tends to want its own version number; RPM has epochs. Alternatively you could make 7 corrections by uploading a py30-none-any wheel and then incrementing the tag all the way to py38-none-any.
@ssbarnea I don't think pypi can curate. Is someone maintaining an awesome-python website with a list of great stuff?
How will strict metadata benefit the authors and not just the consumers of a package, and not just in their role as consumers of other packages?

[dstufft]

Not really related to packaging itself but I wonder if PYPI could start archiving obsolete/unmaintained packages from the main index. PYPI UX could be improved if we would have a curated index with currently maintained packages. At this moment even if you search for a "foo" package, you will get a very poor search result as the default listing (relevance) does not display the last release date.

The biggest problem with any sort of system that tries to determine if something is obsolete/unmaintained or not... is how do you actually determine if something is obsolete and/or unmaintained and ensure that you don't get false positives for software that is just "done" and just doesn't need any further updates?

[brainwane]

@ssbarnea I'd like to keep this issue focused on the metadata strictness issue. For more on archiving unmaintained projects or excluding them from search or otherwise making frequently-maintained packages easier to find, you might want to follow up in pypi/warehouse#4004 , pypi/warehouse#1388 , pypi/warehouse#4319 , pypi/warehouse#4021 , or pypi/warehouse#1971 . Thanks for your ideas!

[brainwane]

@dstufft is "Disallow runs of special characters in project names" pypi/warehouse#469 or "Clean database of UNKNOWN and validates against it" pypi/warehouse#69 part of what's necessary, or part of what we want to do, in this increasing-metadata-strictness work?

[brainwane]

@crwilcox perhaps you'd like to take a look at pypi/warehouse#194 where people discuss what automated checks, including on metadata, they'd like performed on uploads to PyPI.