Adjust path_to_url et al. to produce the same results on Python 3.14+ #13423

hroncok · 2025-06-12T22:06:05Z

See python/cpython#125974 and #13138 (comment)

This makes the tests pass on Fedora with Python 3.14.0b2 (except test_get_index_content_directory_append_index, which segfaults python/cpython#135448)

hroncok · 2025-06-12T22:41:08Z

src/pip/_internal/models/link.py

-    return urllib.request.pathname2url(urllib.request.url2pathname(part))
+    ret = urllib.request.pathname2url(urllib.request.url2pathname(part))
+    if sys.version_info >= (3, 14):
+        ret = ret.removeprefix("//")


I am not proud of this :(

It wasn't obvious to me how else to do this either 🙁

Is there something in cpython we can reference here so people reading this later knows what’s going on?

I can try digging a reference to the change.

Ideally, if we could find something that explained why cpython changed what they did, and why the new result is "correct" and why pip needs something different from the "correct" answer, that would be best (as it would give us a much better basis for informed decisions if this code ever needs to change again).

Unfortunately, I get the impression that there's no real "correct" answer here, and the cpython change was "because it's more consistent with (something or other that pip maybe doesn't even care about)". If so, then documenting what precisely pip is using to base its idea of what "the url for a pathname" is, would be better than nothing.

Worst case scenario would be that there's simply no standard for how to convert a pathname to a URL in general, and it's all just a mess of guesswork and hacks 🙁

What form should the input to url2pathname take? It seems like it should take a file: URL with the initial file: removed.

I'll list the URLs with file: prefixes so it's a little easier to grok:

file:foo - relative path

file:///foo - absolute path (POSIX), path beginning with single slash (Windows)

file:///c:/foo - DOS drive path (Windows)

file://server/foo - UNC path (Windows)

But that's only part of the story - does it accept all the various variants of file: URLs (things like file:/etc/hosts, and the various ways of encoding Windows drives)?

In 3.14 it should do, yeah. Acceptable slash-prefix variants:

file:/foo - absolute path (POSIX), path beginning with single slash (Windows)

file:c:/foo, file:/c:/foo - DOS drive path (Windows)

file:////server/foo, file://///server/foo - UNC path (Windows)

Windows DOS paths also support a pipe (|) rather than a colon after the drive letter.

The authority can also vary for non-UNC paths.

Is the "correct" form accepted in all Python versions, or was there an actual bug in <=3.13 which mean that we can't just pass correct forms and be done with it? If the latter is the case, what was the bug in 3.13? Because converting correct form to "acceptable-to-3.13" form seems like a better fix than hacking up the output.

It should be accepted in the latest versions of 3.12 and 3.13 (see list of backported changes), but in older versions they're mishandled.

Is pathname2url always correct, or were there bugs in that in 3.13? If so, did those bugs take the form of returning an incorrect URL? If so, what's incorrect about the output, because we'd need to know that to "fix up" the return value.

The major 3.14-only change here is that pathname2url('/tmp') now returns '///tmp', whereas in 3.13 it returns '/tmp'. Otherwise it's a similar story to url2pathname above - the latest bugfix releases of 3.13 and 3.12 are fine, but older maintenance releases (and anything before 3.12) is buggy

As a very basic example, I cannot see why url2pathname(pathname2url(path)) shouldn't be the identity (or at worst, normalise the path). I can understand that pathname2url(url2pathname(url)) might be a little more complicated, as there are many ways to encode a path (certainly on Windows) as a URL, so what you get back might be different to what you passed. But even then, I'd expect it to the original URL and the returned one to represent the same path.

Can you share examples of where this isn't the case? Paths should roundtrip

Can you share examples of where this isn't the case? Paths should roundtrip

Sorry, I'm getting confused. The pip code is

ret = urllib.request.pathname2url(urllib.request.url2pathname(part))

which I misread as path -> URL -> path. So I couldn't see why that wasn't (effectively) a null operation. But it's not, it's the other way around, making it (in effect) a URL normalisation operation.

If that's the case, my understanding of what's going on has definitely been helped by this discussion, because I couldn't have said that before¹. But I'm now left with another question, which is why, if we're returning a normalised URL (with the file: prefix omitted), we even care about the extra 2 slashes. I feel like we're either doing some incorrect parsing on the returned URL, or we're (incorrectly) treating it as a path rather than a schemeless URL somewhere 🙁

Or, to put this another way, I think that removing the // is simply patching over a more significant bug in our handling of the return value from this function, and we should be looking for that bug, rather than trying to undo the changes CPython made.

Of course, there's also a "practicality beats purity" question here - if this means pip doesn't work on Python 3.14, we might need to make the expedient choice in order to get a fix into pip 25.2, which is the release that will end up in Python 3.14, and will be the current release when 3.14 is released.

Footnotes

We should definitely include a comment above that code saying that it's normalising the provided URL. ↩

But I'm now left with another question, which is why, if we're returning a normalised URL (with the file: prefix omitted), we even care about the extra 2 slashes.

I think our motivating care is that the tests no longer pass. Pip is not apparently broken on the Python 3.14 betas.

So it's a question of modifying the function to keep the behavior consistent across Python versions, or modifying the tests, and the later requires significantly more confidence about what the right behavior is.

notatallshaw · 2025-06-21T23:31:15Z

With no other feedback, are you planning to remove draft status from this and add a small news item?

I tested this locally and it appears to be good, but because it's slightly changing the behavior of brittle path functions I think there should be a small bug fix note.

hroncok · 2025-06-22T11:07:23Z

Can do.

hroncok · 2025-06-30T09:14:46Z

See https://discuss.python.org/t/pathname2url-changes-in-python-3-14-breaking-pip-tests/97091

See python/cpython#125974 and pypa#13138 (comment) and https://discuss.python.org/t/pathname2url-changes-in-python-3-14-breaking-pip-tests/97091

hroncok · 2025-06-30T09:20:43Z

Anyway. I added a news entry and a link to the upstream discussion to a comment. Marked as non-draft. That's about as much as I have the capacity to do.

ichard26 · 2025-07-05T19:25:23Z

With my RM hat on, barring any major issues (I still need to review this), I plan on including this in the upcoming 25.2 release regardless of whether CPython replies or not. I'd like to include it in the release so downstream can use/package pip for the Python 3.14 betas.

barneygale · 2025-07-19T17:48:14Z

Here's a list of url2pathname() and pathname2url() changes in the last year or so.

Affecting only `url2pathname()`

Changes backported to 3.12 and 3.13:

GH-126212: Fix removal of slashes in file URIs on Windows python/cpython#126214
GH-126766: url2pathname(): handle empty authority section. python/cpython#126767
GH-126766: url2pathname(): handle 'localhost' authority python/cpython#127129
GH-127078: url2pathname(): handle extra slash before UNC drive in URL path python/cpython#127132

Changes in 3.14+ only:

Affecting only `pathname2url()`

Changes backported to 3.12 and 3.13:

Changes in 3.14+ only:

Affecting both functions

Changes backported to 3.12 and 3.13:

GH-85168: Use filesystem encoding when converting to/from file URIs python/cpython#126852

Changes in 3.14+ only:

barneygale · 2025-07-19T21:36:41Z

Here's an implementation of src/pip/_internal/utils/urls.py that mostly vendors the 3.14+ implementation of pathname2url() and url2pathname(). It seems to solve some test failures but not all.

import os
import string
import sys
import urllib.parse
import urllib.request

from .compat import WINDOWS


def path_to_url(path: str) -> str:
    """
    Convert a path to a file: URL.  The path will be made absolute and have
    quoted path parts.
    """
    path = os.path.abspath(path)
    if WINDOWS:
        path = path.replace('\\', '/')
    encoding = sys.getfilesystemencoding()
    errors = sys.getfilesystemencodeerrors()
    drive, root, tail = os.path.splitroot(path)
    if drive:
        # First, clean up some special forms. We are going to sacrifice the
        # additional information anyway
        if drive[:4] == '//?/':
            drive = drive[4:]
            if drive[:4].upper() == 'UNC/':
                drive = '//' + drive[4:]
        if drive[1:] == ':':
            # DOS drive specified. Add three slashes to the start, producing
            # an authority section with a zero-length authority, and a path
            # section starting with a single slash.
            drive = '///' + drive
        drive = urllib.parse.quote(drive, encoding=encoding, errors=errors, safe='/:')
    elif root:
        # Add explicitly empty authority to absolute path. If the path
        # starts with exactly one slash then this change is mostly
        # cosmetic, but if it begins with two or more slashes then this
        # avoids interpreting the path as a URL authority.
        root = '//' + root
    tail = urllib.parse.quote(tail, encoding=encoding, errors=errors)
    return 'file:' + drive + root + tail


def url_to_path(url: str) -> str:
    """
    Convert a file: URL to a path.
    """
    scheme, netloc, path = urllib.parse.urlsplit(url)[:3]
    assert scheme == "file", f"You can only turn file: urls into filenames (not {url!r})"
    if WINDOWS:
        if netloc and netloc != 'localhost':
            # e.g. file://server/share/file.txt
            path = '//' + authority + path
        elif path[:3] == '///':
            # e.g. file://///server/share/file.txt
            path = path[1:]
        else:
            if path[:1] == '/' and path[2:3] in (':', '|'):
                # Skip past extra slash before DOS drive in URL path.
                path = path[1:]
            if path[1:2] == '|':
                # Older URLs use a pipe after a drive letter
                path = path[:1] + ':' + path[2:]
        path = path.replace('/', '\\')
    elif netloc and netloc != 'localhost':
        raise ValueError(
            f"non-local file URIs are not supported on this platform: {url!r}"
        )
    encoding = sys.getfilesystemencoding()
    errors = sys.getfilesystemencodeerrors()
    return urllib.parse.unquote(path, encoding=encoding, errors=errors)

pfmoore · 2025-07-19T22:06:57Z

TBH, I don't really want to take on responsibility for maintaining our own URL handling logic if we can avoid it. We don't have the expertise to do a decent job, so if any bugs arose, we'd be in a difficult position.

barneygale · 2025-07-20T14:19:55Z

Fair enough - I'll try a different approach then!

If it's not too much trouble, could someone provide a list of relevant tests that are failing in 3.14? I'm seeing some failures that don't seem to be related to URL/path conversion.

barneygale · 2025-07-20T14:29:02Z

Perhaps I could provide a PyPI package that backports 3.14's url2pathname() and pathname2url() for older Pythons, which pip could then vendor. Would that be a good way forward?

hroncok commented Jun 12, 2025

View reviewed changes

ichard26 mentioned this pull request Jun 22, 2025

Release 25.2 (July) #13438

Open

Adjust path_to_url et al. to produce the same results on Python 3.14+

dc20f20

See python/cpython#125974 and pypa#13138 (comment) and https://discuss.python.org/t/pathname2url-changes-in-python-3-14-breaking-pip-tests/97091

hroncok force-pushed the python3.14 branch from acbe19b to dc20f20 Compare June 30, 2025 09:17

psf-chronographer bot added the bot:chronographer:provided label Jun 30, 2025

hroncok marked this pull request as ready for review June 30, 2025 09:19

ichard26 added this to the 25.2 milestone Jul 5, 2025

Adjust path_to_url et al. to produce the same results on Python 3.14+ #13423

Are you sure you want to change the base?

Adjust path_to_url et al. to produce the same results on Python 3.14+ #13423

Uh oh!

Conversation

hroncok commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barneygale Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notatallshaw commented Jun 21, 2025

Uh oh!

hroncok commented Jun 22, 2025

Uh oh!

hroncok commented Jun 30, 2025

Uh oh!

hroncok commented Jun 30, 2025

Uh oh!

ichard26 commented Jul 5, 2025

Uh oh!

barneygale commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Affecting only url2pathname()

Changes backported to 3.12 and 3.13:

Changes in 3.14+ only:

Affecting only pathname2url()

Changes backported to 3.12 and 3.13:

Changes in 3.14+ only:

Affecting both functions

Changes backported to 3.12 and 3.13:

Changes in 3.14+ only:

Uh oh!

barneygale commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pfmoore commented Jul 19, 2025

Uh oh!

barneygale commented Jul 20, 2025

Uh oh!

barneygale commented Jul 20, 2025

Uh oh!

Uh oh!

hroncok commented Jun 12, 2025 •

edited

Loading

barneygale Jul 19, 2025 •

edited

Loading

barneygale commented Jul 19, 2025 •

edited

Loading

Affecting only `url2pathname()`

Affecting only `pathname2url()`

barneygale commented Jul 19, 2025 •

edited

Loading