WSGITransport PATH_INFO changes UTF-8 bytes and breaks URL reconstruction

[adenzhou1350]

Reproduced on current master (b5addb64) with Python 3.12.13 on Windows and httpcore 1.0.9. No server or third-party WSGI framework is needed:

import wsgiref.util
import httpx

def app(environ, start_response):
    uri = wsgiref.util.request_uri(environ)
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [uri.encode("ascii")]

with httpx.Client(transport=httpx.WSGITransport(app=app)) as client:
    for path in ("/café", "/中"):
        try:
            print(client.get(f"http://testserver{path}").text)
        except UnicodeEncodeError as exc:
            print(type(exc).__name__ + ": " + str(exc))

Actual output:

http://testserver/caf%E9
UnicodeEncodeError: 'latin-1' codec can't encode character '\u4e2d' in position 1: ordinal not in range(256)

Expected output:

http://testserver/caf%C3%A9
http://testserver/%E4%B8%AD

WSGITransport currently sets PATH_INFO to request.url.path, which is decoded Unicode. PEP 3333 instead requires metadata strings to preserve bytes through Latin-1. That distinction changes the bytes of é and makes the standard-library URL reconstruction fail for 中.

This is related to #1365/#1391, but unquoting alone doesn't preserve that WSGI byte representation. Taking the raw path without its query, percent-decoding to bytes, then decoding with Latin-1 fixes these cases without changing HTTPX's public URL.path behavior. It also preserves non-UTF-8 escaped bytes such as %FF rather than replacing them.

I have a small local patch with 10 regression cases covering Unicode, escaped Unicode, non-UTF-8 bytes, encoded slashes, double escapes and query separation. The original fails 5 of those cases; the patch passes all 22 WSGI tests and all 136 tests in the combined WSGI/ASGI/URL suites. Ruff also passes. Would you be open to a PR for this?

[lunar-me]

Yes, this is a confirmed bug in WSGITransport.

The WSGI spec (PEP 3333) requires PATH_INFO to be a native string with the URL-decoded path. httpx currently assigns it from request.url.path, which is URL-decoded, so far so good, but only for ASCII-safe paths. The real problem is that the WSGI transport never uses the raw request target at all; it reconstructs the WSGI environ from URL.path/URL.query, and on Python 3 the decoded string is then encoded by the WSGI server/app boundary using environ["PATH_INFO"].encode("latin-1") for URL reconstruction. When the path contains UTF-8 bytes that were percent-encoded, the decoded string round-trips through latin-1 and produces the wrong bytes, breaking URL reconstruction.

The fix is for WSGITransport to expose the original percent-encoded path, or at least to avoid the double-decode/encode round-trip. The adjacent ASGITransport already does the right thing: it passes request.url.raw_path.split(b"?")[0] as raw_path. WSGITransport has no equivalent handling and simply assigns request.url.path to PATH_INFO.

[BenkiNew]

Independently reproduced on Linux / Python 3.12.14 at master b5addb64f0161ff6bfe94c124ef76f6a1fba5254.

I checked eight paths, including literal and escaped UTF-8, %FF, encoded slashes, double escapes, and a separate query string. The original code produced wrong bytes in three cases and raised UnicodeEncodeError in three more. With unquote_to_bytes(request.url.raw_path.split(b"?", 1)[0]).decode("latin-1") for PATH_INFO, all eight produced the expected path bytes; the query was preserved separately.

One clarification to the previous reply: PATH_INFO should not retain the percent-encoded spelling. In this representation /a%2Fb correctly becomes /a/b, whereas /a%252Fb becomes /a%2Fb; it is decoded once into bytes and then represented through Latin-1.

I also checked an adjacent boundary: with script_name="/café", wsgiref.util.request_uri still yields http://testserver/caf%E9/child after the PATH_INFO change. Passing the already byte-preserving WSGI string ("/café") yields http://testserver/caf%C3%A9/child. That could merit documenting what encoding the separate script_name option expects; it does not invalidate the PATH_INFO fix.

These are focused in-memory checks, not a full test-suite run or approval of your unpublished patch.