Encode spaces in query parameter with %20 instead of +

[chbndrhnns]

Hi,

I came across a web server today which wants percent encoding for spaces in the query parameter.
Httpx encodes them with the ‘+’ sign which is not wrong at all.
Now I am wondering what would be the least intrusive/most elegant way to have httpx do the percent encoding for me.

It seems that Python internals [1] are capable of doing that since 2015.

What do you think?

bye,
Jo

[1] python/cpython#58074

[lovelydinosaur]

Can you provide a more specific example? What behaviour do you see with different http clients?

[chbndrhnns]

wget uses percent encoding:

 wget -v "http://test.com/query=SUM(qty) as quantity"
--2022-12-30 12:02:45--  http://test.com/query=SUM(qty)%20as%20quantity

curl rejects spaces:

❯ curl -vv "http://test.com/query=SUM(qty) as quantity"
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL

Postman sends percent encoding:

GET http://test.com/?query=SUM(qty)%20as%20quantity

[lovelydinosaur]

Right. Both + and %20 are valid here. It looks to me like + is more spec-correct, but %20 is used more in practise

We're doing the same as requests which uses +...

>>> r = requests.get("https://www.example.com", params={"a": "b c"})
>>> r.request.url
'https://www.example.com/?a=b+c'

And...

>>> httpx.URL("https://www.example.com", params={"a": "b c"})
URL('https://www.example.com?a=b+c')

However using https://www.example.com/?a=b c in the address bar in various browsers...

Chrome, Safari, and Firefox all switch this out to https://www.example.com/?a=b%20c.
(Firefox displays "https://www.example.com/?a=b c" but I can see from the "page info" the actual URL it requested.)

So... we could probably switch to %20, and it might be a marginally better choice. (?)

[chbndrhnns]

This would be awesome. Making it configurable would so be possible but maybe also confusing.
Should I turn this topic into an issue?

[lovelydinosaur]

I came across a web server today which wants percent encoding for spaces in the query parameter.

Is there a public URL for this?

Should I turn this topic into an issue?

I'm probably? okay with this escalating this. We're not doing anything wrong here, but having behaviour that's closer to browser behaviour seems like a marginal improvement to me, even if it's a slight divergence from requests.

Implementation-wise I don't think we'd want to make any changes here until after #2252

[chbndrhnns]

I don't have a public URL at the moment but it's the v2/summary endpoint of the cloudkitty module which is part of Openstack.
That's a Flask application but I don't know the internals of the parsing and validation logic for incoming requests.

[nschloe]

Has anything ever come out of this? (I need this for a particular server which doesn't consume + in URLS.) Right now, I'm doing

urllib.parse.urlencode({"a": "b c"}, quote_via=urllib.parse.quote)

which gives

'a=b%20c'

[chbndrhnns]

I do not remember how I worked around the issue. The issue still exists IMO.

[michealroberts]

I've just pinged @tomchristie on usage, I can't see usage guidelines in the documentation and can't quite see how to use from the code base, hopefully Tom can advise on guidelines ...

[michealroberts]

However, @chbndrhnns @nschloe the workaround seems to be to fall back to manually encoding the URL, so rather than doing something this:

params = {"line_1": parts[0], "line_2": parts[1], "line_3": parts[2]}

response = self._client.get(url="/mount/ping", params=params)

... do this instead:

url = f"{self._client.base_url}/mount/ping?{
    urlencode(
        {'line1': parts[0], 'line2': parts[1], 'line3': parts[2]},
        quote_via=quote,
        safe='',
    )
}"

response = self._client.get(url)

response.raise_for_status()