fix(py-client): Treat object keys as literal strings#548
Merged
Conversation
This comment has been minimized.
This comment has been minimized.
lcian
changed the title
lcian
force-pushed
the
lcian/fix/py-client-literal-key-encoding
branch
from
c1edfc2 to
f547282
Compare
jan-auer
approved these changes
Jul 9, 2026
Percent-encode request paths in `_make_url`/`_make_multipart_url` with `presign.encode_path` instead of relying on urllib3's implicit encoding. urllib3 treats a literal `%XX` in a key as an existing escape and leaves it on the wire, so `put`/`get`/`delete` and pre-signed URLs disagreed on which object a `%XX` key referred to. Encoding ourselves treats the key as a literal string everywhere (a literal `%` becomes `%25`), matching pre-signed URLs and the Rust client (whose `url` crate already escapes `%`). Re-enables the `looks%20encoded` encoding corner case and adds a test asserting `put` stores under the literal key echoed in its response body.
`encode_path`/`encode_query` are general on-the-wire URL encoding, not specific to pre-signed URLs. Move them (and the `_PATH_SAFE`/`_QUERY_SAFE` sets) from `presign` to `utils` so `_make_url`/`_make_multipart_url` and the pre-signing code call one shared, neutrally-located helper instead of the request path depending on the `presign` module.
lcian
force-pushed
the
lcian/fix/py-client-literal-key-encoding
branch
from
f547282 to
308068d
Compare
This reverts commit 1674d58.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #544. Makes the Python client encode request paths itself (via
presign.encode_path) instead of relying on urllib3's implicit encoding, so a key is always treated as a literal string.The bug
urllib3's encoder treats a literal
%XX(percent + two hex digits) in a key as an existing escape and leaves it on the wire unchanged. Pre-signed URLs (which useurllib.parse.quote) instead escape the%to%25. So for a key likelooks%20encoded,put/getand the pre-signed GET disagreed on which object was addressed:put/get(old urllib3)quote)…/looks%20encoded…/looks%2520encodedlooks encoded(space!)looks%20encoded(literal)The fix
_make_url/_make_multipart_urlnow percent-encode the assembled path withpresign.encode_path. Our output is a subset of urllib3's safe set and contains only valid%XX, so urllib3 transmits it verbatim — meaning wire bytes only change for keys containing a literal%XX; every other key is byte-identical to before. The whole client now treats keys as opaque literal strings, consistent with pre-signed URLs.Rust client
Verified not affected: the Rust client builds URLs via the
urlcrate'spath_segments_mut().push(), which already escapes a literal%to%25(looks%20encoded→looks%2520encoded). It already treats keys as literals, so no change is needed.Tests
looks%20encodedcase intest_presigned_get_encoding_corner_cases(previously commented out because it diverged).test_put_stores_under_literal_key, which asserts the object lands under the literal key by checking the key the server echoes in theputresponse body (would belooks encodedunder the old behavior).Verification
ruff format/ruff check/mypy: cleancargo build,cargo fmt --check,cargo clippy --workspace --all-targets --all-features: cleancargo test -p objectstore-client --all-features: passedNote for reviewers
This is a wire-visible behavior change for the (pathological) case of keys containing a literal
%XX: such objects are now stored/fetched under the literal key rather than urllib3's percent-decoded interpretation. Objects previously written under the old interpretation would live at a different path.Close FS-416