fix(pdf): carry cookies across manual redirect hops - #2159
Merged
Conversation
#2150 switched the PDF download to allow_redirects=False and followed hops by hand so url_validator can vet each one before it is fetched. Each hop used a bare requests.get(), which starts with an empty cookie jar, so a host that sets a cookie and then redirects never gets its own cookie back and answers 403. That is the normal shape for gated and CDN-signed PDFs, and it worked before #2150 because allow_redirects=True carried cookies implicitly. Use one requests.Session for the chain. The SSRF guarantee is unchanged: hops are still validated before the fetch and still not auto-followed. Verified by mutation: reverting session.get to requests.get fails the new test and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Regression from #2150, found while verifying that PR locally. Affects the library, not just the Docker server.
What broke
#2150 set
allow_redirects=Falseand follows redirect hops by hand, sourl_validatorcan vet each hop before it is fetched. Correct goal — but each hop calls a barerequests.get(), and every one of those starts with an empty cookie jar.So a host that sets a cookie and then redirects never gets its own cookie back, and answers 403. That is the ordinary shape for gated and CDN-signed PDFs. It worked before #2150 because
allow_redirects=Truecarried cookies for us.Reproduced against a local server (302 +
Set-Cookie, then a target that requires it):fc6ec13(before #2150)37ee60a(after #2150)RuntimeError: ... 403 Client Error: ForbiddenFix
One
requests.Sessionfor the chain instead of a freshrequests.getper hop.The SSRF guarantee is untouched: hops are still validated before the fetch, still not auto-followed, still capped. A Session only adds a cookie jar.
Testing
tests/test_docker_pdf_crawler_pairing.py, reusing the existingredirect_serverfixture with a/gatedroute.session.gettorequests.getfails that test and nothing else.deploy/docker/tests/test_security_*.py(the CI suite): 319 passed, 1 xfailed.tests/regression: 320 passed, 1 failed —test_soft_404_filters_probes, which hits a live site and fails identically onfc6ec13, so it is unrelated.cc @SohamKukreti