Enable PDF scraping through the Docker API

[SohamKukreti]

Summary

Depends on #2130 - needs pypdf in the image and the client's scraping_strategy honored on /crawl/stream.

Fixes the remaining blocker for PDF scraping through the Docker API (#2127). Even with PR: #2130, every PDF crawl still failed at browser navigation with Page.goto: Download is starting, because the server always used the pooled Playwright crawler and headless Chromium can't render PDFs inline.

When a request's scraping_strategy is PDFContentScrapingStrategy, both crawl handlers now build a per-request AsyncWebCrawler(crawler_strategy=PDFCrawlerStrategy()) - the exact pairing the library documents - instead of borrowing a pooled browser. PDFCrawlerStrategy starts no browser (it returns a placeholder and the scraping strategy downloads/parses the PDF itself), so a per-request instance is cheap and is closed after use.

This is gated on the scraping strategy, mirroring the library, rather than sniffing the URL or Content-Type.

Also included

• Hooks guard: a hooks block combined with the PDF strategy now returns a clean 400 instead of AttributeError: 'PDFCrawlerStrategy' object has no attribute 'set_hook' → 500.
• Datetime serialization: PDF metadata datetimes are stringified so JSONResponse can serialize /crawl results (gated to PDF requests, so normal crawls pay no cost).
• SSRF protection: PDFContentScrapingStrategy downloads the PDF itself via requests, which previously followed redirects blindly. It now follows redirects manually (cap 10) and, when the server passes its validate_url_destination in as a url_validator, vets the download URL and every redirect hop before it is fetched — closing a redirect-to-internal-address (e.g. cloud metadata) SSRF vector. The url_validator param defaults to None, so library users are unaffected.

List of files changed and why

• deploy/docker/api.py — route PDF-strategy requests to PDFCrawlerStrategy, guard hooks, close the per-request crawler, sanitize datetimes, wire the SSRF validator.
• crawl4ai/processors/pdf/__init__.py — optional url_validator hook and manual per-hop redirect handling in PDFContentScrapingStrategy._get_pdf_path; MAX_PDF_DOWNLOAD_REDIRECTS constant.
• tests/test_docker_pdf_crawler_pairing.py — handler-level routing/hooks/pool tests and library-level redirect/validator tests.

How Has This Been Tested?

Image built from develop + #2130 + this branch:

• PDF works on both endpoints: arxiv, pdf.js, pdfobject on /crawl and /crawl/stream.
• SSRF blocked: a redirect to 169.254.169.254 is vetted and rejected before fetch (verified with the real validate_url_destination + the redirect loop); a direct internal seed is still 400 at the seed layer.
• No regression: plain HTML, explicit LXMLWebScrapingStrategy, multi-URL, streaming, screenshot, /md, and hooks-on-HTML all behave as before; repeated PDF crawls leave no lingering crawlers in the pool.
• tests/test_docker_pdf_crawler_pairing.py (6 tests) pass; each fix verified to fail with its change reverted.
• Full regression suite: 293 passed (one pre-existing unrelated transformers-missing failure).

Note for after #2130 merges

Its test_stream_handler_preserves_requested_scraping_strategy mocks crawler_pool.get_crawler, which PDF requests no longer call (they build a PDFCrawlerStrategy crawler directly). The test uses a PDF strategy, so after this merge the mock is never hit and its arun_many assertion fails on await_args is None. The fix is small — spy on api.AsyncWebCrawler (or otherwise intercept the PDF-branch crawler) instead of mocking the pool. Happy to include the adjustment on rebase.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[ntohidi]

Thanks @SohamKukreti — reviewed alongside #2130 and verified both locally. Gating on the scraping strategy is the right trigger.

Merging #2130 first. This branch doesn't contain it, so it needs a rebase. Two things for that pass:

1. The test conflict you flagged is real. Stacked on current develop, your six tests pass and only this one breaks:

tests/test_issue_2127_docker_pdf.py::test_stream_handler_preserves_requested_scraping_strategy
AttributeError: 'NoneType' object has no attribute 'kwargs'

2. The SSRF guard misses the crawler_configs path. That's a public per-URL field on /crawl (schemas.py:11 → server.py:907), but is_pdf_crawl only inspects the top-level config and the config_list loop never sets url_validator. A PDF strategy sent that way still does the unvalidated download. Not a regression — just so the claim holds everywhere the strategy is reachable. Setting the validator inside that loop should cover it.

Minor, not blocking: the stream handler's except blocks still call release_crawler unconditionally, so a PDF crawler isn't closed on error paths. Harmless today, but inconsistent with the finally blocks you fixed.

[SohamKukreti]

@ntohidi I have rebased my commits and made all the changes you mentioned.
Thanks!

[ntohidi]

Verified 2341507 locally — all three addressed.

Rebase is in (branch contains #2130) and test_stream_handler_preserves_requested_scraping_strategy passes now; spying on AsyncWebCrawler instead of the pool is the right fix, since a PDF crawl never goes through the pool. 8/8 across both PDF test files.

On the SSRF gap I checked the behaviour rather than the diff: sending a PDF strategy via crawler_configs and inspecting what reaches arun_many, url_validator is None on 4138ada and validate_url_destination on 2341507. Closed.

_dispose_crawler is nicer than what I asked for — one helper across the finally and all three except blocks, and I confirmed a PDF crawler gets close() while a pooled one is released and never closed.

One follow-up, not blocking: nothing in the suite covers the per-URL crawler_configs path, so that guard is unprotected against future refactors. I'll send a regression test for it separately.

Merging. Thanks!