Skip to content

Update FAQ data handling section for /crawl endpoint #30811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kathayl wants to merge 1 commit into
base: production
Choose a base branch
from
+4 −2
Open

Update FAQ data handling section for /crawl endpoint #30811

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions src/content/docs/browser-run/faq.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,12 +185,14 @@ Yes. If your webpage or PDF requires a font that is not pre-installed, you can l

### Does Cloudflare store or retain the HTML content I submit for rendering?

No. Cloudflare processes content ephemerally and does not retain customer-submitted HTML or generated output (such as PDFs or screenshots) beyond what is required to perform the rendering operation. Once the response is returned, the content is immediately discarded from the rendering environment.
For [Quick Actions](/browser-run/quick-actions/) (except the [`/crawl` endpoint](/browser-run/quick-actions/crawl-endpoint/)), [Puppeteer](/browser-run/puppeteer/), [Playwright](/browser-run/playwright/), and [CDP](/browser-run/cdp/), Cloudflare processes content ephemerally and does not retain customer-submitted HTML or generated output (such as PDFs or screenshots) beyond what is required to perform the rendering operation. Once the response is returned, the content is immediately discarded from the rendering environment.

This applies to all integration methods, including [Quick Actions](/browser-run/quick-actions/), [Puppeteer](/browser-run/puppeteer/), [Playwright](/browser-run/playwright/), and [CDP](/browser-run/cdp/).
The [`/crawl` endpoint](/browser-run/quick-actions/crawl-endpoint/) is an exception. Because crawl jobs run asynchronously, job results (including crawled page content in HTML, Markdown, or JSON format) are stored for 14 days after the job completes, after which the data is deleted. Crawl jobs have a maximum run time of seven days.

### Is there any temporary caching of submitted content?

For [Quick Actions](/browser-run/quick-actions/), generated content is cached by default for five seconds (configurable up to one day via the `cacheTTL` parameter, or set to `0` to disable caching). This cache protects against repeated requests for the same URL by the same account. Customer-submitted HTML content itself is not cached.

For [Puppeteer](/browser-run/puppeteer/), [Playwright](/browser-run/playwright/), and [CDP](/browser-run/cdp/), no caching is used. Content exists only in memory for the duration of the rendering operation and is discarded immediately after the response is returned.

For the [`/crawl` endpoint](/browser-run/quick-actions/crawl-endpoint/), crawled content can be cached in R2 via the `maxAge` parameter (default is 86,400 seconds, configurable up to 604,800 seconds). If a subsequent crawl job requests the same URL with matching parameters, the cached result is served instead of re-fetching from the origin. Job results are retained for 14 days after the job completes.
Loading