Move fullDomFetcher to Playwright

[LVerneyEC]

Hi,

Here is a proposal for a rewriting of the full DOM fetcher, moving it to Playwright instead of Puppeteer.

This edited browser also has support for HTTP/HTTPS proxy (e.g. corporate proxy) and behavior can be adjusted by two environment variables:

• PLAYWRIGHT_NO_SANDBOX to disable all the sandboxing in Chrome (required for running in Docker, depending on the Docker setup).
• PLAYWRIGHT_NO_HEADLESS to run it in headful mode (sometimes useful for debugging purposes)

This is using patchright wrapper around Playwright, which adds several patches for obvious Playwright detection mechanisms. Similar to the previously used puppeteer-extra-plugin-stealth.

Best,

[LVerneyEC]

Just flagging that this is an explicit pin at the moment due to Kaliiiiiiiiii-Vinyzu/patchright#58. Issue is closed but the fix is not yet part of the latest release. This version should be adjusted after review, prior to merging.

[LVerneyEC]

I noticed the tests are failing due to linting and commit/changelog issues. I'll fix these, but happy to have a first high-level review first to ensure this is useful and worth merging and fix everything at once afterwards :)

[MattiSG]

Thanks @LVerneyEC for this contribution! Fully agree with a first high-level overview before ironing out details :)
The intervention seems minimal. Do you have examples of cases that were blocked with the previous implementation and are unblocked with that switch? 🙂

[LVerneyEC]

Do you have examples of cases that were blocked with the previous implementation and are unblocked with that switch? 🙂

Not so much. I have another PR to come for the htmlOnlyFetcher, for which this increases widely coverage.

Here, the main benefit is to move away from puppeteer-extra-stealth which is unmaintained for a couple years: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth.

Also, more high-level updates such as supporting corporate proxies and offering the ability to run headful for debugging purposes.

[Ndpnt]

Hi @LVerneyEC,

I've conducted a series of benchmark tests to evaluate the potential benefits of switching from Puppeteer to Playwright. Below are the detailed results:

Browser Automation Tool	Run #	Total Failures	403 Errors	Navigation Timeouts	Selector Timeouts	404 Errors	Duration
Puppeteer	1	57	48	9	0	0	6m 8s
	2	47	36	10	0	1	6m 22s
	3	30	18	11	0	1	5m 50s
	4	31	19	11	0	1	5m 37s
	5	31	29	1	0	0	5m 54s
Playwright	1	76	59	0	17	0	3m 9s
	2	75	59	0	16	0	2m 46s
	3	72	59	0	13	0	2m 47s
	4	76	59	0	17	0	2m 54s
	5	69	59	0	10	0	2m 59s

Observations:

• Playwright is faster than Puppeteer
• Playwright shows more consistent 403 error counts
• Puppeteer shows more variation in error types and counts
• Puppeteer is less frequently blocked than Playwright

Based on these benchmark results, I do not recommend switching to Playwright at this time. Even if it has faster execution times, its higher failure rates and blocking issues is a blocking point for me.

Regarding the other points mentioned:

• It seems that Puppeteer natively supports HTTP proxies through environment variables
• Puppeteer supports non-headless mode through the headless: false parameter in the launch function
• While recommended to keep enabled, Puppeteer allows sandbox disabling using the --no-sandbox option

Have I missed any key points in my analysis, and do you still see reasons to switch to Playwright despite these results?

[LVerneyEC]

Would you have more details the benchmark and the results? I am a bit surprised about the 403 and selector errors, since it does not really match my experience so far.