Camoufox integration PoC

[B4nan]

Create a PoC of crawlee project using camoufox. Can be a template initially, before we agree on the next steps.

It should be able to run the python script to generate browser launch options and feed that to the camoufox as part of our code.

In the long run, we can consider having CamoufoxCrawler or a camoufox template with a guide.

[janbuchar]

Are you suggesting running Camoufox from JS? It's just a hunch, but that might be tricky.

[B4nan]

Yes, we already discussed how to approach this today with @barjin and @Pijukatel, it should be actually quite simple. But we'll need to depend on running one python script under the hood for now to generate the launch options.

[Pijukatel]

Summary of the discussion and some links:

Regarding the plan for gradual implementation, it can have following steps:

• First version of POC can be fully with JS
  • Getting the camoufox binary. Either run camoufox fetch in docker image build or rewrite the code to Javascript and fetch binary on demand. It gets binary from here.
  • Use pre-generated fixed start up option for Camoufox (options generated in Python and saved as hardcoded string)
  • start Camoufox via Playwright playwright_.firefox.launch(**PRE_GENERATED_OPTIONS) and make sure that executablePath options points to the Camoufox binary
• Next version draft can generate options for Camoufox dynamically by calling the Python function launch_options and dumping result to json file. Input arguments for such function must be properly populated or rely on default. To generate custom fingerprint input argument you can use FingerprintGenerator.generate
• Next version can rewrite argument preparation utility functions from Python to Javascript and use our Javascript implementation of FingerprintGenerator.

Here is reference of how Camoufox is started through Playwright in Python: https://github.com/daijro/camoufox/blob/main/pythonlib/camoufox/async_api.py#L95
Here is another JS implementation example showing how to use Camoufox in JS: https://github.com/DemonMartin/node-camoufox