Skip to content

Commit

Permalink
Add playwright execution mode (#36)
Browse files Browse the repository at this point in the history
* #

* add-playwright

* add playwright in requirements

* Update README.md

* fix navigation and wait options

* Update playwright_browser_manager.py

* fix wait_options in pyp

* Update playwright_browser_manager.py

* Update pyppeteer_browser_manager.py

* Update playwright_browser_manager.py

* Removing unused imports

* Update playwright_browser_manager.py

* Update playwright_browser_manager.py

* Update playwright_browser_manager.py

* increment version

* fix version

* fix requirements

* delete "no value was provided"

* rename function

* Update README.md

* refactoring

* del __del__

* "" to None

* Update playwright_browser_manager.py
  • Loading branch information
AndrewKorzh authored Aug 23, 2024
1 parent c22a53c commit b6acfb4
Show file tree
Hide file tree
Showing 6 changed files with 430 additions and 48 deletions.
13 changes: 9 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,15 @@ DOWNLOADER_MIDDLEWARES = {

PUPPETEER_SERVICE_URL = 'http://localhost:3000'

#To run locally (without scrapy-puppeteer-service started), you need to enable the setting:
PUPPETEER_LOCAL = True
# To change the execution method, you must add the corresponding setting:
EXECUTION_METHOD = "Puppeteer"
```
For local execution it is also necessary to install Chromium for Pyppeteer.
Available methods: `Puppeteer`, `Pyppeteer`, `Playwright`

The `Pyppeteer` and `Playwright` methods do not require a running service. They use the pyppeteer and playwright libraries for Python to interact with the browser. Actions such as `CustomJsAction`, `RecaptchaSolver`, and `Har` are not available when using these methods.

To use the `Pyppeteer` or `Playwright` methods you need to install Chromium.


## Configuration

Expand Down Expand Up @@ -84,7 +89,7 @@ Here is the list of available actions:
- `Scroll(selector, wait_options)` - scroll page
- `Screenshot(options)` - take screenshot
- `Har()` - to get the HAR file, pass the `har_recording=True` argument to `PuppeteerRequest` at the start of execution.
- `FormAction(input_mapping, submit_button)` - to fill out and submit forms on page.
- `FillForm(input_mapping, submit_button)` - to fill out and submit forms on page.
- `RecaptchaSolver(solve_recaptcha)` - find or solve recaptcha on page
- `CustomJsAction(js_function)` - evaluate JS function on page

Expand Down
3 changes: 2 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
scrapy>=2.6
pyppeteer
syncer
bs4
bs4
playwright
Loading

0 comments on commit b6acfb4

Please sign in to comment.