You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -11,61 +11,83 @@ import Exercises from './_exercises.mdx';
11
11
12
12
---
13
13
14
-
Using browser tools for developers is crucial for understanding the structure of a particular page, but it's a manual task. Let's start building our first automation, a Python program which downloads HTML code of the product listing.
14
+
Using browser tools for developers is crucial for understanding the structure of a particular page, but it's a manual task. Let's start building our first automation, a JavaScript program which downloads HTML code of the product listing.
15
15
16
-
## Starting a Python project
16
+
## Starting a Node.js project
17
17
18
-
Before we start coding, we need to set up a Python project. Let's create new directory with a virtual environment. Inside the directory and with the environment activated, we'll install the HTTPX library:
18
+
Before we start coding, we need to set up a Node.js project. Let's create new directory and let's name it `product-scraper`. Inside the directory, we'll initialize new project:
19
19
20
20
```text
21
-
$ pip install httpx
21
+
$ npm init
22
+
This utility will walk you through creating a package.json file.
22
23
...
23
-
Successfully installed ... httpx-0.0.0
24
-
```
25
-
26
-
:::tip Installing packages
27
-
28
-
Being comfortable around Python project setup and installing packages is a prerequisite of this course, but if you wouldn't say no to a recap, we recommend the [Installing Packages](https://packaging.python.org/en/latest/tutorials/installing-packages/) tutorial from the official Python Packaging User Guide.
29
24
30
-
:::
25
+
Press ^C at any time to quit.
26
+
package name: (product-scraper)
27
+
version: (1.0.0)
28
+
description: Product scraper
29
+
entry point: (index.js)
30
+
test command:
31
+
git repository:
32
+
keywords:
33
+
author:
34
+
license: (ISC)
35
+
# highlight-next-line
36
+
type: (commonjs) module
37
+
About to write to /Users/.../product-scraper/package.json:
38
+
39
+
{
40
+
"name": "product-scraper",
41
+
"version": "1.0.0",
42
+
"description": "Product scraper",
43
+
"main": "index.js",
44
+
"scripts": {
45
+
"test": "echo \"Error: no test specified\" && exit 1"
46
+
},
47
+
"author": "",
48
+
"license": "ISC",
49
+
# highlight-next-line
50
+
"type": "module"
51
+
}
52
+
```
31
53
32
-
Now let's test that all works. Inside the project directory we'll create a new file called `main.py` with the following code:
54
+
The above creates a `package.json` file with configuration of our project. While most of the values are arbitrary, it's important that the project's type is set to `module`. Now let's test that all works. Inside the project directory we'll create a new file called `index.js` with the following code:
33
55
34
-
```py
35
-
importhttpx
56
+
```js
57
+
importprocessfrom'node:process';
36
58
37
-
print("OK")
59
+
console.log(`All is OK, ${process.argv[2]}`);
38
60
```
39
61
40
-
Running it as a Python program will verify that our setup is okay and we've installed HTTPX:
62
+
Running it as a Node.js program will verify that our setup is okay and we've correctly set the type to `module`. The program takes a single word as an argument and will address us with it, so let's pass it "mate", for example:
41
63
42
64
```text
43
-
$ python main.py
44
-
OK
65
+
$ node index.js mate
66
+
All is OK, mate
45
67
```
46
68
47
69
:::info Troubleshooting
48
70
49
-
If you see errors or for any other reason cannot run the code above, it means that your environment isn't set up correctly. We're sorry, but figuring out the issue is out of scope of this course.
71
+
If you see `ReferenceError: require is not defined in ES module scope, you can use import instead`, double check that in your `package.json` the type property is set to `module`.
72
+
73
+
If you see other errors or for any other reason cannot run the code above, it means that your environment isn't set up correctly. We're sorry, but figuring out the issue is out of scope of this course.
50
74
51
75
:::
52
76
53
77
## Downloading product listing
54
78
55
-
Now onto coding! Let's change our code so it downloads HTML of the product listing instead of printing `OK`. The [documentation of the HTTPX library](https://www.python-httpx.org/) provides us with examples how to use it. Inspired by those, our code will look like this:
56
-
57
-
```py
58
-
import httpx
79
+
Now onto coding! Let's change our code so it downloads HTML of the product listing instead of printing `All is OK`. The [documentation of the Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch) provides us with examples how to use it. Inspired by those, our code will look like this:
If we run the program now, it should print the downloaded HTML:
66
88
67
89
```text
68
-
$ python main.py
90
+
$ node index.js
69
91
<!doctype html>
70
92
<html class="no-js" lang="en">
71
93
<head>
@@ -79,15 +101,15 @@ $ python main.py
79
101
</html>
80
102
```
81
103
82
-
Running `httpx.get(url)`, we made a HTTP request and received a response. It's not particularly useful yet, but it's a good start of our scraper.
104
+
Running `await fetch(url)`, we made a HTTP request and received a response. It's not particularly useful yet, but it's a good start of our scraper.
83
105
84
106
:::tip Client and server, request and response
85
107
86
108
HTTP is a network protocol powering the internet. Understanding it well is an important foundation for successful scraping, but for this course, it's enough to know just the basic flow and terminology:
87
109
88
110
- HTTP is an exchange between two participants.
89
111
- The _client_ sends a _request_ to the _server_, which replies with a _response_.
90
-
- In our case, `main.py` is the client, and the technology running at `warehouse-theme-metal.myshopify.com` replies to our request as the server.
112
+
- In our case, `index.js` is the client, and the technology running at `warehouse-theme-metal.myshopify.com` replies to our request as the server.
91
113
92
114
:::
93
115
@@ -109,28 +131,30 @@ First, let's ask for trouble. We'll change the URL in our code to a page that do
We could check the value of `response.status_code` against a list of allowed numbers, but HTTPX already provides `response.raise_for_status()`, a method that analyzes the number and raises the `httpx.HTTPError` exception if our request wasn't successful:
134
+
We could check the value of `response.status` against a list of allowed numbers, but the Fetch API already provides `response.ok`, a property which returns `false` if our request wasn't successful:
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://warehouse-theme-metal.myshopify.com/does/not/exist'
133
-
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
150
+
$ node index.js
151
+
file:///Users/.../index.js:7
152
+
throw new Error(`HTTP ${response.status}`);
153
+
^
154
+
155
+
Error: HTTP 404
156
+
at file:///Users/.../index.js:7:9
157
+
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
134
158
```
135
159
136
160
Letting our program visibly crash on error is enough for our purposes. Now, let's return to our primary goal. In the next lesson, we'll be looking for a way to extract information about products from the downloaded HTML.
Right in your Terminal or Command Prompt, you can create files by _redirecting output_ of command line programs:
176
202
177
203
```text
178
-
python main.py > products.html
204
+
node index.js > products.html
179
205
```
180
206
181
-
If you want to use Python instead, it offers several ways how to create files. The solution below uses [pathlib](https://docs.python.org/3/library/pathlib.html):
207
+
If you want to use Node.js instead, it offers several ways how to create files. The solution below uses the [Promises API](https://nodejs.org/api/fs.html#promises-api):
Download a product image, then save it on your disk as a file. While HTML is _textual_ content, images are _binary_. You may want to scan through the [HTTPX QuickStart](https://www.python-httpx.org/quickstart/) for guidance. You can use this URL pointing to an image of a TV:
227
+
Download a product image, then save it on your disk as a file. While HTML is _textual_ content, images are _binary_. You may want to scan through the [Fetch API documentation](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#reading_the_response_body) for guidance. Especially check `Response.arrayBuffer()`. You can use this URL pointing to an image of a TV:
Being comfortable around Python project setup and installing packages is a prerequisite of this course, but if you wouldn't say no to a recap, we recommend the [Installing Packages](https://packaging.python.org/en/latest/tutorials/installing-packages/) tutorial from the official Python Packaging User Guide.
44
+
45
+
:::
46
+
47
+
:::info Troubleshooting
48
+
49
+
If you see other errors or for any other reason cannot run the code above, it means that your environment isn't set up correctly. We're sorry, but figuring out the issue is out of scope of this course.
50
+
51
+
:::
52
+
-->
53
+
40
54
Now let's use it for parsing the HTML. The `BeautifulSoup` object allows us to work with the HTML elements in a structured way. As a demonstration, we'll first get the `<h1>` element, which represents the main heading of the page.
0 commit comments