Skip to content

Commit

Permalink
Introduce a timeout on page requests
Browse files Browse the repository at this point in the history
A run has been blocked for ~2 days with the following stack trace:

Thread 0x7FC44C36D000 (active): "MainThread"
    read (ssl.py:1159)
    recv_into (ssl.py:1303)
    readinto (socket.py:705)
    _read_status (http/client.py:279)
    begin (http/client.py:318)
    getresponse (http/client.py:1375)
    getresponse (urllib3/connection.py:507)
    _make_request (urllib3/connectionpool.py:538)
    urlopen (urllib3/connectionpool.py:805)
    send (requests/adapters.py:681)
    send (requests/sessions.py:703)
    request (requests/sessions.py:589)
    request (requests/api.py:59)
    get (requests/api.py:73)
    _request_page (canadiantracker/triangle.py:330)
    __iter__ (canadiantracker/triangle.py:337)
    scrape_skus (canadiantracker/scraper.py:184)
    invoke (click/core.py:782)
    invoke (click/core.py:1434)
    invoke (click/core.py:1688)
    main (click/core.py:1078)
    __call__ (click/core.py:1157)
    <module> (<string>:1)

Set a timeout and handle any exceptions by simply requesting
the next product page.

Signed-off-by: Jérémie Galarneau <[email protected]>
  • Loading branch information
jgalar committed Nov 22, 2024
1 parent 7161c81 commit d8c58fc
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion src/canadiantracker/triangle.py
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,14 @@ def __iter__(self) -> Iterator[Product]:
logger.debug(
f"Fetching listing of category {cat.full_name} (page {page}/{num_pages})"
)
response = ProductInventory._request_page(cat, level, page_number=page)
try:
response = ProductInventory._request_page(cat, level, page_number=page)
except Exception as e:
logger.warning(
f"Page request failed with exception: {e}"
)
continue

response = response.json()

if num_pages is None:
Expand Down Expand Up @@ -330,6 +337,7 @@ def _request_page(product_code: str) -> requests.Response:
return requests.get(
f"https://apim.canadiantire.ca/v1/product/api/v1/product/productFamily/{product_code}?baseStoreId=CTR&lang=en_CA&storeId=64",
headers=headers,
timeout=10
)

def __iter__(self):
Expand Down

0 comments on commit d8c58fc

Please sign in to comment.