Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Introduce a timeout on page requests
A run has been blocked for ~2 days with the following stack trace: Thread 0x7FC44C36D000 (active): "MainThread" read (ssl.py:1159) recv_into (ssl.py:1303) readinto (socket.py:705) _read_status (http/client.py:279) begin (http/client.py:318) getresponse (http/client.py:1375) getresponse (urllib3/connection.py:507) _make_request (urllib3/connectionpool.py:538) urlopen (urllib3/connectionpool.py:805) send (requests/adapters.py:681) send (requests/sessions.py:703) request (requests/sessions.py:589) request (requests/api.py:59) get (requests/api.py:73) _request_page (canadiantracker/triangle.py:330) __iter__ (canadiantracker/triangle.py:337) scrape_skus (canadiantracker/scraper.py:184) invoke (click/core.py:782) invoke (click/core.py:1434) invoke (click/core.py:1688) main (click/core.py:1078) __call__ (click/core.py:1157) <module> (<string>:1) Set a timeout and handle any exceptions by simply requesting the next product page. Signed-off-by: Jérémie Galarneau <[email protected]>
- Loading branch information