NCIt import issues

[jsstevenson]

Some users (@korikuzma) have reported timeout-like issues when importing NCIt. We should resolve this.

[korikuzma]

╰─$ python3 -m disease.cli --sources ncit                                                     130 ↵
***Using Disease Database Endpoint: http://localhost:8000***

Deleting NCIt...
Deleted NCIt in 0.03680 seconds.

Loading NCIt...
* Owlready2 * Warning: optimized Cython parser module 'owlready2_optimized' is not available, defaulting to slower Python implementation
Traceback (most recent call last):
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 1395, in getresponse
    response.begin()
  File ".pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.9/lib/python3.11/http/client.py", line 286, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.9/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.9/lib/python3.11/ssl.py", line 1314, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.9/lib/python3.11/ssl.py", line 1166, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "disease-normalization/venv/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 469, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "disease-normalization/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='ncithesaurus.nci.nih.gov', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "disease-normalization/src/disease/cli.py", line 305, in <module>
    update_db()
  File "disease-normalization/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/src/disease/cli.py", line 301, in update_db
    _update_sources(sources_to_update, db, update_merged, from_local)
  File "disease-normalization/src/disease/cli.py", line 124, in _update_sources
    _load_source(n, db, delete_time, processed_ids, from_local)
  File "disease-normalization/src/disease/cli.py", line 181, in _load_source
    processed_ids += source.perform_etl(use_existing=from_local)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/src/disease/etl/base.py", line 65, in perform_etl
    self._extract_data(use_existing)
  File "disease-normalization/src/disease/etl/base.py", line 80, in _extract_data
    self._data_file, self._version = self._data_source.get_latest(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/wags_tails/base_source.py", line 105, in get_latest
    latest_version = self._get_latest_version()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/wags_tails/ncit.py", line 24, in _get_latest_version
    r = requests.get(
        ^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "disease-normalization/venv/lib/python3.11/site-packages/requests/adapters.py", line 713, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='ncithesaurus.nci.nih.gov', port=443): Read timed out. (read timeout=30)

[korikuzma]

It's funny that as soon as I sent that ncit is back up

[jsstevenson]

@korikuzma that being said, I think we should be catching these on the wags tails side: GenomicMedLab/wags-tails#34

[korikuzma]

@jsstevenson yes, I was wondering if there was a way to fall back to local data in these cases