Improve robustness for HTTP/2

[tomchristie]

We've got a handful of open issues around HTTP/2 robustness.

I'm going to collate these into a single issue, to help us keep the issue easily reviewable & improve prioritisation.

• "Server disconnected" on HTTP/2 connection pool with big keepalive_expiry. #2983
• httpx + http2 raises a lot of exceptions when parallelized with ThreadPoolExecuton #3002
• HTTP 2.0 Throws KeyError rather than the internal exception thrown in the thread #3072

[justin-jeffery-ipf]

I just had a customer experience the following issue. No async is being used by our product at this time.

When I try my code with a try/except and log the process, I see the same error occurring the same; however, the stream id is reset back to 1, and the data is properly processed.

Traceback (most recent call last):
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 72, in map_httpcore_exceptions
    yield
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\connection_pool.py", line 216, in handle_request
    raise exc from None
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 185, in handle_request
    raise exc
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 148, in handle_request
    status, headers = self._receive_response(
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 292, in _receive_response
    event = self._receive_stream_event(request, stream_id)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 333, in _receive_stream_event
    self._receive_events(request, stream_id)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 352, in _receive_events
    raise RemoteProtocolError(self._connection_terminated)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:1999, additional_data:None>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Data\python_test\ipf_stream_test.py", line 25, in <module>
    get_interfaces_from_json(intent['sn'])
  File "C:\Data\python_test\ipf_stream_test.py", line 16, in get_interfaces_from_json
    json_log = json.loads(device.get_json())
  File "C:\Data\python_test\venv\lib\site-packages\ipfabric\models\device.py", line 124, in get_json
    return raise_for_status(self.client.get(f"/snapshots/{_}/devices/{encoded_sn}/json")).text
  File "C:\Data\python_test\venv\lib\site-packages\ipfabric\api.py", line 142, in get
    return self._client.get(url, *args, params=params, **kwargs)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 1066, in get
    return self.request(
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 837, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 926, in send
    response = self._send_handling_auth(
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "C:\Users\jdro\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:1999, additional_data:None>

It seems that once the stream ID hits 1999 it fails and then resets back to 1.

[mvasilkov]

Can confirm, consistently getting httpx.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:1999, additional_data:None> with http2.

[tomchristie]

Initial step in resolving that is reproducibility.

Does anyone have time to run @jonathanslenders example here #2983 and confirm that reproduces the issue?

[sc-google]

@tomchristie I've reproduced the issue from 2983 and it occurs whether connecting synchronously or asynchronously. When connecting asynchronously it doesn't seem to matter which async library is used the issue continues to occur.

I've also tried to reproduce the issue using different libraries like request and curl and they do not run into the issue reported from 2983.

[ppsmilesi]

Hey ! For what it worth, and if I can help push this issue forward, I have enabled http2 last week in production on our workers and we get a lot of various errors:

• RemoteProtocolError: <StreamReset stream_id:19, error_code:1, remote_reset:True>
• RemoteProtocolError <ConnectionTerminated error_code:0, last_stream_id:19999, additional_data:None>
• LocalProtocolError 103
• RemoteProtocolError: Server disconnected without sending a response.
• LocalProtocolError: Max outbound streams is 100, 100 open

I am going to revert but if you need some traces, I could get you some of them.
Thanks for all the work !

[TheGreatAlgo]

We have a ConnectionTerminated issue as well. Here are the reproducible steps #3549 (comment)

[Faolain]

Bumping the example #3549 (comment) which easily reproduces httpx.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:1999, additional_data:None>, is there any suggested approach or idea as to where this problem might be @tomchristie ? Is this in httpcore itself?

[bigcat88]

We have the same problem. And we don't really want to switch from httpx to another library just because of this one bug.

Can I ask if someone provides a PR with a fix, can we expect it to be merged if it fixes this(ConnectionTerminated) correctly?

httpx has a large community, I hope someone will have time to make a PR.