ConnectionTerminated error_code:0 not gracefully retried

[injust]

#2112 (reply in thread):

The previous fix just made it so you can catch this error and retry the request yourself without it failing again.

encode/httpcore#683 is needed to automatically retry the request.

Based on the above, I saw that encode/httpcore#683 was superseded by encode/httpcore#733, so I expected this error to be gracefully retried by httpx. However, I've been running into this error ~daily, and have worked around it by retrying the httpx request with tenacity.

Unfortunately, I have no information about the server and I am unable to share the code for the client in its entirety. Please let me know what information I can provide that might be helpful, e.g. debug logging.

I am making a POST request to the same URL every ~10 seconds, until it inevitably fails. This is approximately how I am calling httpx:

import httpx
from httpx._config import DEFAULT_LIMITS

httpx_client: httpx.AsyncClient = httpx.AsyncClient(
    headers={"Accept-Encoding": "gzip", "User-Agent": "xxx"},
    http2=True,
    limits=httpx.Limits(
        max_connections=DEFAULT_LIMITS.max_connections,
        max_keepalive_connections=DEFAULT_LIMITS.max_keepalive_connections,
        keepalive_expiry=60,
    )
)

del httpx_client.headers["Accept"]

await httpx_client.post("https://example.com/foo/", data={"foo": "bar"})

h2==4.2.0
httpcore==1.0.7
httpx==0.28.1

Traceback

Traceback (most recent call last):

  File "~/foo/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "~/foo/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
                 │    │     │                    └ <Request [b'POST']>
                 │    │     └ <function AsyncConnectionPool.handle_async_request at 0x111e68720>
                 │    └ <AsyncConnectionPool [Requests: 0 active, 0 queued | Connections: 0 active, 0 idle]>
                 └ <httpx.AsyncHTTPTransport object at 0x111c8e120>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
                     │          └ <function AsyncHTTPConnection.handle_async_request at 0x111e4f880>
                     └ <AsyncHTTPConnection ['https://example.com:443', HTTP/2, CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
                 │    │           │                    └ <Request [b'POST']>
                 │    │           └ <function AsyncHTTP2Connection.handle_async_request at 0x111e6a5c0>
                 │    └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
                 └ <AsyncHTTPConnection ['https://example.com:443', HTTP/2, CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 183, in handle_async_request
    raise exc
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 146, in handle_async_request
    status, headers = await self._receive_response(
                            │    └ <function AsyncHTTP2Connection._receive_response at 0x111e6a980>
                            └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 290, in _receive_response
    event = await self._receive_stream_event(request, stream_id)
                  │    │                     │        └ 647
                  │    │                     └ <Request [b'POST']>
                  │    └ <function AsyncHTTP2Connection._receive_stream_event at 0x111e6aac0>
                  └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 329, in _receive_stream_event
    await self._receive_events(request, stream_id)
          │    │               │        └ 647
          │    │               └ <Request [b'POST']>
          │    └ <function AsyncHTTP2Connection._receive_events at 0x111e6ab60>
          └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 348, in _receive_events
    raise RemoteProtocolError(self._connection_terminated)
          │                   │    └ <ConnectionTerminated error_code:0, last_stream_id:647, additional_data:None>
          │                   └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
          └ <class 'httpcore.RemoteProtocolError'>

httpcore.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:647, additional_data:None>

Originally posted by @injust in #2112 (reply in thread)

[TheGreatAlgo]

We are running into the same thing!

We made a reproducable test for you easy enough.

https://github.com/dClimate/py-hamt/tree/example-connection-error

should be able to just do

uv sync

then run this

uv run pytest tests/test_performance_tests.py::test_benchmark_hamt_store -s

And you will get this error

This is because on our server we have it rejecting requests after 1000.

My understanding is that it should handle it more gracefully and restart a new connection when the server says it will cut you off.

Important code is in store_httpx.py with where we enable http2=True

        """Get or create a client for the current event loop."""
        loop: asyncio.AbstractEventLoop = asyncio.get_running_loop()
        try:
            return self._client_per_loop[loop]
        except KeyError:
            # Create a new client
            client = httpx.AsyncClient(
                timeout=60.0,
                headers=self._default_headers,
                auth=self._default_auth,
                limits=httpx.Limits(max_connections=64, max_keepalive_connections=32),
                http2=True,
            )
            self._client_per_loop[loop] = client
            return client```