"[Errno 104] Connection reset by peer" error emanating from _keepalive_sweep in connection pool

[eirikur-grid]

Today we rolled out an upgrade to from HTTPX 0.16.1 to 0.18.0. Consequently, we started getting alerts from Sentry. Here's a typical call stack from one such event:

httpcore/_exceptions.py in map_exceptions at line 12
contextlib.py in __exit__ at line 135
httpcore/_backends/asyncio.py in aclose at line 198
httpcore/_async/http11.py in aclose at line 199
httpcore/_async/connection.py in aclose at line 187
httpcore/_async/connection_pool.py in _keepalive_sweep at line 345
httpcore/_async/connection_pool.py in handle_async_request at line 203
httpx/_transports/default.py in handle_async_request at line 277
httpx/_transports/default.py in map_httpcore_exceptions at line 61

We had several long-lived httpx.AsyncClient() objects. Seeing as this error is emanating from the connection pool, we decided to change the HTTP request pattern so that we now create a new AsyncClient per request (as a context manager). That change has now been live for 30 minutes and so far we haven't seen any more occurrences of this error.

I suspect that this is a regression in the management of stale connections. It's surprising that an error during a connection sweep should bubble all the way up like this.

UPDATE - further information:
Runtime: CPython 3.9.1 using asyncio
Baked into a debian buster-slim docker image and deployed on k8s v1.17.12 on AWS EKS
httpx.AsyncClient() is constructed with no explicit arguments
httpcore version 0.13.0

[eirikur-grid]

I just came across encode/httpcore#310 so perhaps this has already been fixed.

[lovelydinosaur]

Yes looks like that to me. Want to try updating to the latest httpx and httpcore versions and let us know if you're still seeing issues?

[eirikur-grid]

Yesterday we upgraded to httpcore 0.13.3 and httpx 0.18.1 and re-enabled connection pooling. So far, so good.