Allow Response to be pickled

[hannseman]

It would be nice if Response could be made picklable. This would make it play nicer with the multiprocessing module. I noticed this in celery using the prefork pool when a unhandled httpx.HTTPStatusError is raised in a task. This results in

MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: HTTPStatusError('500 Server Error: Internal Server Error for url: https://www.example.com\\nFor more information check: https://httpstatuses.com/500')>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'httpx.HTTPStatusError\'>: it\'s not the same object as httpx.HTTPStatusError")''.>

The issue here is that httpx.HTTPStatusError takes a Response instance in __init__ and this makes it unpicklable.

Trying to pickle a response directly fails like this:

Python 3.9.1 (default, Dec 11 2020, 14:22:09)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import pickle, httpx
>>> pickle.dumps(httpx.get("https://www.github.com"))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
AttributeError: Can't pickle local object 'Client._send_single_request.<locals>.on_close'

I found a similar discussion on psf/requests#1367

[simonw]

The code that closed that requests feature request is a useful clean example of how this could be implemented: https://github.com/psf/requests/pull/1733/files

[florimondmanca]

I think this is quite interesting, especially thanks to pointers from @simonw. I didn't know about the __getstate__ / __setstate__ pair.

@tomchristie Is this an example of promoting this to a proper issue?

[hannseman]

I had a look at implementing a __getstate__ and from what I can see the things that needs to be taken into consideration are:

• The response body needs to be read. Not sure how this is best handled sync vs async.
• Response.on_close should be discarded by __get_state__
• Response.extensions has the key close which contains an inner function and is thus not picklable. Maybe it's OK to pop close?

[lovelydinosaur]

Presumably we want both Request and Response to be pickle-able.

Implementation-wise we may well want to wait until after #1550 is resolved, as well as a follow-up that also removes Response(on_close=...), since both of those affect points that @hannseman has raised.

The response body needs to be read. Not sure how this is best handled sync vs async.

If a response is pickled, then we want to pickle it in the same state in which it was currently in, but the response should always be restored in a closed state.

[hannseman]

If a response is pickled, then we want to pickle it in the same state in which it was currently in, but the response should always be restored in a closed state.

Would that mean that we drop Response.stream from the pickle? Generators can't be pickled async or not.

[hannseman]

PR: #1579