High CPU usage if request content is bytes

[kice]

I noticed a increase of CPU usage after I changed request content from content=io.BytesIO(b'<actual data>') to content=b'<actual data>'. The content I want to send is 10MB in size, which is not that much, but still considerable amount of data.

My code is running on a VPS with low end CPU, which makes it much more dramatic (from peek at 40% to pinned at 100%).

I would like to help debug this issue, but I need some instructions to do so.

[florimondmanca]

@kice For reference, the code for sending POST data bytes lives here:

httpx/httpx/_content.py

Line 107 in 2880c08

def encode_content(

As you can see, the code path is different between bytes and Iterable[bytes], which BytesIO is.

In the first case, the entire 10MB body is sent through in one go. On the other hand, it is sent through in chunks of 64 kB via [IteratorByteStream](

httpx/httpx/_content.py

Line 42 in 2880c08

class IteratorByteStream(SyncByteStream):

.

The actual "sending" happens in httpcore, here (HTTP/1.1 sync case): https://github.com/encode/httpcore/blob/master/httpcore/_sync/http11.py#L137

It could be that the treatment of a huge 10 MB chunk by any part of the networking stack (h11, socket library, actual system sockets) is much more CPU intensive than sending smaller chunks one at a time.

To confirm, it would be interesting to compare CPU usage on your machine with some sample code sending 10 MB random data with each method. Are you able to replicate a bump?

(I am moving this to a discussion for now.)