Stream upload problem for binary files without newline

[ysn2233]

client = httpx.Client()
client.put(url=url, data=open("filename", "rb"))

When I use httpx to upload a large binary file(>2G), it will lead to OverflowError: string longer than 2147483647 bytes. It is because my binary file don't contain '\n' byte and httpx stream the data with default iterator of the fileobj(_io.BufferedReader). It will read chunk by '\n', so in my case all bytes will be read into memory and it is definetely not reasonable. This problem may also occur in case of a file with an extremely long line.

I can externally solve this problem by wrapper a custom class extended BufferedReader and rewrite __next__. However do you think we should support fixed-size chunk streaming upload in httpx? My solution is to rewrite __iter__ in class IteratorByteStream like that:

    def __iter__(self) -> Iterator[bytes]:
        if self._is_stream_consumed and self._is_generator:
            raise StreamConsumed()

        self._is_stream_consumed = True

        # old
        # for part in self._stream:
            # yield part
        # new
        while True:
            part = self._stream.read(self.block_size)
            if not part:
                break
            yield part

Or some better design could be made.

[lovelydinosaur]

• Double check you're using the latest version of httpx.
• You want content=... for byte uploads not data=....

If you take a look at the internals you'll see that we do prefer .read(chunk_size) where possible...

httpx/httpx/_content.py

Lines 50 to 64 in b82fbe2

	def __iter__(self) -> Iterator[bytes]:
	if self._is_stream_consumed and self._is_generator:
	raise StreamConsumed()
	self._is_stream_consumed = True
	if hasattr(self._stream, "read"):
	# File-like interfaces should use 'read' directly.
	chunk = self._stream.read(self.CHUNK_SIZE) # type: ignore
	while chunk:
	yield chunk
	chunk = self._stream.read(self.CHUNK_SIZE) # type: ignore
	else:
	# Otherwise iterate.
	for part in self._stream:
	yield part

We didn't get this right in the past, but it was fixed in #1948