Why does aiter_raw raise StreamConsumed but aiter_bytes doesn't ?

[adnaanbheda]

What was the thought process behind this design choice ?

I'd expect both of the functions to behave in the same manner but turns out they're very different, had to read through the inner workings to figure out what was happening !

Also, when does it make sense to use aiter_raw / aiter_bytes, examples in docs will also help explain their behavior better if we want to stick to this.

[lovelydinosaur]

Why does aiter_raw raise StreamConsumed but aiter_bytes doesn't?

Could you include a super simple example that demonstrates the difference in behaviour?

[adnaanbheda]

Sorry for the late response.
As you can see that I'm reading the stream using aread(), here aiter_bytes prints the response fine, but aiter_raw will raise the aforementioned exception.

import httpx
import requests
import asyncio
import pickle

URL = "https://api.github.com/users/adnaanbheda/repos"


async def httpx_test():

    async with httpx.AsyncClient() as client:
        req = client.build_request("get", URL)

        res = await client.send(req, stream=True)
        # Read the stream
        await res.aread()
        # Doesn't raise StreamConsumed
        async for chunk in res.aiter_bytes():
            print(chunk)
        # Raises StreamConsumed
        async for chunk in res.aiter_raw():
            print(chunk)

asyncio.run(httpx_test())

[lovelydinosaur]

Right.

So .read() or .aread() reads and returns the bytewise-content of the response. To be more specific, it returns the bytewise-content of the response after any decompression is applied. (HTTP response boys might be compressed using gzip, brotli or other encodings.)

Once you've read the response body it's still available, so you can still call iter_bytes() or aiter_bytes(), which will just re-return the content again.

In contrast, the iter_raw() iterator is for when you want to access the raw byte-wise content of the response, without decoding it. This isn't something you typically need to do. As an end user you're almost always interested in the actual (decompressed) content of the response, rather than the compressed bytes-on-the-wire. It's also only available once and once only. Once the response has been read, we no longer cache and store the compressed content, because it's almost never required.

[adnaanbheda]

My concern is related to consistent behaviour, the functions are named very similarly and thus, can lead to unexpected behaviour for the end developer, which is what happened with me, I assumed that the raw/bytes would behave in the same manner and differ in the encoding of the response.

Just my 2 cents.