Unable to recieve Content-Length response header for some websites.

[wizard-28]

I'm not able to receive the Content-Length response header from this url.

Here's the headers I receive:

Headers([('accept-ranges', 'bytes'), ('etag', '"88ab8a45681e1d86e8347d6faf8d121e:1649410299.413639"'), ('last-modified', 'Fri, 08 Apr 2022 09:33:07 GMT'), ('server', 'AkamaiNetStorage'), ('vary', 'Accept-Encoding'), ('content-encoding', 'gzip'), ('date', 'Fri, 29 Apr 2022 19:44:03 GMT'), ('transfer-encoding', 'chunked'), ('connection', 'keep-alive'), ('connection', 'Transfer-Encoding'), ('content-type', 'application/octet-stream')])

---

Other HTTP clients are able to get the Content-Length however. Here's a comparison with urllib:

>>> from urllib.request import urlopen
>>> urlopen("https://downloads.bitwig.com/4.2.3/bitwig-studio-4.2.3.deb").info()["Content-Length"]
'245235928'
>>> import httpx
>>> httpx.get("https://downloads.bitwig.com/4.2.3/bitwig-studio-4.2.3.deb").headers["Content-Length"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/httpx/_models.py", line 993, in __getitem__
    raise KeyError(key)
KeyError: 'Content-Length'
>>>

And this online tool is also able to get the Content-Length response header.

[lovelydinosaur]

I've done some digging, and we've got the same behaviour as requests and the httpie command line tool here.

You're seeing a difference when using urlopen, because that doesn't support the gzip encoding.

The server is responding differently depending on if it's sending a gzip encoding response, or if it's sending an unencoded response. For gzipped responses it can't determine the content-length up front, but the response will be compressed.

You can see the difference by switching gzip compression off...

$ httpx https://downloads.bitwig.com/4.2.3/bitwig-studio-4.2.3.deb -v --headers Accept-Encoding 'identity'
* Connecting to 'downloads.bitwig.com'
* Connected to '62.252.115.10' on port 443
* SSL established using 'TLSv1.2' / 'ECDHE-RSA-AES256-GCM-SHA384'
* Selected ALPN protocol: 'http/1.1'
* Server certificate:
*   subject:
*     commonName: 'bitwig.net'
*   issuer:
*     countryName: 'US'
*     organizationName: "Let's Encrypt"
*     commonName: 'R3'
*   version: 3
*   serialNumber: '047CB707CBDE39542F63CAF5D364C30F76E4'
*   notBefore: 'Mar 22 08:55:14 2022 GMT'
*   notAfter: 'Jun 20 08:55:13 2022 GMT'
*   subjectAltName:
*     DNS: 'assets-stage.bitwig.net'
*     DNS: 'assets.bitwig.net'
*     DNS: 'bitwig.net'
*     DNS: 'dash.bitwig.com'
*     DNS: 'dash.bitwig.net'
*     DNS: 'downloads-secure.bitwig.com'
*     DNS: 'downloads-secure.bitwig.net'
*     DNS: 'downloads-stage.bitwig.net'
*     DNS: 'downloads.bitwig.com'
*     DNS: 'downloads.bitwig.net'
*     DNS: 'maven.bitwig.com'
*     DNS: 'maven.bitwig.net'
*     DNS: 'packs-stage.bitwig.net'
*     DNS: 'packs.bitwig.com'
*     DNS: 'packs.bitwig.net'
*   OCSP:
*     'http://r3.o.lencr.org'
*   caIssuers:
*     'http://r3.i.lencr.org/'
GET /4.2.3/bitwig-studio-4.2.3.deb HTTP/1.1
Host: downloads.bitwig.com
Accept: */*
Connection: keep-alive
User-Agent: python-httpx/0.22.0
Accept-Encoding: identity

HTTP/1.1 200 OK
Accept-Ranges: bytes
ETag: "88ab8a45681e1d86e8347d6faf8d121e:1649410299.413639"
Last-Modified: Fri, 08 Apr 2022 09:33:07 GMT
Server: AkamaiNetStorage
Content-Length: 245235928
Date: Tue, 03 May 2022 09:36:21 GMT
Connection: keep-alive
Content-Type: application/octet-stream

Compared to this...

$ httpx https://downloads.bitwig.com/4.2.3/bitwig-studio-4.2.3.deb -v
* Connecting to 'downloads.bitwig.com'
* Connected to '62.252.115.17' on port 443
* SSL established using 'TLSv1.2' / 'ECDHE-RSA-AES256-GCM-SHA384'
* Selected ALPN protocol: 'http/1.1'
* Server certificate:
*   subject:
*     commonName: 'bitwig.net'
*   issuer:
*     countryName: 'US'
*     organizationName: "Let's Encrypt"
*     commonName: 'R3'
*   version: 3
*   serialNumber: '047CB707CBDE39542F63CAF5D364C30F76E4'
*   notBefore: 'Mar 22 08:55:14 2022 GMT'
*   notAfter: 'Jun 20 08:55:13 2022 GMT'
*   subjectAltName:
*     DNS: 'assets-stage.bitwig.net'
*     DNS: 'assets.bitwig.net'
*     DNS: 'bitwig.net'
*     DNS: 'dash.bitwig.com'
*     DNS: 'dash.bitwig.net'
*     DNS: 'downloads-secure.bitwig.com'
*     DNS: 'downloads-secure.bitwig.net'
*     DNS: 'downloads-stage.bitwig.net'
*     DNS: 'downloads.bitwig.com'
*     DNS: 'downloads.bitwig.net'
*     DNS: 'maven.bitwig.com'
*     DNS: 'maven.bitwig.net'
*     DNS: 'packs-stage.bitwig.net'
*     DNS: 'packs.bitwig.com'
*     DNS: 'packs.bitwig.net'
*   OCSP:
*     'http://r3.o.lencr.org'
*   caIssuers:
*     'http://r3.i.lencr.org/'
GET /4.2.3/bitwig-studio-4.2.3.deb HTTP/1.1
Host: downloads.bitwig.com
Accept: */*
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
User-Agent: python-httpx/0.22.0

HTTP/1.1 200 OK
Accept-Ranges: bytes
ETag: "88ab8a45681e1d86e8347d6faf8d121e:1649410299.413639"
Last-Modified: Fri, 08 Apr 2022 09:33:07 GMT
Server: AkamaiNetStorage
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Tue, 03 May 2022 09:42:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Connection: Transfer-Encoding
Content-Type: application/octet-stream