illegal chunk header #1735

edenzaraf · 2021-07-04T13:08:59Z

edenzaraf
Jul 4, 2021

Hey,
I'm trying to send normal request which worked for me in other libraries. but Im getting this error:
httpx.RemoteProtocolError: illegal chunk header: bytearray(b'e30 \r\n')
code:
async with httpx.AsyncClient(verify=False) as client: res = await client.get(url)

Answered by lovelydinosaur

Jul 5, 2021

Could you provide some more information. (In particular an example that demonstrates a failing URL would be super helpful here.)

It's very very likely that the cause of this is a server that's responding with non-compliant HTTP. The parsing library that we use - h11 - is very strict on catching invalid HTTP. It's always useful for us to find real world cases that it's picking up on.

View full answer

lovelydinosaur · 2021-07-05T10:29:16Z

lovelydinosaur
Jul 5, 2021
Maintainer

Could you provide some more information. (In particular an example that demonstrates a failing URL would be super helpful here.)

It's very very likely that the cause of this is a server that's responding with non-compliant HTTP. The parsing library that we use - h11 - is very strict on catching invalid HTTP. It's always useful for us to find real world cases that it's picking up on.

9 replies

edenzaraf Jul 5, 2021
Author

Now I just noticed that in some cases you dont get this error on that one. Sorry for that...
Try on that one:
https://31.210.182.120:10443/remote/login?lang=en

lovelydinosaur Jul 6, 2021
Maintainer

I've taken a snapshot of the raw bytes that are sent over the wire for that response...

HTTP/1.1 200 OK\r\n
Date: Tue, 06 Jul 2021 07:55:57 GMT\r\n
Server: xxxxxxxx-xxxxx\r\n
Set-Cookie:  SVPNCOOKIE=; path=/; expires=Sun, 11 Mar 1984 12:00:00 GMT; secure; httponly;\r\n
Set-Cookie: SVPNNETWORKCOOKIE=; path=/remote/network; expires=Sun, 11 Mar 1984 12:00:00 GMT; secure; httponly\r\n
X-UA-Compatible: requiresActiveX=true\r\n
Keep-Alive: timeout=10, max=100\r\n
Connection: Keep-Alive\r\n
Transfer-Encoding: chunked\r\n
Content-Type: text/html; charset=utf-8\r\n
X-Frame-Options: SAMEORIGIN\r\n
Content-Security-Policy: frame-ancestors \'self\'; object-src \'none\'; script-src \'self\' https \'unsafe-eval\' \'unsafe-inline\';\r\n
X-XSS-Protection: 1; mode=block\r\n
X-Content-Type-Options: nosniff\r\nStrict-Transport-Security: max-age=31536000\r\n
\r\n
e22   \r\n
<!DOCTYPE html>\n<html lang="en" class="main-app">\n    <head>\n        <meta charset="UTF-8">\n        <meta http-equiv="X-UA-Compatible" content="IE=8; IE=EDGE">\n        <meta name="viewport" content="width=device-width, initial-scale=1">\n        <meta name="apple-itunes-app" content="app-id=525600370">\n        <link href="/css/main-blue.css" rel="stylesheet" type="text/css">\n        <title>Please Login</title>\n    </head>\n    <body>\n        <div class="view-container">\n            <form class="prompt" action="/remote/logincheck" method="post" name="f" autocomplete="off">\n                <div class="content with-header">\n                    <div class="header">\n                        <f-icon class="ftnt-fortinet-grid icon-xl"></f-icon>\n                        <div>Please Login</div>\n                    </div>\n                    <div class="sub-content">\n                        <div class="wide-inputs">\n                            <!--remoteauthtimeout=10-->\n<input type="text" name="username" id="username" placeholder="Name"><input type="password" name="credential" id="credential" placeholder="Password" maxlength="128"><div class="info-message" id="token_msg" style="display: none;"><div class="message-content" id="token_label"></div></div><input type="password" style="display: none;" maxlength="128" name="credential2" id="credential2"><input type="password" style="display: none;" maxlength="128" name="credential3" id="credential3"><div class="button-actions wide"><button class="primary" type="button" name="ftm_push_button" id="ftm_push_button" onclick="try_ftm_push()    " style="display: none" disabled>Use FTM Push</button></div><input type="password" style="display: none;" placeholder="Token" name="code" id="code"><div id="driftmsg" style="display: none;" class="warning-message">Token clock drift detected. Please input the next code and continue.</div><input type="password" style="display: none;" name="code2" id="code2" placeholder="Next Token Code">\n                        </div>\n                        <div class="button-actions wide">\n                            <button class="primary" type="button" name="login_button" id="login_button" onClick="try_login()">\n                                Login\n                            </button>\n                            <button id="launch-forticlient-button" type="button" onClick="launchFortiClient()">\n                                <f-icon class="ftnt-forticlient"></f-icon>\n                                <span>Launch FortiClient</span>\n                            </button>\n                            <iframe id="launch-forticlient-iframe" style="display:none"></iframe>\n                        </div>\n                    </div>\n                </div>\n            </form>\n        </div>\n    </body>\n    <input type=hidden name="ftm_push_enabled" id="ftm_push_enabled" value="1"><input type=hidden name=just_logged_in value=1><input type=hidden name=magic id=magic_id value=""><input type=hidden name=reqid id=reqid_id value="0"><input type=hidden name=grpid id=grpid_id value=""><input type=hidden name=realm id=realm_id value=""><input type=hidden name=redir value="/sslvpn/portal.html"><script type="text/javascript" src="/sslvpn/js/login.js?q=86e0a7c0ca774c01b91cc296ac7c31c9"></script><script type="text/javascript" src="/remote/fgt_lang?lang=en"></script><script>document.onkeydown = key_pressdown;function load_login_strings() {var tmp = document.getElementById("err_val");if (tmp) {tmp.innerHTML = fgt_lang["error"] + ": " + fgt_lang[tmp.getAttribute(\'title\')];}}window.onload = load_login_strings;</script>\n</html>\n\r\n\r\n0\r\n\r\n'

Everything is correct about the response except that the chunked transfer encoding includes some trailing whitespace, which is presumably not spec-compliant, but which it'd be beneficial if h11 was able to deal with.

For a bit more context it'd be interesting to know what webserver you're using here?
I can see a "Server: xxxxxxxx-xxxxx" header which doesn't give a lot of information away?

edenzaraf Jul 6, 2021
Author

Im unable to answer that... I dont know :(
What I do know is that the problem appears on every fortinet site that I have tested. If this helps I can search for some more sites with fortinet if that will be usefull to fix it.

lovelydinosaur Jul 6, 2021
Maintainer

If this helps I can search for some more sites with fortinet if that will be usefull to fix it.

That'd be great if you can.

We don't need that info, but if there are some public URLs that we can show have the same issue, then it just gives us a little more context about how likely it is that other users might see the same issue.

Thanks so much for your time, either way.

lovelydinosaur Jul 6, 2021
Maintainer

Raised on the h11 repository here: python-hyper/h11#133

drschlaumeier · 2022-07-21T14:15:05Z

drschlaumeier
Jul 21, 2022

Hallo,

It seems I have similar issue with "Illegal chunk header" and I don't know how I can solve it.

I'm using HomeAssistant software which is using httpx in a module to scrape sensor values from web sites.
Please see the issue reporting and full documentation here:
danieldotnl/ha-multiscrape#134

How can I resolve the problem? on my end?

Thanks for helping

2 replies

drschlaumeier Aug 4, 2022

Bump, any comments ?

eseglem Sep 25, 2022

h11 has put out a new release (v0.14.0) addressing this, and that will hopefully make its way through httpcore and httpx soon.

I have temporarily set up my code facing the same issue to monkeypatch in the fix, but wouldn't recommend doing it that way.

rohrsh · 2023-02-14T23:56:25Z

rohrsh
Feb 14, 2023

I suspect I am suffering the same problem. Header errors when trying to parse webpage on a local solar monitor.

ksheumaker/homeassistant-apsystems_ecur#156

Have the fixes mentioned previously worked through to Home Assistant multiscrape? https://github.com/danieldotnl/ha-multiscrape
That's where I get the error "# Updating failed with exception: illegal header line: bytearray(b'debug9')"

1 reply

rohrsh Feb 15, 2023

Perhaps the semicolon in the content-type response?

Uh oh!

illegal chunk header #1735

Uh oh!

edenzaraf Jul 4, 2021

Replies: 3 comments · 12 replies

Uh oh!

lovelydinosaur Jul 5, 2021 Maintainer

Uh oh!

edenzaraf Jul 5, 2021 Author

Uh oh!

lovelydinosaur Jul 6, 2021 Maintainer

Uh oh!

edenzaraf Jul 6, 2021 Author

Uh oh!

lovelydinosaur Jul 6, 2021 Maintainer

Uh oh!

lovelydinosaur Jul 6, 2021 Maintainer

Uh oh!

drschlaumeier Jul 21, 2022

Uh oh!

drschlaumeier Aug 4, 2022

Uh oh!

eseglem Sep 25, 2022

Uh oh!

Uh oh!

rohrsh Feb 14, 2023

Uh oh!

rohrsh Feb 15, 2023

edenzaraf
Jul 4, 2021

Replies: 3 comments 12 replies

lovelydinosaur
Jul 5, 2021
Maintainer

edenzaraf Jul 5, 2021
Author

lovelydinosaur Jul 6, 2021
Maintainer

edenzaraf Jul 6, 2021
Author

lovelydinosaur Jul 6, 2021
Maintainer

lovelydinosaur Jul 6, 2021
Maintainer

drschlaumeier
Jul 21, 2022

rohrsh
Feb 14, 2023