GET request to Brightcove returns `200 OK` using requests but `404 NOT FOUND` in httpx

[whom-goodinside]

Hello! We've been using httpx. when fetching videos from Brightcove. Last week we noticed a large spike of 404s coming back and we couldn't figure out why. Running out of ideas, we swapped to using the requests module and lo and behold we started getting 200s again! The good news was that we resolved the issue; the bad news is that it's about as unsatisfactory a resolution as you can imagine.

We were using 0.27.0 in production, but for testing purposes prior to creating this issue, I used 0.27.2.

In the course of trying to figure it out, I can't discern anything different between the two modules. I also can't really provide steps to reproduce this due to needing authentication to execute the request. But basically, I tried the following (including the script with redacted information):

1. Wrote a script and reconstructed the request headers.
2. Executed a GET request for the same URL using the same headers using httpx and requests. The former gave me a 404, the latter gave me a 200.
3. Printed out the request headers and asserted they match.
4. Used curlify2 to translate the requests into CURL commands. The output looks identical; the only difference is that in requests there is a literal "None" in -d while httpx has 'b'''.
5. Taking the CURL commands and running them in my terminal, they both give me back a 200.

We're planning to speak with Brightcove to see if they have logs that give us more information, but I wanted to flag this to the project to see if there's something else going on. Thank you!

Here is the output of curlify2:

curl -X GET -H "authorization: <REDACTED>" -H "content-type: application/json" -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.3" -H "host: edge.api.brightcove.com" -H "accept-encoding: gzip, deflate" -H "accept: */*" -H "connection: keep-alive" -d 'b''' https://edge.api.brightcove.com/playback/v1/accounts/<REDACTED>/videos/<REDACTED>  # httpx

curl -X GET -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.3" -H "accept-encoding: gzip, deflate" -H "accept: */*" -H "connection: keep-alive" -H "authorization: <REDACTED>" -H "content-type: application/json" -H "host: edge.api.brightcove.com" -d 'None' https://edge.api.brightcove.com/playback/v1/accounts/<REDACTED>/videos/<REDACTED>  # requests

import requests
import httpx
from curlify2 import Curlify

PUBLIC_ID = ""
POLICY_KEY = ""
VIDEO_ID = ""
BASE_URL = f"https://edge.api.brightcove.com/playback/v1/accounts/{PUBLIC_ID}"


headers = {
	"authorization": f"BCOV-Policy {POLICY_KEY}",
	"content-type": "application/json",
	"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.3",
	"host": "edge.api.brightcove.com",
	"accept-encoding": "gzip, deflate",
	"accept": "*/*",
	"connection": "keep-alive"
}

request_url = f"{BASE_URL}/videos/{VIDEO_ID}"

print(request_url)

req_requests = requests.get(headers=headers, url=request_url)
response = Curlify(req_requests.request)

print(req_requests.status_code)
print(response.to_curl())
print("---------------------")

req_httpx = httpx.get(headers=headers, url=request_url, follow_redirects=True)
response = Curlify(req_httpx.request)

print(req_httpx.status_code)
print(response.to_curl())

print("---------------------")

print(req_requests.request.headers)
print(dict(req_httpx.request.headers))
assert dict(req_httpx.request.headers) == req_requests.request.headers

[lovelydinosaur]

We're planning to speak with Brightcove to see if they have logs that give us more information, but I wanted to flag this to the project to see if there's something else going on.

Thanks. Yep that’d be my next step here too.
No obvious causes here on the client side.

Could be possible they’ve been spammed by a different user of httpx and are using digital fingerprinting to prevent misuse?

[lovelydinosaur]

Going to convert this into a discussion since it’s not a reproducible issue, presumably something being done server side? Please do follow up the convo tho.