va: Inaccurate "Redirect loop detected" error message with redirect + connectivity issues(?)

[mnordhoff]

I have a setup (really just for testing) with 2 web servers (Akamai Dallas, DigitalOcean NYC1) that redirect all HTTP-01 validation requests to AWS S3 (us-west-2). (I'm using Certbot and a custom manual auth hook.)

Validation with the Let's Encrypt staging environment often fails with a seemingly-spurious "Redirect loop detected" error. I think there may be some kind of DNS or HTTP timeout issues, but I am almost certain the error message is incorrect. In all of my testing, my web servers and AWS behave correctly. Short of cosmic rays, an MITM or an S3 feature I haven't heard of, no redirect loop should be possible.

(The Let's Encrypt production environment and another CA usually or always succeed.)

This is mainly an "I suspect the error message is wrong" issue; I'm not asserting the validation outcome itself is wrong. If that's so, the only problem is some user confusion, and expanding the error message would solve the issue.

When it fails, according to my logs, the primary VA will make an IPv6 request to one of my web servers, and an IPv4 request to one of my web servers; and there will be zero requests to S3 or from the remote VAs, e.g.:

2600:3000:2710:300::81 s3.letsencrypt.mattnordhoffdns.com - [15/Aug/2026:20:44:40 +0000] "GET /.well-known/acme-challenge/2Jwb-yVEhjeHXTWaUqdXcxrcVmDsxwnRsjeZFGwTumo HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 s3.letsencrypt.mattnordhoffdns.com - [15/Aug/2026:20:44:50 +0000] "GET /.well-known/acme-challenge/2Jwb-yVEhjeHXTWaUqdXcxrcVmDsxwnRsjeZFGwTumo HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

(AWS claims S3 logs are best-effort and only mostly complete, but that's probably not usually an issue.)

The authz will look like, for example:

2026-08-15 20:44:53,189:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz/18633122/3737265094 HTTP/1.1" 200 None
2026-08-15 20:44:53,190:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 15 Aug 2026 20:44:53 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Boulder-Requester: 18633122
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: M-nBOc8e7Ttxvu7sS8YEKsoVRgAwraF8Ot0WTEET2LbDI5ndEl0
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{ 
  "identifier": {
    "type": "dns",
    "value": "s3.letsencrypt.mattnordhoffdns.com"
  },
  "status": "invalid",
  "expires": "2026-08-15T21:44:38Z",
  "challenges": [
    { 
      "type": "http-01",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/18633122/3737265094/7REB2w",
      "status": "invalid",
      "validated": "2026-08-15T20:44:39Z",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "104.237.139.208: Fetching http://mnordhoff-usw2-acme-challenge.s3.dualstack.us-west-2.amazonaws.com/.well-known/acme-challenge/2Jwb-yVEhjeHXTWaUqdXcxrcVmDsxwnRsjeZFGwTumo: Redirect loop detected",
        "status": 400
      },
      "token": "2Jwb-yVEhjeHXTWaUqdXcxrcVmDsxwnRsjeZFGwTumo",
      "validationRecord": [
        { 
          "url": "http://s3.letsencrypt.mattnordhoffdns.com/.well-known/acme-challenge/2Jwb-yVEhjeHXTWaUqdXcxrcVmDsxwnRsjeZFGwTumo",
          "hostname": "s3.letsencrypt.mattnordhoffdns.com",
          "port": "80",                                                 
          "addressesResolved": [
            "2600:3c00::f03c:91ff:fe96:6bcc",
            "104.237.139.208",
            "159.203.179.213"
          ],
          "addressUsed": "2600:3c00::f03c:91ff:fe96:6bcc"
        },
        {
          "url": "http://mnordhoff-usw2-acme-challenge.s3.dualstack.us-west-2.amazonaws.com/.well-known/acme-challenge/2Jwb-yVEhjeHXTWaUqdXcxrcVmDsxwnRsjeZFGwTumo",
          "hostname": "mnordhoff-usw2-acme-challenge.s3.dualstack.us-west-2.amazonaws.com",
          "port": "80",
          "addressesResolved": [
            "2600:1f60:4000:40::305:5132",
            "2600:1f60:4000:c0::305:542c",
            "2600:1f60:4000:20::305:571c",
            "2600:1f60:4000:c0::305:54c4",
            "2600:1f60:4000:40::305:512b",
            "2600:1f60:4000:80::305:4ede",
            "2600:1f60:4000:80::305:4d55",
            "16.15.33.179",
            "3.5.84.226",
            "52.218.178.145",
            "16.15.43.63",
            "3.5.80.128",
            "16.15.34.191",
            "52.218.137.233",
            "3.5.83.166"
          ],
          "addressUsed": "2600:1f60:4000:40::305:5132"
        },
        {
          "url": "http://s3.letsencrypt.mattnordhoffdns.com/.well-known/acme-challenge/2Jwb-yVEhjeHXTWaUqdXcxrcVmDsxwnRsjeZFGwTumo",
          "hostname": "s3.letsencrypt.mattnordhoffdns.com",
          "port": "80",
          "addressesResolved": [
            "2600:3c00::f03c:91ff:fe96:6bcc",
            "104.237.139.208",
            "159.203.179.213"
          ],
          "addressUsed": "104.237.139.208"
        } 
      ]   
    }
  ]
}

When it succeeds, I get the expected plethora of requests to my web servers and S3, from the primary and remote VAs, all using IPv6.

My half-developed theory is that unknown circumstances (connection timeout to redirect destination + IPv4 fallback retry? TCP error mid-response?) can create multiple records with the same URL, which the redirect loop detector then incorrectly assumes must be a redirect loop:

boulder/va/http.go

Lines 570 to 576 in 4aeb577

	// Check for a redirect loop. If any URL is found twice before the
	// redirect limit, return error.
	for _, record := range records {
	if req.URL.String() == record.URL {
	return berrors.ConnectionFailureError("Redirect loop detected")
	}
	}

Someone else recently posted on the forum about what may be the same issue:

https://community.letsencrypt.org/t/intermittent-loop-detected-error/250322

Can that happen? Am I totally out to lunch?

[aarongable]

I don't know what changed to cause this bug to trigger more frequently, since the logic hasn't changed in several years. But I suspect that the bug you're seeing is real and that it will be fixed by #8905, which fixes a long-standing bug in how we do IPv6-to-IPv4 fallback.

[mnordhoff]

D'oh! Yes, that makes sense. Thank you!

I looked at open issues but didn't think to check open PRs before filing this. >_<