FYI: https://github.com/Kagee/crtdumper

[Kagee]

FYI, i forked your repo and went a bit cracy in my own fork - https://github.com/Kagee/crtdumper. I don't know if you want to merge it, but just so you know :)

[c3l3si4n]

Hey, yeah I have interest in merging :)).
BTW, we can remove the retryable http task from the TODO list since go-retryablehttp already does that with exponential backoff.

[c3l3si4n]

Could you open a pull request??

[Kagee]

I have created a PR.

However, regarding the backoff - if go-retryablehttp is supposed to fix this, id does not appear to be working as intended. Here is an example of me running the code right now. As you can see, a 429 is returned, but if you look at the timestamps, it is retried within 1-2 seconds anyway 🤔

2024/06/21 14:42:51 WARN Got HTTP 429, increasing sleep for log=https://yeti2025.ct.digicert.com/log/
2024/06/21 14:42:52 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/
2024/06/21 14:42:53 WARN Got HTTP 429, increasing sleep for log=https://yeti2025.ct.digicert.com/log/
2024/06/21 14:42:54 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/
2024/06/21 14:42:55 WARN Got HTTP 429, increasing sleep for log=https://yeti2025.ct.digicert.com/log/
2024/06/21 14:42:56 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/
2024/06/21 14:42:57 WARN Got HTTP 429, increasing sleep for log=https://yeti2025.ct.digicert.com/log/
2024/06/21 14:42:57 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/
2024/06/21 14:42:58 WARN Got HTTP 429, increasing sleep for log=https://yeti2025.ct.digicert.com/log/
2024/06/21 14:42:59 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/
2024/06/21 14:43:00 WARN Got HTTP 429, increasing sleep for log=https://yeti2025.ct.digicert.com/log/
2024/06/21 14:43:01 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/
2024/06/21 14:43:02 WARN Got HTTP 429, increasing sleep for log=https://yeti2025.ct.digicert.com/log/

[Kagee]

It does not really help that they are not sending a Retry-After-header, so you have to guess/hope.
headers="map[Content-Length:[162] Content-Type:[text/html] Date:[Fri, 21 Jun 2024 12:56:38 GMT] Server:[nginx] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]]"

[Kagee]

Ok, it looks like there might actually be a backoff (turned on more logging), but it returns quickly. Maybe one just needs a limiter for som logs so it does not have to retry all the time

2024/06/21 15:04:28 DEBUG Querying log  url=https://yeti2024.ct.digicert.com/log/ operator=DigiCert
2024/06/21 15:04:28 DEBUG performing request method=GET url=https://yeti2024.ct.digicert.com/log/ct/v1/get-sth?
2024/06/21 15:04:29 INFO Progress log=https://yeti2024.ct.digicert.com/log/ prcnt=0
2024/06/21 15:04:29 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=945 endIndex=965 endIndex=1063720258
2024/06/21 15:04:29 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=965&start=945"
2024/06/21 15:04:30 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=966 endIndex=986 endIndex=1063720258
2024/06/21 15:04:30 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=986&start=966"
2024/06/21 15:04:30 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=987 endIndex=1007 endIndex=1063720258
2024/06/21 15:04:30 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1007&start=987"
2024/06/21 15:04:30 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1008 endIndex=1028 endIndex=1063720258
2024/06/21 15:04:30 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1028&start=1008"
2024/06/21 15:04:31 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1029 endIndex=1049 endIndex=1063720258
2024/06/21 15:04:31 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1049&start=1029"
2024/06/21 15:04:31 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/ headers="map[Content-Length:[162] Content-Type:[text/html] Date:[Fri, 21 Jun 2024 13:04:31 GMT] Server:[nginx] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]]"
2024/06/21 15:04:31 DEBUG retrying request request="GET https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1049&start=1029 (status: 429)" timeout=1s remaining=10
2024/06/21 15:04:32 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1050 endIndex=1070 endIndex=1063720258
2024/06/21 15:04:32 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1070&start=1050"
2024/06/21 15:04:32 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1071 endIndex=1091 endIndex=1063720258
2024/06/21 15:04:32 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1091&start=1071"
2024/06/21 15:04:33 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1092 endIndex=1112 endIndex=1063720258
2024/06/21 15:04:33 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1112&start=1092"
2024/06/21 15:04:33 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/ headers="map[Content-Length:[162] Content-Type:[text/html] Date:[Fri, 21 Jun 2024 13:04:33 GMT] Server:[nginx] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]]"
2024/06/21 15:04:33 DEBUG retrying request request="GET https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1112&start=1092 (status: 429)" timeout=1s remaining=10
2024/06/21 15:04:34 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1113 endIndex=1133 endIndex=1063720258
2024/06/21 15:04:34 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1133&start=1113"
2024/06/21 15:04:34 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1134 endIndex=1154 endIndex=1063720258
2024/06/21 15:04:34 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1154&start=1134"
2024/06/21 15:04:35 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1155 endIndex=1175 endIndex=1063720258
2024/06/21 15:04:35 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1175&start=1155"
2024/06/21 15:04:35 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/ headers="map[Content-Length:[162] Content-Type:[text/html] Date:[Fri, 21 Jun 2024 13:04:35 GMT] Server:[nginx] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]]"
2024/06/21 15:04:35 DEBUG retrying request request="GET https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1175&start=1155 (status: 429)" timeout=1s remaining=10
2024/06/21 15:04:36 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1176 endIndex=1196 endIndex=1063720258
2024/06/21 15:04:36 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1196&start=1176"
2024/06/21 15:04:36 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/ headers="map[Content-Length:[162] Content-Type:[text/html] Date:[Fri, 21 Jun 2024 13:04:36 GMT] Server:[nginx] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]]"
2024/06/21 15:04:36 DEBUG retrying request request="GET https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1196&start=1176 (status: 429)" timeout=1s remaining=10
2024/06/21 15:04:37 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1197 endIndex=1217 endIndex=1063720258
2024/06/21 15:04:37 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1217&start=1197"
2024/06/21 15:04:38 DEBUG Querying url=https://yeti2024.ct.digicert.com/log/ startIndex=1218 endIndex=1238 endIndex=1063720258
2024/06/21 15:04:38 DEBUG performing request method=GET url="https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1238&start=1218"
2024/06/21 15:04:38 WARN Got HTTP 429, increasing sleep for log=https://yeti2024.ct.digicert.com/log/ headers="map[Content-Length:[162] Content-Type:[text/html] Date:[Fri, 21 Jun 2024 13:04:38 GMT] Server:[nginx] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]]"
2024/06/21 15:04:38 DEBUG retrying request request="GET https://yeti2024.ct.digicert.com/log/ct/v1/get-entries?end=1238&start=1218 (status: 429)" timeout=1s remaining=10

[c3l3si4n]

Maybe there's a way of configuring this in the library without resorting to a custom implementation. I gotta check though.

[Kagee]

As mentioned in https://medium.com/mflow/rate-limiting-in-golang-http-client-a22fba15861a, relaying on repeatedly hitting 429 and backing off might not be smart, as repeat HTTP 429 offenders could be banned

[c3l3si4n]

TBH I agree. Maybe something like setting a minimum sleep delay, and increasing that sleep delay in case a 429 is returned could work? Biggest issue here is that CT logs don't have a static rate limiting policy, depends entirely on their mood. I've seen complaints on Google Groups about Google's CT logs ratelimiting people at 20 rps in some days.