Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Different rate limits depending on HTTP method (#5555) #5869

Open
wants to merge 6 commits into
base: develop
Choose a base branch
from
Open

Different rate limits depending on HTTP method (#5555) #5869

wants to merge 6 commits into from
+194 −131

Conversation

dsotirho-ucsc
Copy link
Contributor

@dsotirho-ucsc dsotirho-ucsc commented Jan 19, 2024 •
edited
Loading

Connected issues: #5555

Checklist

Author

  • PR is a draft
  • Target branch is develop
  • Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
  • On ZenHub, PR is connected to all issues it (partially) resolves
  • PR description links to connected issues
  • PR title matches1 that of a connected issue or comment in PR explains why they're different
  • PR title references all connected issues
  • For each connected issue, there is at least one commit whose title references that issue

1 when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

  • Added p tag to titles of partial commits
  • This PR is labeled partial or completely resolves all connected issues
  • This PR partially resolves each of the connected issues or does not have the partial label

Author (chains)

  • This PR is blocked by previous PR in the chain or is not chained to another PR
  • The blocking PR is labeled base or this PR is not chained to another PR
  • This PR is labeled chained or is not chained to another PR

Author (reindex, API changes)

  • Added r tag to commit title or the changes introduced by this PR will not require reindexing of any deployment
  • This PR is labeled reindex:dev or the changes introduced by it will not require reindexing of dev
  • This PR is labeled reindex:anvildev or the changes introduced by it will not require reindexing of anvildev
  • This PR is labeled reindex:anvilprod or the changes introduced by it will not require reindexing of anvilprod
  • This PR is labeled reindex:prod or the changes introduced by it will not require reindexing of prod
  • This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod
  • This PR and its connected issues are labeled API or this PR does not modify a REST API
  • Added a (A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST API
  • Updated REST API version number in app.py or this PR does not modify a REST API

Author (upgrading deployments)

  • Ran make image_manifests.json and committed the resulting changes or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable
  • Documented upgrading of deployments in UPGRADING.rst or this PR does not require upgrading deployments
  • Added u tag to commit title or this PR does not require upgrading deployments
  • This PR is labeled upgrade or does not require upgrading deployments
  • This PR is labeled deploy:shared or does not modify image_manifests.json, and does not require deploying the shared component for any other reason
  • This PR is labeled deploy:gitlab or does not require deploying the gitlab component
  • This PR is labeled deploy:runner or does not require deploying the runner image

Author (hotfixes)

  • Added F tag to main commit title or this PR does not include permanent fix for a temporary hotfix
  • Reverted the temporary hotfixes for any connected issues or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues connected to this PR

Author (before every review)

  • Rebased PR branch on develop, squashed old fixups
  • Ran make requirements_update or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile
  • Added R tag to commit title or this PR does not modify requirements*.txt
  • This PR is labeled reqs or does not modify requirements*.txt
  • make integration_test passes in personal deployment or this PR does not modify functionality that could affect the IT outcome

Peer reviewer (after approval)

  • PR is not a draft
  • Ticket is in Review requested column
  • PR is awaiting requested review from system administrator
  • PR is assigned to only the system administrator

System administrator (after approval)

  • Actually approved the PR
  • Labeled connected issues as demo or no demo
  • Commented on connected issues about demo expectations or all connected issues are labeled no demo
  • Decided if PR can be labeled no sandbox
  • A comment to this PR details the completed security design review
  • PR title is appropriate as title of merge commit
  • N reviews label is accurate
  • Moved ticket to Approved column
  • PR is assigned to only the operator

Operator (before pushing merge the commit)

  • Checked reindex:… labels and r commit title tag
  • Checked that demo expectations are clear or all connected issues are labeled no demo
  • Squashed PR branch and rebased onto develop
  • Sanity-checked history
  • Pushed PR branch to GitHub
  • Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply or this PR is not labeled deploy:gitlab
  • Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply or this PR is not labeled deploy:gitlab
  • Checked the items in the next section or this PR is labeled deploy:gitlab
  • PR is assigned to only the system administrator or this PR is not labeled deploy:gitlab

System administrator

  • Background migrations for dev.gitlab are complete or this PR is not labeled deploy:gitlab
  • Background migrations for anvildev.gitlab are complete or this PR is not labeled deploy:gitlab
  • PR is assigned to only the operator

Operator (before pushing merge the commit)

  • Ran _select dev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner
  • Ran _select anvildev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner
  • Added sandbox label or PR is labeled no sandbox
  • Pushed PR branch to GitLab dev or PR is labeled no sandbox
  • Pushed PR branch to GitLab anvildev or PR is labeled no sandbox
  • Build passes in sandbox deployment or PR is labeled no sandbox
  • Build passes in anvilbox deployment or PR is labeled no sandbox
  • Reviewed build logs for anomalies in sandbox deployment or PR is labeled no sandbox
  • Reviewed build logs for anomalies in anvilbox deployment or PR is labeled no sandbox
  • Deleted unreferenced indices in sandbox or this PR does not remove catalogs or otherwise causes unreferenced indices in dev
  • Deleted unreferenced indices in anvilbox or this PR does not remove catalogs or otherwise causes unreferenced indices in anvildev
  • Started reindex in sandbox or this PR is not labeled reindex:dev
  • Started reindex in anvilbox or this PR is not labeled reindex:anvildev
  • Checked for failures in sandbox or this PR is not labeled reindex:dev
  • Checked for failures in anvilbox or this PR is not labeled reindex:anvildev
  • The title of the merge commit starts with the title of this PR
  • Added PR # reference to merge commit title
  • Collected commit title tags in merge commit title but only included p if the PR is also labeled partial
  • Moved connected issues to Merged column in ZenHub
  • Pushed merge commit to GitHub

Operator (chain shortening)

  • Changed the target branch of the blocked PR to develop or this PR is not labeled base
  • Removed the chained label from the blocked PR or this PR is not labeled base
  • Removed the blocking relationship from the blocked PR or this PR is not labeled base
  • Removed the base label from this PR or this PR is not labeled base

Operator (after pushing the merge commit)

  • Pushed merge commit to GitLab dev
  • Pushed merge commit to GitLab anvildev
  • Build passes on GitLab dev
  • Reviewed build logs for anomalies on GitLab dev
  • Build passes on GitLab anvildev
  • Reviewed build logs for anomalies on GitLab anvildev
  • Ran _select dev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • Ran _select anvildev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • Deleted PR branch from GitHub
  • Deleted PR branch from GitLab dev
  • Deleted PR branch from GitLab anvildev

Operator (reindex)

  • Deindexed all unreferenced catalogs in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Deindexed all unreferenced catalogs in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Deindexed specific sources in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Deindexed specific sources in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Indexed specific sources in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Indexed specific sources in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Started reindex in dev or this PR does not require reindexing dev
  • Started reindex in anvildev or this PR does not require reindexing anvildev
  • Checked for, triaged and possibly requeued messages in both fail queues in dev or this PR does not require reindexing dev
  • Checked for, triaged and possibly requeued messages in both fail queues in anvildev or this PR does not require reindexing anvildev
  • Emptied fail queues in dev or this PR does not require reindexing dev
  • Emptied fail queues in anvildev or this PR does not require reindexing anvildev

Operator

  • Propagated the deploy:shared, deploy:gitlab, deploy:runner, reindex:partial, reindex:anvilprod and reindex:prod labels to the next promotion PRs or this PR carries none of these labels
  • Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, reindex:partial, reindex:anvilprod and reindex:prod labels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labels
  • PR is assigned to no one

Shorthand for review comments

  • L line is too long
  • W line wrapping is wrong
  • Q bad quotes
  • F other formatting problem

@github-actions github-actions bot added the orange [process] Done by the Azul team label Jan 19, 2024
@coveralls
Copy link

coveralls commented Jan 19, 2024 •
edited
Loading

Coverage Status

coverage: 85.584% (+0.02%) from 85.563%
when pulling 7c47ea6 on issues/dsotirho-ucsc/5555-rate-limit-on-http-method
into 54a2200 on develop.

@codecov Codecov
Copy link

codecov bot commented Jan 19, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 82.35294% with 3 lines in your changes missing coverage. Please review.

Project coverage is 85.56%. Comparing base (54a2200) to head (7c47ea6).

Files with missing lines Patch % Lines
test/integration_test.py 0.00% 3 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #5869      +/-   ##
===========================================
+ Coverage    85.54%   85.56%   +0.02%     
===========================================
  Files          147      147              
  Lines        21226    21216      -10     
===========================================
- Hits         18158    18154       -4     
+ Misses        3068     3062       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 2 times, most recently from 34336cf to 1259465 Compare January 19, 2024 19:09
@dsotirho-ucsc
Copy link
Contributor Author

Tested against personal deployment

Using watch to flood API with requests:

daniel@Crispin ~ $ curl -X 'POST' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
{"message":"Missing Authentication Token"}daniel@Crispin ~ $
daniel@Crispin ~ $
daniel@Crispin ~ $ watch -n 1 curl -X 'POST' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
daniel@Crispin ~ $

Showing POST requests are blocked while GET request are allowed:

daniel@Crispin ~ $ curl -X 'POST' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
{"message":"Forbidden"}daniel@Crispin ~ $
daniel@Crispin ~ $
daniel@Crispin ~ $ curl 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
{"projectCount":105,"specimenCount":889,"speciesCount":2,"fileCount":3440,"totalFileSize":10967869203586.0,"donorCount":653,"labCount":204,"organTypes":["blood","brain","skin of body","kidney","large intestine","gonad","pancreas","hematopoietic system","central nervous system","endometrium","heart","lung","embryo","eye","lymph node","placenta","ovary","decidua","umbilical cord","colon","digestive system","liver","thymus","retina","tumor",null,"adipose tissue","arterial blood vessel","testis","tonsil","hindlimb","oral cavity","whole embryos","blastocyst","immune system","muscle","nose","skeletal muscle organ","skin","spleen","stem cell","uterus","wall of lateral ventricle","abdomen","immune organ","mouse kidney","small intestine","tail","trachea"],"fileTypeSummaries":[{"count":2885,"totalSize":7838690358439.0,"matrixCellCount":0.0,"format":"fastq.gz"},{"count":154,"totalSize":197202319290.0,"matrixCellCount":0.0,"format":"loom"},{"count":129,"totalSize":2395989149587.0,"matrixCellCount":0.0,"format":"bam"},{"count":83,"totalSize":506850977636.0,"matrixCellCount":0.0,"format":"fastq"},{"count":32,"totalSize":2298010985.0,"matrixCellCount":0.0,"format":"txt.gz"},{"count":29,"totalSize":156508385.0,"matrixCellCount":0.0,"format":"tsv.gz"},{"count":24,"totalSize":4228874240.0,"matrixCellCount":0.0,"format":"tar"},{"count":15,"totalSize":13529137.0,"matrixCellCount":160631.0,"format":"txt"},{"count":14,"totalSize":265348615.0,"matrixCellCount":51516.0,"format":"tsv"},{"count":12,"totalSize":38143884.0,"matrixCellCount":0.0,"format":"pdf"},{"count":11,"totalSize":8367867567.0,"matrixCellCount":0.0,"format":"zip"},{"count":10,"totalSize":603217402.0,"matrixCellCount":0.0,"format":"csv"},{"count":9,"totalSize":17863772.0,"matrixCellCount":0.0,"format":"csv.gz"},{"count":8,"totalSize":1921436840.0,"matrixCellCount":0.0,"format":"mtx.gz"},{"count":6,"totalSize":1739858713.0,"matrixCellCount":51516.0,"format":"mtx"},{"count":5,"totalSize":6764456.0,"matrixCellCount":0.0,"format":"xlsx"},{"count":4,"totalSize":3960869846.0,"matrixCellCount":800000.0,"format":"h5ad"},{"count":3,"totalSize":2637887113.0,"matrixCellCount":162420.0,"format":"Rds"},{"count":2,"totalSize":8961366.0,"matrixCellCount":0.0,"format":"h5"},{"count":2,"totalSize":1686428099.0,"matrixCellCount":0.0,"format":"rds"},{"count":1,"totalSize":719215660.0,"matrixCellCount":0.0,"format":"Robj"},{"count":1,"totalSize":428090368.0,"matrixCellCount":0.0,"format":"rds.gz"},{"count":1,"totalSize":37522186.0,"matrixCellCount":0.0,"format":"tar.gz"}],"cellCountSummaries":[{"organType":["hematopoietic system"],"countOfDocsWithOrganType":9637,"totalCellCountByOrgan":1489527.0},{"organType":["endometrium"],"countOfDocsWithOrganType":2157,"totalCellCountByOrgan":2148.0},{"organType":["blood"],"countOfDocsWithOrganType":208,"totalCellCountByOrgan":913740.0},{"organType":["skin of body"],"countOfDocsWithOrganType":104,"totalCellCountByOrgan":15460.0},{"organType":["brain"],"countOfDocsWithOrganType":93,"totalCellCountByOrgan":558844.0},{"organType":["pancreas"],"countOfDocsWithOrganType":77,"totalCellCountByOrgan":52358.0},{"organType":["kidney"],"countOfDocsWithOrganType":71,"totalCellCountByOrgan":163820.0},{"organType":["large intestine"],"countOfDocsWithOrganType":67,"totalCellCountByOrgan":6.0},{"organType":["gonad"],"countOfDocsWithOrganType":43,"totalCellCountByOrgan":43.0},{"organType":["umbilical cord"],"countOfDocsWithOrganType":42,"totalCellCountByOrgan":20032.0},{"organType":["central nervous system"],"countOfDocsWithOrganType":30,"totalCellCountByOrgan":34400.0},{"organType":["decidua"],"countOfDocsWithOrganType":28,"totalCellCountByOrgan":14000.0},{"organType":["heart"],"countOfDocsWithOrganType":28,"totalCellCountByOrgan":90000.0},{"organType":["lung"],"countOfDocsWithOrganType":25,"totalCellCountByOrgan":21775.0},{"organType":["colon"],"countOfDocsWithOrganType":21,"totalCellCountByOrgan":4744.0},{"organType":["embryo"],"countOfDocsWithOrganType":18,"totalCellCountByOrgan":15.0},{"organType":["eye"],"countOfDocsWithOrganType":16,"totalCellCountByOrgan":53535.0},{"organType":["ovary"],"countOfDocsWithOrganType":16,"totalCellCountByOrgan":11.0},{"organType":["lymph node"],"countOfDocsWithOrganType":14,"totalCellCountByOrgan":8.0},{"organType":["placenta"],"countOfDocsWithOrganType":14,"totalCellCountByOrgan":22526.0},{"organType":["testis"],"countOfDocsWithOrganType":14,"totalCellCountByOrgan":7511.0},{"organType":["abdomen"],"countOfDocsWithOrganType":13,"totalCellCountByOrgan":13.0},{"organType":["liver"],"countOfDocsWithOrganType":11,"totalCellCountByOrgan":30006.0},{"organType":["arterial blood vessel"],"countOfDocsWithOrganType":10,"totalCellCountByOrgan":0.0},{"organType":["digestive system"],"countOfDocsWithOrganType":9,"totalCellCountByOrgan":72000.0},{"organType":["thymus"],"countOfDocsWithOrganType":9,"totalCellCountByOrgan":0.0},{"organType":[null],"countOfDocsWithOrganType":9,"totalCellCountByOrgan":30000.0},{"organType":["tumor"],"countOfDocsWithOrganType":8,"totalCellCountByOrgan":8.0},{"organType":["retina"],"countOfDocsWithOrganType":6,"totalCellCountByOrgan":8217.0},{"organType":["adipose tissue"],"countOfDocsWithOrganType":5,"totalCellCountByOrgan":0.0},{"organType":["oral cavity"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":4.0},{"organType":["stem cell"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":0.0},{"organType":["tonsil"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":0.0},{"organType":["trachea"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":4.0},{"organType":["hindlimb"],"countOfDocsWithOrganType":3,"totalCellCountByOrgan":11590.0},{"organType":["skin"],"countOfDocsWithOrganType":3,"totalCellCountByOrgan":2413.0},{"organType":["whole embryos"],"countOfDocsWithOrganType":3,"totalCellCountByOrgan":0.0},{"organType":["blastocyst"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":2.0},{"organType":["immune system"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["muscle"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":3214.0},{"organType":["nose"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["skeletal muscle organ"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["spleen"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":30043.0},{"organType":["uterus"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["wall of lateral ventricle"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["immune organ"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":1.0},{"organType":["mouse kidney"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":5648.0},{"organType":["small intestine"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":0.0},{"organType":["tail"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":1.0}],"projects":[{"projects":{"estimatedCellCount":12086.0},"cellSuspensions":{"totalCells":null}},{"projects":{"estimatedCellCount":null},"cellSuspensions":{"totalCells":2559060.0}},{"projects":{"estimatedCellCount":927877.0},"cellSuspensions":{"totalCells":1093788.0}}]}daniel@Crispin ~ $

Alarm raised:
Screenshot 2024-01-19 at 11 20 41 AM

@hannes-ucsc
Copy link
Member

watch is inadequate for this experiment because it it sequential, not parallel.

@hannes-ucsc
Copy link
Member

Also, /index/summary does not support POST, only GET, so the 403 is not from the WAF, but from API Gateway. So I'm not sure what you are testing for here. We discussed 5, 9, 10, 11 and 15 concurrent requests against the manifest endpoint with an inconsequential variation in the filters parameter so that you get different manifest keys and thereby subvert the caching algorithm. That's not at all what you are doing here.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 1259465 to caad149 Compare January 20, 2024 00:41
@dsotirho-ucsc
Copy link
Contributor Author

Also, /index/summary does not support POST, only GET, so the 403 is not from the WAF, but from API Gateway. So I'm not sure what you are testing for here. We discussed 5, 9, 10, 11 and 15 concurrent requests against the manifest endpoint with an inconsequential variation in the filters parameter so that you get different manifest keys and thereby subvert the caching algorithm. That's not at all what you are doing here.

The above test was run prior to deciding to run a test with concurrent requests.

While /index/summary doesn't support POST, it sufficed for the test since the WAF can still block these requests. Note the response from the initial request is

{"message":"Missing Authentication Token"}

and after the WAF starts blocking the requests, the response is

{"message":"Forbidden"}

Also, it's actually PUT, not POST we want to block as an expensive request. This will be fixed before the next test with concurrent requests.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 2 times, most recently from 2a95628 to 0139402 Compare January 22, 2024 23:18
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 0139402 to 13e373d Compare January 23, 2024 00:27
@dsotirho-ucsc
Copy link
Contributor Author

Test results making concurrent manifest generation requests with the manifest Lambda's Reserved concurrency set to 10.

5 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 1 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 6 seconds
Thread 1 got a 302, done.
Thread 4 got a 301, retrying after 6 seconds
Thread 5 got a 302, done.
Thread 3 got a 301, retrying after 6 seconds
Thread 2 got a 302, done.
Thread 4 got a 302, done.
Thread 3 got a 302, done.
9 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 9 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 8 got a 301, retrying after 4 seconds
Thread 9 got a 302, done.
Thread 4 got a 302, done.
Thread 2 got a 302, done.
Thread 3 got a 302, done.
Thread 1 got a 302, done.
Thread 7 got a 302, done.
Thread 5 got a 302, done.
Thread 6 got a 301, retrying after 6 seconds
Thread 8 got a 301, retrying after 6 seconds
Thread 6 got a 302, done.
Thread 8 got a 302, done.
10 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 10 making initial PUT request
Thread 2 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 8 got a 301, retrying after 4 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 10 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 2 got a 302, done.
Thread 8 got a 302, done.
Thread 10 got a 302, done.
Thread 4 got a 302, done.
Thread 6 got a 302, done.
Thread 3 got a 302, done.
Thread 9 got a 302, done.
Thread 1 got a 302, done.
Thread 7 got a 302, done.
Thread 5 got a 302, done.
11 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 10 making initial PUT request
Thread 11 making initial PUT request
Thread 9 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 3 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 5 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 11 got a 301, retrying after 4 seconds
Thread 10 got a 301, retrying after 4 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 8 got a 301, retrying after 4 seconds
Thread 4 got a 302, done.
Thread 1 got a 302, done.
Thread 7 got a 302, done.
Thread 11 got a 302, done.
Thread 5 got a 302, done.
Thread 10 got a 302, done.
Thread 2 got a 302, done.
Thread 6 got a 302, done.
Thread 8 got a 302, done.
Thread 9 got a 302, done.
15 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 10 making initial PUT request
Thread 11 making initial PUT request
Thread 12 making initial PUT request
Thread 13 making initial PUT request
Thread 14 making initial PUT request
Thread 15 making initial PUT request
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 15 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 14 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 14 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 15 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 1 seconds
Thread 1 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 14 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 15 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 14 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 13 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 4 seconds
Thread 11 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 15 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 5 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 8 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 10 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 13 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 4 got a 302, done.
Thread 6 got a 302, done.
Thread 14 got a 302, done.
Thread 3 got a 302, done.
Thread 7 got a 302, done.
Thread 12 got a 302, done.
Thread 11 got a 302, done.
Thread 9 got a 302, done.
Thread 2 got a 302, done.
Thread 15 got a 302, done.

@hannes-ucsc
Copy link
Member

It's impossible to reproduce your experiment without the source to the script that you presumably used. Please repeat the experiment with a simple command line that uses rush or gargs for running commands in parallel and curl for making the actual request. Prepare one file with 15 manifest URLs, one per line, and use head to limit the number of URLs that gets passed to gargs or rush. Post the file contents and the command line that was used. For one of the failed requests, please post relevant log entries from the service and manifest lambdas.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 6 times, most recently from 4c4bfc6 to 9b79b0a Compare January 30, 2024 17:33
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 3 times, most recently from 2d5008a to 3366611 Compare February 3, 2024 00:02
@dsotirho-ucsc
Copy link
Contributor Author

It's impossible to reproduce your experiment without the source to the script that you presumably used. Please repeat the experiment with a simple command line that uses rush or gargs for running commands in parallel and curl for making the actual request. Prepare one file with 15 manifest URLs, one per line, and use head to limit the number of URLs that gets passed to gargs or rush. Post the file contents and the command line that was used. For one of the failed requests, please post relevant log entries from the service and manifest lambdas.

Tested 15 concurrent manifest requests with the manifest Lambda concurrency set to 10.

manifest_urls.txt

https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10001000000%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000001%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000002%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000003%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000004%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000005%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000006%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000007%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000008%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000009%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000010%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000011%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000012%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000013%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000014%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000015%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000016%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000017%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000018%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000019%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000020%5D%5D%7D%7D&format=curl

Command line

head -n 15 manifest_urls.txt | xargs -P 15 -I % -S 512 python -c "from furl import furl; from integration_test import IndexingIntegrationTest as iit; from azul.logging import configure_script_logging; configure_script_logging(); t = iit(); t.setUp(); m = t._get_url_content('PUT', furl('%'))"

Terminal log

(abridged to show the all initial requests, and then the thread for 1 failed result)
(full terminal log: 5555-test-terminal-log.txt)

(.venv) daniel@Crispin ~/repo/azul1 $ head -n 15 manifest_urls.txt | xargs -P 15 -I % -S 512 python -c "from furl import furl; from integration_test import IndexingIntegrationTest as iit; from azul.logging import configure_script_logging; configure_script_logging(); t = iit(); t.setUp(); m = t._get_url_content('PUT', furl('%'))"
2024-02-02 11:17:37,682    INFO MainThread test.integration_test: Using random seed 9200391225568754709
2024-02-02 11:17:37,682    INFO MainThread test.integration_test: Using random seed 4956485098573584590
2024-02-02 11:17:37,682    INFO MainThread test.integration_test: Using random seed 8727454292430743500
2024-02-02 11:17:37,683    INFO MainThread test.integration_test: Using random seed 343936599250221468
2024-02-02 11:17:37,683    INFO MainThread test.integration_test: Using random seed 714364583535640742
2024-02-02 11:17:37,684    INFO MainThread test.integration_test: Using random seed 3746667822296243315
2024-02-02 11:17:37,683    INFO MainThread test.integration_test: Using random seed 7430512986588854954
2024-02-02 11:17:37,685    INFO MainThread test.integration_test: Using random seed 3706013619302031698
2024-02-02 11:17:37,685    INFO MainThread test.integration_test: Using random seed 8406687919479253520
2024-02-02 11:17:37,686    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000009%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,686    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000003%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,687    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000002%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,688    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000004%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,688    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000013%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,686    INFO MainThread test.integration_test: Using random seed 5971879927632929552
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Using random seed 2449699439315315317
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000005%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000001%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Using random seed 4220658395087540698
2024-02-02 11:17:37,693    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000008%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,684    INFO MainThread test.integration_test: Using random seed 2359823532252616098
2024-02-02 11:17:37,694    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000014%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,695    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000011%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,696    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000010%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,697    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10001000000%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,698    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000007%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,698    INFO MainThread test.integration_test: Using random seed 812345280441374550
2024-02-02 11:17:37,700    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000006%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,702    INFO MainThread test.integration_test: Using random seed 8019449667869935403
2024-02-02 11:17:37,705    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000012%5D%5D%7D%7D&format=curl'
…
2024-02-02 11:17:42,734    INFO MainThread test.integration_test: Got 301 response after 5.046s from PUT to https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000013%5D%5D%7D%7D&format=curl
2024-02-02 11:17:42,734    INFO MainThread test.integration_test: Sleeping 1.000s to honor Retry-After header
…
2024-02-02 11:17:43,736    INFO MainThread test.integration_test: Making GET request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ=='
…
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 795, in _get_url_content
    response = self._get_url(method, url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 812, in _get_url
    response = self._get_url_unchecked(method, url, stream=stream)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 840, in _get_url_unchecked
    response = http.request(method=method,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/request.py", line 77, in request
    return self.request_encode_url(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/request.py", line 99, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/src/azul/http.py", line 68, in urlopen
    response = super().urlopen(method, url, body=body, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/src/azul/http.py", line 42, in urlopen
    return self._inner.urlopen(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 894, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 894, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 894, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 884, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='service.daniel.dev.singlecell.gi.ucsc.edu', port=443): Max retries exceeded with url: /manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ== (Caused by ResponseError('too many 500 error responses'))
…

CloudWatch Log Insights (/aws/lambda/azul-service-daniel)

| filter @message like /START|END|Received|Returning/

[
    {
        "@timestamp": "2024-02-02 19:17:42.156",
        "@message": "START RequestId: bbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:42.157",
        "@message": "[INFO]\t2024-02-02T19:17:42.157Z\tbbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac\tazul.chalice\tReceived PUT request for '/manifest/files', with {\"query\": {\"catalog\": \"dcp3\", \"filters\": \"{\\\"fileSize\\\":{\\\"within\\\":[[1,10002000013]]}}\", \"format\": \"curl\"}, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"gbKNKEkubbCV8tXHfDBD3_d0V9wE6Rzq_ot1uQ5NpNY9esh5Fd-cRg==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd2-01f04c33326720b013c99034\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.154\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:42.766",
        "@message": "[DEBUG]\t2024-02-02T19:17:42.765Z\tbbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac\tazul.chalice\tReturning 301 response with headers {\"Location\": \"https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==\", \"Retry-After\": \"1\", \"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\nnull"
    },
    {
        "@timestamp": "2024-02-02 19:17:42.769",
        "@message": "END RequestId: bbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:43.876",
        "@message": "START RequestId: a4c11e0f-7273-4047-9b0e-e48a9ac28780 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:43.876",
        "@message": "[INFO]\t2024-02-02T19:17:43.876Z\ta4c11e0f-7273-4047-9b0e-e48a9ac28780\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"WKeYDnUxwOuawSOOCe-Lg3HuXBZ3D13DAxOD54DUDYa0YSTMg-Bz3Q==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd7-7f6601c96a399fa102e4fce0\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.106\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:43.911",
        "@message": "[DEBUG]\t2024-02-02T19:17:43.911Z\ta4c11e0f-7273-4047-9b0e-e48a9ac28780\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:43.913",
        "@message": "END RequestId: a4c11e0f-7273-4047-9b0e-e48a9ac28780\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.145",
        "@message": "START RequestId: 8da73087-4dd3-4018-ab8e-5f10a578c5f0 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.146",
        "@message": "[INFO]\t2024-02-02T19:17:44.146Z\t8da73087-4dd3-4018-ab8e-5f10a578c5f0\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"ViQlGZVfgAtGOEN6OZDK5laHXKryLwsr5O8ny6X-h7mgLtV1VZmG7w==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-396f8557067115166b3e2dc7\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.84\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.177",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.177Z\t8da73087-4dd3-4018-ab8e-5f10a578c5f0\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.179",
        "@message": "END RequestId: 8da73087-4dd3-4018-ab8e-5f10a578c5f0\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.378",
        "@message": "START RequestId: 75fae844-be68-4a5e-ae0f-8dfb9a6e32d9 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.378",
        "@message": "[INFO]\t2024-02-02T19:17:44.378Z\t75fae844-be68-4a5e-ae0f-8dfb9a6e32d9\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"aYs4D_kDPVQCh0k_EatqK5N9vttmbzS9dKaUrODgxzo3hDXNrOqIsA==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-1514d5b642dc81e70fc6a038\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.154\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.413",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.412Z\t75fae844-be68-4a5e-ae0f-8dfb9a6e32d9\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.414",
        "@message": "END RequestId: 75fae844-be68-4a5e-ae0f-8dfb9a6e32d9\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.547",
        "@message": "START RequestId: c0d907be-9972-49c5-8a8e-8173c811e8ff Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.548",
        "@message": "[INFO]\t2024-02-02T19:17:44.548Z\tc0d907be-9972-49c5-8a8e-8173c811e8ff\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"mGFP3_i2TW9UjYBkeCnMCuUZWUm7Db-y2JfG_lKCUvbq39iOUjE93g==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-73b89351210a323c48ebbc73\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.161\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.578",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.578Z\tc0d907be-9972-49c5-8a8e-8173c811e8ff\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.579",
        "@message": "END RequestId: c0d907be-9972-49c5-8a8e-8173c811e8ff\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.678",
        "@message": "START RequestId: 21856a47-aa92-46d7-827e-24b4cdfb0e3b Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.679",
        "@message": "[INFO]\t2024-02-02T19:17:44.679Z\t21856a47-aa92-46d7-827e-24b4cdfb0e3b\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"tmIyiwQANcLIywz52LtThsYPgiYxY6mvgPe0fd0AY733ljUMqUa8ig==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-5831b8676984d5f93ec19377\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.84\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.711",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.711Z\t21856a47-aa92-46d7-827e-24b4cdfb0e3b\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.712",
        "@message": "END RequestId: 21856a47-aa92-46d7-827e-24b4cdfb0e3b\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.801",
        "@message": "START RequestId: 5b2a0177-81ff-4437-9efc-32235082d651 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.802",
        "@message": "[INFO]\t2024-02-02T19:17:44.802Z\t5b2a0177-81ff-4437-9efc-32235082d651\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"X-iw6l7NuTxvbW2i33KdcAjXDZ3-tinmR0O7dwKKU-PI3Z7BAyLvvA==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-71be938c3105e75d6eec7d68\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.153\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.839",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.838Z\t5b2a0177-81ff-4437-9efc-32235082d651\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.840",
        "@message": "END RequestId: 5b2a0177-81ff-4437-9efc-32235082d651\n"
    }
]

(No logs for the throttled requests could be found in /aws/lambda/azul-service-daniel-manifest)

azul-service_manifest_throttles-daniel.alarm

Alarm notification sent to azul-group.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 3366611 to 15e202f Compare February 5, 2024 18:11
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 15e202f to 2e74dd5 Compare February 6, 2024 02:29
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 3 times, most recently from d071387 to 764d6af Compare March 21, 2024 23:52
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 2 times, most recently from 463f80c to 2afa30f Compare May 24, 2024 17:45
@hannes-ucsc hannes-ucsc force-pushed the develop branch 3 times, most recently from 0762297 to 3f7f6ce Compare May 28, 2024 06:58
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 2afa30f to 8abea24 Compare June 21, 2024 17:51
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 8abea24 to 2ffb642 Compare July 9, 2024 23:30
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 2ffb642 to ae31fa4 Compare January 29, 2025 18:23
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 4 times, most recently from 3c6f903 to 72df1e4 Compare February 7, 2025 01:22
@hannes-ucsc hannes-ucsc force-pushed the develop branch 2 times, most recently from 33a7a95 to 118ade8 Compare February 7, 2025 02:20
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 72df1e4 to b2b7076 Compare February 7, 2025 02:56
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from b2b7076 to 7c47ea6 Compare February 7, 2025 22:27
@dsotirho-ucsc
Copy link
Contributor Author

5869_IT_2025-02-07.txt

nadove-ucsc
nadove-ucsc requested changes Feb 8, 2025
View reviewed changes
Copy link
Contributor

@nadove-ucsc nadove-ucsc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, only nits

src/azul/__init__.py

waf_rate_rule_default = RateRule(name='rate_limit_default',
limit=1000,
period=300,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
period=300,
period=5 * 60,

etc

src/azul/__init__.py

assert 100 <= waf_rate_rule_limit <= 2_000_000_000 # mandated by AWS
def __attrs_post_init__(self):
assert 10 <= self.limit <= 2_000_000_000 # mandated by AWS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assert 10 <= self.limit <= 2_000_000_000 # mandated by AWS
assert 10 <= self.limit <= 2_000_000_000, self # mandated by AWS

src/azul/__init__.py
Comment on lines +1703 to +1705
# We are aiming for a global limit of 60 file downloads per 10 min. Based on
# an observed average of 2.9 concurrently downloading IPs in a average
# 10-minute window, the max limit we can allow is 20/10min, or 10/5min.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small tweaks

Suggested change
# We are aiming for a global limit of 60 file downloads per 10 min. Based on
# an observed average of 2.9 concurrently downloading IPs in a average
# 10-minute window, the max limit we can allow is 20/10min, or 10/5min.
# We aim for a global limit of 60 file downloads per 10 minutes. Based on
# an observed average of 2.9 distinct downloading IPs per 10-minute window,
# the maximum per-IP request rate we can allow is 20/10min, or 10/5min.

@nadove-ucsc nadove-ucsc removed their assignment Feb 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hannes-ucsc hannes-ucsc hannes-ucsc requested changes

@nadove-ucsc nadove-ucsc nadove-ucsc requested changes

@achave11-ucsc achave11-ucsc achave11-ucsc left review comments

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
4+ reviews [process] Lead requested changes four times or more orange [process] Done by the Azul team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@dsotirho-ucsc @coveralls @hannes-ucsc @achave11-ucsc @nadove-ucsc