Anonymous AWS Access

[adamnovak]

This should fix #5203 by adding a flag and environment variable for doing s3:// URL accesses anonymously.

I implemented the ability for job stores to provide command-line options to make this work. (Though with my later refactoring maybe I really need to move that to URL scheme implementations???)

To do this I needed to give URL accesses access to the Toil config, so I refactored them out of the AbstractJobStore. I also moved the implementations out of the AbstractJobStore and to their own interface (though the job store implementations still implement the various URL schemes by also implementing that interface).

I also had to adjust the CWL runner to prepare the RuntimeContext carefully for pickling, since now it needs a URL accessing widget, which might be the Toil object or the whole JobStore and shouldn't be moved from machine to machine. Now that's wrapped in a couple of functions that handle setting and clearing all the file access hooks, but to avoid thinking about different levels of context set-up-ness now every CWL job that uses a RuntimeContext sets up and tears down the while file streaming thread system. That code could probably be consolidated.

And I needed access to the config early on in the CWL runner to use it to access URLs, so I changed how the Toil object sets up the config so you can get it after construction and don't need to go into the context manager.

Changelog Entry

To be copied to the draft changelog by merger:

• Added --awsAnonymousUrlAccess/TOIL_AWS_ANONYMOUS_URL_ACCESS to allow accessing public data in S3 without logging in with an account that might need MFA or not grant access to it through IAM.

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
• Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.