boto3 paginator key listing runs very slow

[metazool]

We originally had s3fs interface to s3 object storage in this project.

It was replaced with the lower-level boto3 in this PR - for consistency with other projects / developer familiarity. It meant a one-line call to s3.ls() was replaced with an involved key listing method borrowed from SO

Now (with 10x more data but still only thousands of objects) it's returning very slowly, with s3.ls() still taking seconds - you can see an observable lag in each call to the paginator.

Is it worth digging into why (e.g. the possibility of request throttling at the JASMIN end, because it feels like this behaviour has changed?) or take the simplest course just to return to s3fs? I don't have expertise in s3