Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create mirror queues and bucket #6859

Open
hannes-ucsc opened this issue Jan 29, 2025 · 1 comment
Open

Create mirror queues and bucket #6859

hannes-ucsc opened this issue Jan 29, 2025 · 1 comment
Assignees
Labels
+ [priority] High enh [type] New feature or request mirror [subject] Code and infrastructure for mirroring data files orange [process] Done by the Azul team

Comments

@hannes-ucsc
Copy link
Member

hannes-ucsc commented Jan 29, 2025

Create the cloud infrastructure needed for mirroring, excluding any Lambda functions, most prominently the mirror queue (FIFO), the corresponding fail queue and the actual mirror bucket. The bucket should be non-public for now, matching the default access for all of our buckets. Ensure existing functionality isn't impacted by that infrastructure. Extend the manage_queues.py script to be able to handle the new queue. Resetting the indexer shouldn't affect the mirror queues.

@github-actions github-actions bot added the orange [process] Done by the Azul team label Jan 29, 2025
@hannes-ucsc hannes-ucsc added mirror [subject] Code and infrastructure for mirroring data files + [priority] High enh [type] New feature or request labels Jan 29, 2025
@dsotirho-ucsc
Copy link
Contributor

@hannes-ucsc: "The specifics of the S3 mirror bucket are still being discussed. It is clear that it will have to reside in a separate AWS account, but ownership and control of that account is yet to be determined. In lower deployments we will have to use a non-ODP bucket anyways. Assignee to implement this issue for lower deployments where the mirror bucket resides in the same account as the reset of the infrastructure and the bucket should be part of the main component. Whether mirroring is enabled or not should be configurable. By default, it should be disabled. Assignee to also update the prototype (#6857) to mirror to that new bucket."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
+ [priority] High enh [type] New feature or request mirror [subject] Code and infrastructure for mirroring data files orange [process] Done by the Azul team
Projects
None yet
Development

No branches or pull requests

3 participants