Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data mirroring #6856

Open
hannes-ucsc opened this issue Jan 28, 2025 · 0 comments
Open

Data mirroring #6856

hannes-ucsc opened this issue Jan 28, 2025 · 0 comments
Assignees
Labels
enh [type] New feature or request epic [type] Issue consists of multiple smaller issues mirror [subject] Code and infrastructure for mirroring data files orange [process] Done by the Azul team

Comments

@hannes-ucsc
Copy link
Member

We would like the ability for Azul to manage a mirror in AWS of most/all of the data file in a source that it indexes. A source is a TDR snapshot, for example. When a client requests a file from the /repository/files endpoint, Azul should respond with a signed URL of the mirrored file, if it exists. Manifests, especially the curl manifest should also use the mirror.

The mirroring should be done for all sources in a catalog.

Currently, the most pressing use case is mirroring HCA in AWS.

Design: https://docs.google.com/document/d/197_hBqv9f3RP9Crr6ISI3G0JjEECDVyrIlKICpruL1A/edit?tab=t.0

@hannes-ucsc hannes-ucsc added the epic [type] Issue consists of multiple smaller issues label Jan 28, 2025
@github-actions github-actions bot added the orange [process] Done by the Azul team label Jan 28, 2025
@hannes-ucsc hannes-ucsc added mirror [subject] Code and infrastructure for mirroring data files enh [type] New feature or request labels Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enh [type] New feature or request epic [type] Issue consists of multiple smaller issues mirror [subject] Code and infrastructure for mirroring data files orange [process] Done by the Azul team
Projects
None yet
Development

No branches or pull requests

1 participant