Skip to content

Separate weekly tasks for tarball-integrity checks. #8858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

isoos
Copy link
Collaborator

@isoos isoos commented Jul 18, 2025

  • Split integrity check into multiple process and run them independently. #8848
  • Refactored the IntegrityChecker to have a base functionality and split out tarballStorage-related checks. This reduces the time it takes to run the regular integrity checks, and it is also a natural split point for the future SQL migration.
  • Also added further cross-reference in the other direction: that each file in the canonical storage should have a matching database entry.
  • Note: I think the canonical bucket's cross-reference is a good test, but I'm not sure if and how many stale uploads we had that left an archive there. At least this will get us some insight about that through the logs.
  • We could also consider a similar scan for the public bucket, though it seems to be not critical, since the update of that bucket is always predicated on the canonical one.

@isoos isoos requested a review from sigurdm July 18, 2025 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant