OCPBUGS-87180: [release-4.19] catalogd: clean up orphaned temp dirs in catalog storage#748
Conversation
…log storage
LocalDirV1.Store creates a temp dir (.{catalog}-{random}) and renames it
into place atomically. If the process is interrupted before the deferred
RemoveAll runs, the temp dir persists. Each restart adds another, eventually
filling the disk.
Primary fix: remove any matching temp dirs at the start of Store before
creating a new one. Since Store holds the write mutex, any dirs found are
guaranteed to be from a previous run.
Defense-in-depth: add /var/cache/catalogs to GarbageCollector's list of
scanned paths (CachePath string -> CachePaths []string), so orphaned temp
dirs are also removed by the periodic GC cycle.
Backport of OCPBUGS-76453 to release-4.19.
Signed-off-by: Rashmi Gottipati <rgottipa@redhat.com>
openshift-ci-robot
added
jira/severity-moderate
|
@rashmigottipati: This pull request references Jira Issue OCPBUGS-87180, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/jira refresh |
|
@rashmigottipati: This pull request references Jira Issue OCPBUGS-87180, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@rashmigottipati: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
oceanc80
left a comment
oceanc80
left a comment
There was a problem hiding this comment.
/lgtm
/label backport-risk-assessed
openshift-ci
Bot
added
the
backport-risk-assessed
|
Should we close #747? |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rashmigottipati, tmshort The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
5 participants
Backport of OCPBUGS-76453 to release-4.19.
LocalDirV1.Store creates a temp dir (.{catalog}-{random}) and renames it into place atomically. If the process is interrupted before the deferred RemoveAll runs, the temp dir persists. Each restart adds another, eventually filling the disk.
Primary fix: remove any matching temp dirs at the start of Store before creating a new one. Since Store holds the write mutex, any dirs found are guaranteed to be from a previous run.
Defense-in-depth: add /var/cache/catalogs to GarbageCollector's list of scanned paths (CachePath string -> CachePaths []string), so orphaned temp dirs are also removed by the periodic GC cycle.
Reference - Previous PRs:
4.21 PR: #742
4.20 PR: #744