PBM-1490: Enable Downloader for split-merge middleware
#1192
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Downloader public methods are merged with S3 storage.
Downloader public methods are merged with GCS storage.
Method returns download statistics. Currently it's only implemented for S3 and GCS. Other storaget types returns zero value.
boris-ilijic
changed the title
Downloader for split-merge middleware
boris-ilijic
force-pushed
the
PBM-1490-split-merge-mw-enable-downloader
branch
from
bc8d6ba to
b414148
Compare
boris-ilijic
force-pushed
the
PBM-1490-split-merge-mw-enable-downloader
branch
from
b414148 to
5ebc206
Compare
If maxObjSizeGB has invalid value (zero or less than zero) PBM will apply default value for it (the same as it is not specified withing config).
There was a problem hiding this comment.
Pull Request Overview
This PR enables the Downloader functionality for the split-merge middleware by implementing the DownloadStat() method across all storage implementations and refactoring the physical restore process to use a unified storage configuration approach.
- Adds
DownloadStat()method to the storage interface and implements it for all storage types - Creates
StorageWithDownloaderFromConfig()function to configure storage with downloader capabilities for S3 and GCS - Refactors physical restore to use the new unified storage configuration instead of type-specific downloader instantiation
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pbm/util/storage.go | Adds new function to create storage with downloader configuration for S3/GCS types |
| pbm/storage/storage.go | Extends storage interface with DownloadStat method |
| pbm/storage/split_merge_mw.go | Implements DownloadStat method for split-merge middleware |
| pbm/storage/s3/s3.go | Adds NewWithDownloader function and refactors downloader initialization |
| pbm/storage/s3/download.go | Consolidates download functionality and replaces interface{} with any |
| pbm/storage/gcs/gcs.go | Adds NewWithDownloader function for GCS storage |
| pbm/storage/gcs/download.go | Consolidates GCS download functionality |
| pbm/storage/fs/fs.go | Implements empty DownloadStat method for filesystem storage |
| pbm/storage/blackhole/blackhole.go | Implements empty DownloadStat method for blackhole storage |
| pbm/storage/azure/azure.go | Implements empty DownloadStat method for Azure storage |
| pbm/restore/physical.go | Refactors to use new unified storage configuration approach |
| pbm/restore/logical_cfgsvr_full_test.go | Implements DownloadStat method for test storage |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
inelpandzic
approved these changes
Sep 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is additional improvement for #1183:
Downloaderlogic for thesplit-mergemiddleware by implementing theDownloadStatmethod across all storages and providing constructor for downloader creation.maxObjSizeGBvalue in case of invalid configuration value (e.g. 0 or less than 0).