[oss]support cross bucket migration of oss files - #9606
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
This unlocks a real Hive-to-Paimon migration path when the source and destination use different OSS buckets in the same region. I traced the migration caller, destination checks, directory copies, multipart fallback and delete-after-copy behavior. One large-file boundary needs fixing before this supports the stated migration path reliably; see the inline comment.
Validation: all 26 OSSFileIOTest tests passed against this head. An additional boundary probe executed the PR's multipart loop with mocked successful copies and the real OSS SDK 3.17.4 validation for the failing part: 83,886,080,000 bytes completed in 10,000 parts, while one additional byte attempted part 10,001, threw IllegalArgumentException and aborted the upload. No large object was allocated or uploaded, and no live OSS/Hive migration was run.
| List<PartETag> partETags = new ArrayList<>(); | ||
| try { | ||
| long partSize = CROSS_BUCKET_COPY_PART_SIZE; | ||
| long remaining = contentLength; |
There was a problem hiding this comment.
[P2] Size copy parts to stay within OSS's 10,000-part limit
Using a fixed 8 MiB part size makes every object larger than 83,886,080,000 bytes (78.125 GiB) fail this new cross-bucket migration path. Such an object must use multipart copy; after 10,000 successful parts this loop sends part 10,001, which OSS SDK 3.17.4 rejects locally with IllegalArgumentException: The part number should be in the range of [1-10000]. I reproduced the exact boundary with this method: the threshold completes, threshold + 1 byte aborts. The source is preserved, but the migration fails after copying the preceding parts. Compute the part size from contentLength and the maximum part count, respecting the service's per-part bounds, and add this boundary to the multipart tests.
Purpose
The current implementaion of the oss migration only support the same bucket. So you can't do a migration of a hive table from the bucket
ato a paimon table at the bucketb. The limitation is due to theAliyunOssFileSystem'srenameimplementation. It hard codes the source and target bucket to the same one. But the offical document of the aliyun oss doesn't have this limitation.So this pr is to overload the
renamefunction to support the cross bucket data movement.Tests
OSSFileIOTest's mock test cases passed. And I test this by the real aliyun AK.