Skip to content

[oss]support cross bucket migration of oss files - #9606

Open
weijietong wants to merge 2 commits into
apache:masterfrom
weijietong:migrate_oss
Open

[oss]support cross bucket migration of oss files#9606
weijietong wants to merge 2 commits into
apache:masterfrom
weijietong:migrate_oss

Conversation

@weijietong

@weijietong weijietong commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Purpose

The current implementaion of the oss migration only support the same bucket. So you can't do a migration of a hive table from the bucket a to a paimon table at the bucket b. The limitation is due to the AliyunOssFileSystem's rename implementation. It hard codes the source and target bucket to the same one. But the offical document of the aliyun oss doesn't have this limitation.

So this pr is to overload the rename function to support the cross bucket data movement.

Tests

OSSFileIOTest's mock test cases passed. And I test this by the real aliyun AK.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

This unlocks a real Hive-to-Paimon migration path when the source and destination use different OSS buckets in the same region. I traced the migration caller, destination checks, directory copies, multipart fallback and delete-after-copy behavior. One large-file boundary needs fixing before this supports the stated migration path reliably; see the inline comment.

Validation: all 26 OSSFileIOTest tests passed against this head. An additional boundary probe executed the PR's multipart loop with mocked successful copies and the real OSS SDK 3.17.4 validation for the failing part: 83,886,080,000 bytes completed in 10,000 parts, while one additional byte attempted part 10,001, threw IllegalArgumentException and aborted the upload. No large object was allocated or uploaded, and no live OSS/Hive migration was run.

List<PartETag> partETags = new ArrayList<>();
try {
long partSize = CROSS_BUCKET_COPY_PART_SIZE;
long remaining = contentLength;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Size copy parts to stay within OSS's 10,000-part limit

Using a fixed 8 MiB part size makes every object larger than 83,886,080,000 bytes (78.125 GiB) fail this new cross-bucket migration path. Such an object must use multipart copy; after 10,000 successful parts this loop sends part 10,001, which OSS SDK 3.17.4 rejects locally with IllegalArgumentException: The part number should be in the range of [1-10000]. I reproduced the exact boundary with this method: the threshold completes, threshold + 1 byte aborts. The source is preserved, but the migration fails after copying the preceding parts. Compute the part size from contentLength and the maximum part count, respecting the service's per-part bounds, and add this boundary to the multipart tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants