Skip to content

reusable_upload-file-s3: increase multipart chunk size - #147

Merged
Ansuel merged 1 commit into
openwrt:mainfrom
BKPepe:ccache-upload-single-request
Nov 15, 2026
Merged

Ansuel merged 1 commit into
openwrt:mainfrom
BKPepe:ccache-upload-single-request

Conversation

@BKPepe

@BKPepe BKPepe commented Nov 12, 2026

Copy link
Copy Markdown
Member

By default, aws s3 cp uses 8 MiB multipart parts, so each 0.8 to 2.4 GB
ccache archive generates 92 to 289 billable write requests.

Use 256 MiB parts, which brings that down to 3 to 10. The largest archive
still fits the 10 requests the AWS CLI sends at once, so uploads stay
parallel and a failed part is retried alone.

A retried part now resends up to 256 MiB instead of 8 MiB.

Tested with AWS CLI 2.40.2 and S3Mock: a 512 MB file went up in two parts
and a 100 MiB file in a single PutObject. Not yet tested against R2.

cc: @Ansuel

aws s3 cp uses 8 MiB parts by default, resulting in 92 to 289
write requests for each ccache archive.

Use 256 MiB parts instead, reducing uploads to 3 to 10 requests
while keeping retries limited to individual parts.

Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
@Ansuel
Ansuel merged commit 1f0392f into openwrt:main Nov 15, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants