Skip to content

feat(gapic-generator): add support for resumable uploads - #18354

Draft
parthea wants to merge 41 commits into
mainfrom
feat/resumable-transfer-gapic-generator
Draft

parthea wants to merge 41 commits into
mainfrom
feat/resumable-transfer-gapic-generator

Conversation

@parthea

@parthea parthea commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Towards b/457416314, b/556259599

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for resumable upload methods in the GAPIC generator, updating schema wrappers, templates, and options to handle media upload protocols, while adding comprehensive system and unit tests. Feedback on the changes suggests avoiding potential breaking changes by generating media upload methods as standard RPCs when resumable_upload_prefix is not configured, rather than omitting them entirely. Additionally, it is recommended to deduplicate helper functions like resume_resumable_upload and make_resumable_upload across the newly added system test files by moving them to a shared utility module or conftest.py.

Comment thread packages/gapic-generator/gapic/schema/api.py Outdated
Comment thread packages/gapic-generator/tests/system/test_resumable_upload_basic.py Outdated
@parthea parthea changed the title DRAFT feat(gapic-generator): add support for resumable uploads feat(gapic-generator): add support for resumable uploads Sep 14, 2026
dependencies = [
"google-api-core[grpc] >= 2.28.0, <3.0.0",
# TODO: For prototyping purposes only, revert once https://github.com/googleapis/google-cloud-python/pull/18352 is merged
"google-api-core[grpc]@git+https://github.com/googleapis/google-cloud-python.git@feat/resumable-transfer-api-core#subdirectory=packages/google-api-core",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For prototyping purposes only, revert once #18352 is merged and released to PyPI

Comment thread .github/workflows/docs.yml Outdated
# and https://github.com/actions/checkout#checkout-head.
with:
fetch-depth: 2
fetch-depth: 0

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert this change once #18352 is merged to main and released

@daniel-sanche daniel-sanche left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to break some of this up into separate PRs? There's a lot here

# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it standard to include this text within the region tags of every sample? Is this because it's in showcase?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is standard. It exists on all generated samples. See details in go/snippet-gen-design


# Initialize request argument(s)
request = showcase_v1beta1.EnumRequest(
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to run ruff on these? The formatting seems unexpected

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't run ruff on golden files. It's a post processing step (post generation). There is a similar formatting issue in other goldens. I will file a bug to follow up on whether we want to run ruff on goldens but let's tackle that separately since it's an existing issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #18439 to add to our backlog to decide if we want to run ruff on golden files


# Iterate over the upload to receive progress updates as each chunk is transmitted
stream = io.BytesIO(b"Example upload data")
for progress in upload_session.iter_upload(stream):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it would be helpful to add a type annotation here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2bd3b15

retry: OptionalRetry = gapic_v1.method.DEFAULT,
timeout: Union[float, object] = gapic_v1.method.DEFAULT,
metadata: Sequence[Tuple[str, Union[str, bytes]]] = (),
) -> resumable_upload.UploadMediaResponse:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to have a different signature than tha samples (i.e. it doesn't accept config, doesn't return ResumableUploadSession)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 011dbe1

self.get_operation,
default_timeout=None,
client_info=client_info,
kind='rest',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't look right, this is the grpc_asyncio transport

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5919937

@parthea

parthea commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

I've addressed all of the review comments. I'll split this up into separate PRs to reduce the diff/make the review easier

@parthea
parthea force-pushed the feat/resumable-transfer-gapic-generator branch 2 times, most recently from cddfde0 to 65b0eed Compare September 24, 2026 16:40
@parthea
parthea force-pushed the feat/resumable-transfer-gapic-generator branch from fc9bdd2 to 7539d1d Compare September 24, 2026 20:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants