Add retry for GCE 403 concurrent operations quota errors

[golgeek]

GCE returns HTTP 403 (not 429) when the concurrent operations quota is exceeded.
The error includes reason CONCURRENT_OPERATIONS_QUOTA_EXCEEDED in the error details.
This is a transient condition that clears as soon as in-flight operations complete, but the provider currently treats it as a hard failure.

This adds a new global retry predicate is403ConcurrentOperationsQuotaError that matches 403 errors with CONCURRENT_OPERATIONS_QUOTA_EXCEEDED in the ErrorInfo details and retries them automatically.

Fixes hashicorp/terraform-provider-google#9207

Release Note Template

transport: Added automatic retry for GCE 403 errors with reason `CONCURRENT_OPERATIONS_QUOTA_EXCEEDED`

[github-actions]

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@trodge, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 85 insertions(+))
google-beta provider: Diff ( 2 files changed, 85 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 34 insertions(+))

[trodge]

/gcbrun

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 85 insertions(+))
google-beta provider: Diff ( 2 files changed, 85 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 34 insertions(+))

[modular-magician]

Tests analytics

Total tests: 6134
Passed tests: 5493
Skipped tests: 638
Affected tests: 3

Click here to see the affected service packages

All service packages are affected

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccComputeResourcePolicy_resourcePolicyWorkloadPolicyAcceleratorTopologyModeExample
• TestAccComputeResourcePolicy_withTopologyMode
• TestAccGKEHubFeatureMembership_gkehubFeatureAcmUpdate

Get to know how VCR tests work

[modular-magician]

🟢 Tests passed during RECORDING mode:
TestAccComputeResourcePolicy_resourcePolicyWorkloadPolicyAcceleratorTopologyModeExample [Debug log]
TestAccComputeResourcePolicy_withTopologyMode [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

---

🔴 Tests failed during RECORDING mode:
TestAccGKEHubFeatureMembership_gkehubFeatureAcmUpdate [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[golgeek]

Thanks for looking at the PR and triggering the tests, @trodge!

It looks like the TestAccGKEHubFeatureMembership_gkehubFeatureAcmUpdate failing is unrelated to this change, and that it might be a known flaky test?

Let me know if you need me to do anything!