Skip to content

Name resolution delay - #12893

Open
AgraVator wants to merge 46 commits into
grpc:masterfrom
AgraVator:name-resolution-delay
Open

Name resolution delay#12893
AgraVator wants to merge 46 commits into
grpc:masterfrom
AgraVator:name-resolution-delay

Conversation

@AgraVator

@AgraVator AgraVator commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

This finishes the remaining work in #12807

AgraVator added 12 commits May 13, 2026 22:15
This commit implements the plumbing required to propagate delay reason tokens from load balancing policies up to the transport layer and tracers, as specified in the LB policy delay design.
…dence invariants

- Refactor ClientStreamTracer to expose delayTypeStarted(String) and delayReasonAttached(String)
- Enhance PickResult with separate delayType and delayReason diagnostic fields
- Implement Mark Roth's hybrid telemetry cadence model in DelayedClientTransport.PendingStream
- Support channel fallback delay states (client_channel_init, subchannel_state_mismatch, wait_for_ready_failed)
- Simplify leaf and container LB policies to emit canonical unified connecting metric labels
# Conflicts:
#	core/src/main/java/io/grpc/internal/PickFirstLeafLoadBalancer.java
@AgraVator
AgraVator marked this pull request as draft July 6, 2026 14:32
AgraVator added 13 commits July 27, 2026 06:59
…elay

# Conflicts:
#	core/src/main/java/io/grpc/internal/DelayedClientTransport.java
#	core/src/main/java/io/grpc/internal/PickFirstLoadBalancer.java
#	core/src/test/java/io/grpc/internal/DelayedClientTransportTest.java
#	opentelemetry/src/main/java/io/grpc/opentelemetry/GrpcOpenTelemetry.java
#	opentelemetry/src/main/java/io/grpc/opentelemetry/OpenTelemetryMetricsModule.java
#	opentelemetry/src/main/java/io/grpc/opentelemetry/OpenTelemetryMetricsResource.java
#	opentelemetry/src/main/java/io/grpc/opentelemetry/OpenTelemetryTracingModule.java
#	opentelemetry/src/test/java/io/grpc/opentelemetry/OpenTelemetryMetricsModuleTest.java
#	opentelemetry/src/test/java/io/grpc/opentelemetry/OpenTelemetryTracingModuleTest.java
#	rls/src/main/java/io/grpc/rls/CachingRlsLbClient.java
#	rls/src/test/java/io/grpc/rls/CachingRlsLbClientTest.java
#	util/src/main/java/io/grpc/util/RoundRobinLoadBalancer.java
#	util/src/test/java/io/grpc/util/RoundRobinLoadBalancerTest.java
#	xds/src/main/java/io/grpc/xds/CdsLoadBalancer2.java
#	xds/src/main/java/io/grpc/xds/PriorityLoadBalancer.java
#	xds/src/test/java/io/grpc/xds/CdsLoadBalancer2Test.java
#	xds/src/test/java/io/grpc/xds/PriorityLoadBalancerTest.java
…y observability (gRFC A66)

- Add nameResolutionDelay, lbPolicyDelay, and baselineNoDelay end-to-end tests to GrpcOpenTelemetryTest

- Update LoadBalancer.PickResult.withError to set delayType="connecting" and delayReason=error.getDescription()

- Fix missing static import checkstyle violation in CdsLoadBalancer2Test
…lay, remove debug prints, restore stress tests and add unit tests for patch coverage
…hStreamTracerFactory tests in LoadBalancerTest
@AgraVator
AgraVator requested a review from kannanjgithub August 3, 2026 14:37
@AgraVator
AgraVator marked this pull request as ready for review August 3, 2026 14:37
@AgraVator
AgraVator requested a review from shivaspeaks August 3, 2026 15:37
Comment thread core/src/test/java/io/grpc/internal/ForwardingClientStreamTracerTest.java Outdated
Comment thread opentelemetry/src/main/java/io/grpc/opentelemetry/OpenTelemetryMetricsModule.java Outdated
Comment thread opentelemetry/src/main/java/io/grpc/opentelemetry/OpenTelemetryMetricsModule.java Outdated
Comment thread opentelemetry/src/main/java/io/grpc/opentelemetry/OpenTelemetryTracingModule.java Outdated
Comment thread opentelemetry/src/main/java/io/grpc/opentelemetry/OpenTelemetryTracingModule.java Outdated
Comment thread opentelemetry/src/test/java/io/grpc/opentelemetry/GrpcOpenTelemetryTest.java Outdated
@kannanjgithub

Copy link
Copy Markdown
Contributor

Can you check the missing code coverages from the report

./gradlew :grpc-api:test :grpc-api:jacocoTestReport :grpc-core:test :grpc-core:jacocoTestReport :grpc-opentelemetry:test :grpc-opentelemetry:jacocoTestReport

and fix them?

@kannanjgithub kannanjgithub left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still some missing branches in codecov report. I have commented about missing coverages for OpenTelemetryMetricsModule.
Also address still existing coverages in the report except for cases where it cannot be done without changing class visibility or other reasons.

*/
synchronized void updateDelay(@Nullable String newType, @Nullable String newReason) {
if (getRealStream() != null) {
if (getRealStream() != null || delayEnded) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Branch getRealStream() == null && delayEnded == true is missing coverage. To cover it, a unit test only needs to invoke endDelay() directly on a pending stream while getRealStream() == null and then call updateDelay(...):

// Example in DelayedClientTransportTest.java
PendingStream pendingStream = ...;
pendingStream.endDelay(); // sets delayEnded = true while getRealStream() is still null
pendingStream.updateDelay("connecting", "new reason"); // triggers missing branch > returns

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets not do this, I didn't realize this change requires changing the class access modifier for PendingStream.

// Delay type changed (e.g., from RLS lookup to connecting). End the previous delay.
if (!Objects.equals(activeDelayType, newType)) {
// Delay type changed (e.g., from RLS lookup to connecting). End the previous delay.
if (activeDelayType != null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The is reported missing coverage for the branch when activeDelayType is null but it cannot be meaningfully tested because determineQueuingDelayType always returns non null value and there are other complications if we make this unconditional.

@Override
public synchronized void recordCallDelayStart(String delayType, String delayReason) {
if (!GrpcOpenTelemetry.isDelayObservabilityEnabled()
|| (activeCallDelayStopwatch != null && Objects.equals(activeCallDelayType, delayType))) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When activeCallDelayStopwatch != null is true, invoking recordCallDelayStart with a different delayType while an existing call delay is already active to transition between delay segments is not exercised.

We can cover this branch by adding a transition call with a different delay type (e.g. "connecting") to clientCallDelayDuration_duplicateStart_ignoresSecondStart in OpenTelemetryMetricsModuleTest.java:

  @Test
  public void clientCallDelayDuration_duplicateStart_ignoresSecondStart() {
    OpenTelemetryMetricsResource resource = GrpcOpenTelemetry.createMetricInstruments(
        openTelemetryTesting.getOpenTelemetry().getMeterProvider().get("grpc-java"),
        ImmutableMap.of("grpc.client.call.delay.duration", true),
        false);
    OpenTelemetryMetricsModule module = new OpenTelemetryMetricsModule(
        new FakeClock().getStopwatchSupplier(), resource, emptyList(), emptyList());
    OpenTelemetryMetricsModule.CallAttemptsTracerFactory factory =
        new OpenTelemetryMetricsModule.CallAttemptsTracerFactory(
            module, "target:///", CallOptions.DEFAULT, method.getFullMethodName(),
            emptyList(), io.opentelemetry.context.Context.root());

    factory.recordCallDelayStart("resolving", "first start");
    // Duplicate call with same delay type is ignored
    factory.recordCallDelayStart("resolving", "second start");

    // Transition to a different delay type while active (exercises Objects.equals == false)
    factory.recordCallDelayStart("connecting", "transition to connecting");
    factory.recordCallDelayEnd();

    assertThat(openTelemetryTesting.getMetrics())
        .anySatisfy(
            metric - assertThat(metric)
                .hasName("grpc.client.call.delay.duration"));
  }

public synchronized void recordCallDelayEnd() {
Stopwatch delayStopwatch = activeCallDelayStopwatch;
String delayType = activeCallDelayType;
if (delayStopwatch != null && delayType != null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the test for the null delay type branch, assert that no metric was recorded for null (e.g. assertThat(openTelemetryTesting.getMetrics()).extracting("name").doesNotContain("grpc.client.call.delay.duration") or verify that only points with valid non-null grpc.delay_type attributes exist).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants