Normalizing Behavior of OTel Spans Created From Custom-Instrumentation and Trace Annotations #9759

mhlidd · 2025-10-14T20:31:58Z

What Does This Do

Currently, OTel Spans created from Trace Annotations do not contain the span.kind tag, and spans created from custom instrumentation do not have the type field. This leads to an inconsistent customer experience where spans are missing data on the backend, and also may have implications on the way that spans are "named" following normalization on the Agent. This PR aims to reduce the inconsistent behavior between these two ways of creating OTel spans.

Motivation

Escalation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

mhlidd · 2025-10-14T21:03:45Z

...src/main/java/datadog/trace/instrumentation/opentelemetry/annotations/WithSpanDecorator.java

  private static String convertToSpanType(SpanKind kind) {
    if (kind == null) {
      return null;
    }
    switch (kind) {
      case SERVER:
        return HTTP_SERVER;
      case CLIENT:
        return HTTP_CLIENT;
      case PRODUCER:
        return MESSAGE_PRODUCER;
      case CONSUMER:
        return MESSAGE_CONSUMER;
      case INTERNAL:
+        // checking for SpanKind.INTERNAL, returning DDSpanTypes.INTERNAL
+        return INTERNAL;
+      default:
+        return null;
+    }
+  }
+
+  private static String convertToSpanKindTag(SpanKind kind) {
+    if (kind == null) {
+      return null;
+    }
+    switch (kind) {
+      case CLIENT:
+        return SPAN_KIND_CLIENT;
+      case SERVER:
+        return SPAN_KIND_SERVER;
+      case PRODUCER:
+        return SPAN_KIND_PRODUCER;
+      case CONSUMER:
+        return SPAN_KIND_CONSUMER;
+      case INTERNAL:
+        return SPAN_KIND_INTERNAL;
      default:
        return null;
    }


This is currently duplicated code between the opentelemetry-shim and opentelemetry-annotation modules. @PerfectSlayer what do you think about introducing a new opentelemetry-utils module in utils for shared helpers? 🤔

Note that the otel-shim classes are already duplicated at build-time and placed under different packages - i.e. one source, but made available in two places in the final jar. This is to support both OTel drop-in support (which cannot expose the OTel packages on the classpath, so must repackage classes both at build-time and run-time) but also instrument OTel client code in the application.

I would investigate whether the annotations instrumentation can also depend on

implementation project(':dd-java-agent:agent-otel:otel-shim')

like the main OTel instrumentation under opentelemetry-1.4

PerfectSlayer

❔ question: ‏My main question for this change is should we change this behavior as it is not part of the spec, or should we evolve the spec and make sure the behavior is consistent across all languages? -- system tests could help here too.

Because I can think of case where customers set span type on purpose and it gets overridden when setting span kind.

PerfectSlayer · 2025-10-15T08:07:38Z

dd-trace-api/src/main/java/datadog/trace/api/DDSpanTypes.java

  public static final String CACHE = "cache";
  public static final String SOAP = "soap";

+  public static final String INTERNAL = "internal";


❔ question: ‏It looks like a duplicate of datadog.trace.bootstrap.instrumentation.api.Tags#SPAN_KIND_INTERNAL

PerfectSlayer · 2025-10-15T08:16:24Z

...ent/agent-otel/otel-shim/src/main/java/datadog/opentelemetry/shim/trace/OtelConventions.java

+   * @param kind The OpenTelemetry span kind to convert.
+   * @return The {@link DDSpanTypes} value.
+   */
+  public static String convertToSpanType(SpanKind kind) {


🎯 suggestion: To keep method naming consistent:‏

Suggested change

public static String convertToSpanType(SpanKind kind) {

public static String toSpanType(SpanKind kind) {

PerfectSlayer · 2025-10-15T08:17:18Z

...ent/agent-otel/otel-shim/src/main/java/datadog/opentelemetry/shim/trace/OtelConventions.java

+    if (kind == null) {
+      return null;
+    }


🎯 suggestion: ‏null is not supposed to be given as parameter as all calls are guarded.
Similarly, I would not have a default case returning null but internal.

mhlidd · 2025-10-15T19:35:29Z

❔ question: ‏My main question for this change is should we change this behavior as it is not part of the spec, or should we evolve the spec and make sure the behavior is consistent across all languages? -- system tests could help here too.

Because I can think of case where customers set span type on purpose and it gets overridden when setting span kind.

I think you are right that this change deviates from the spec. There are 2 concepts that are closely tied together, which makes this situation confusing. Below are the changes I think actually should be done:

Datadog has a type field on the Span, which is semantically equivalent to the kind field of an OTel span (or SpanKind). This has the values depicted in this API. We implement this mostly correctly in the tracer (minus adding in the INTERNAL case in a helper).
Datadog also has a span.kind tag that goes in meta. This maps the same SpanKind values to different tag values.

I looked into the OTel implementation of using a SpanBuilder to create an OTel span, and the behavior of setSpanKind in the Datadog shim is different than the OTel implementation. OTel sets the value passed in to the SpanKind field of the Span, while we currently set the span.kind tag instead. I think it makes sense here to align with OTel by modifying the setSpanKind function to set the type field of the span in our shim. This would also make the behavior identical with using OTel trace annotations in Datadog with code such as @WithSpan(kind=""), as we set the type field of the Span there as well.

What do you think @PerfectSlayer? I can see a world where we should keep the legacy behavior of setSpanKind setting the span.kind field to prevent breaking customer code, but I do think there is a need to align the behavior of the function.

PerfectSlayer · 2025-10-17T07:42:12Z

What do you think @PerfectSlayer? I can see a world where we should keep the legacy behavior of setSpanKind setting the span.kind field to prevent breaking customer code, but I do think there is a need to align the behavior of the function.

No I think it make sens to change to support both span kind and span type on OTel Tracing API and annotation support.

One thing that might be missing is the implementation of the span.type reserved attributes here. Can you add it to? (to call span.setType() on span.type attribute name).

mhlidd added 2 commits October 14, 2025 16:28

adding span.kind for Otel trace annotations

7f420c8

adding type field to custom instrumentation

aba17be

This comment has been minimized.

Sign in to view

mhlidd changed the title ~~adding span.kind for Otel trace annotations~~ Normalizing Behavior of OTel Spans Created From Custom Instrumentation and Trace Annotations Oct 14, 2025

mhlidd changed the title ~~Normalizing Behavior of OTel Spans Created From Custom Instrumentation and Trace Annotations~~ Normalizing Behavior of OTel Spans Created From Custom-Instrumentation and Trace Annotations Oct 14, 2025

mhlidd added type: bug Bug report and fix comp: core Tracer core inst: opentelemetry OpenTelemetry instrumentation labels Oct 14, 2025

mhlidd commented Oct 14, 2025

View reviewed changes

PerfectSlayer reviewed Oct 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Normalizing Behavior of OTel Spans Created From Custom-Instrumentation and Trace Annotations #9759

Normalizing Behavior of OTel Spans Created From Custom-Instrumentation and Trace Annotations #9759

mhlidd commented Oct 14, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

mhlidd Oct 14, 2025

Uh oh!

mcculls Oct 14, 2025

Uh oh!

PerfectSlayer left a comment

Uh oh!

PerfectSlayer Oct 15, 2025

Uh oh!

PerfectSlayer Oct 15, 2025

Uh oh!

PerfectSlayer Oct 15, 2025

Uh oh!

mhlidd commented Oct 15, 2025

Uh oh!

PerfectSlayer commented Oct 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	public static String convertToSpanType(SpanKind kind) {
	public static String toSpanType(SpanKind kind) {

Normalizing Behavior of OTel Spans Created From Custom-Instrumentation and Trace Annotations #9759

Are you sure you want to change the base?

Normalizing Behavior of OTel Spans Created From Custom-Instrumentation and Trace Annotations #9759

Conversation

mhlidd commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

This comment has been minimized.

mhlidd Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

mcculls Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

mhlidd commented Oct 15, 2025

Uh oh!

PerfectSlayer commented Oct 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhlidd commented Oct 14, 2025 •

edited

Loading