Skip to content

Bug: Invalid dict type used for span attributes causes telemetry export failures #4094

@democodex

Description

@democodex

Description

When ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS is set to false (or OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT is disabled), the ADK's telemetry/tracing.py sets span attributes to empty Python dicts {} instead of valid OpenTelemetry attribute types.

OpenTelemetry span attributes can only be primitive types (bool, str, bytes, int, float) or sequences of those types. Setting a dict as an attribute value causes:

  1. Warnings in the logs:

    Invalid type dict for attribute 'gcp.vertex.agent.llm_response' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
    Invalid type dict for attribute 'gcp.vertex.agent.llm_request' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
    Invalid type dict for attribute 'gcp.vertex.agent.tool_response' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
    Invalid type dict for attribute 'gcp.vertex.agent.tool_call_args' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
    
  2. Crashes in the opentelemetry-exporter-gcp-logging when trying to serialize spans:

    File ".../opentelemetry/exporter/cloud_logging/__init__.py", line 190, in _set_payload_in_log_entry
        struct.update(sanitized)
    ...
    ValueError: Unexpected type
    
  3. No traces appearing in Vertex AI Agent Engine's trace viewer, as the telemetry export fails silently.

Location of Bug

google/adk/telemetry/tracing.py - multiple locations:

  • Line 152: span.set_attribute('gcp.vertex.agent.tool_call_args', {})
  • Line 182: span.set_attribute('gcp.vertex.agent.tool_response', {})
  • Line 268: span.set_attribute('gcp.vertex.agent.llm_request', {})
  • Line 293: span.set_attribute('gcp.vertex.agent.llm_response', {})
  • Line 349: span.set_attribute('gcp.vertex.agent.data', {})

Suggested Fix

Change all instances of {} to '{}' (a JSON string representation of an empty object):

# Before (broken)
span.set_attribute('gcp.vertex.agent.tool_call_args', {})

# After (fixed)
span.set_attribute('gcp.vertex.agent.tool_call_args', '{}')

Environment

  • ADK Version: 1.21.0
  • OpenTelemetry SDK: 1.37.0
  • opentelemetry-exporter-gcp-logging: 1.11.0a0
  • Deployment: Vertex AI Agent Engine (Reasoning Engine)

Steps to Reproduce

  1. Deploy an ADK agent to Vertex AI Agent Engine with telemetry enabled (GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true)
  2. Ensure ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS is false or not capturing content
  3. Run a query against the agent
  4. Check Cloud Logging for the warnings and ValueError: Unexpected type errors
  5. Observe that traces do not appear in the Vertex AI trace viewer

Metadata

Metadata

Assignees

No one assigned

    Labels

    tracing[Component] This issue is related to OpenTelemetry tracing

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions