Support opentelemetry traces in json format from file

[gaurang101197]

Is your feature request related to a problem? Please describe.
We write application traces in file as json(schema: trace.json). I want to tail this trace file and ingest into opentelemetry collector using opentelemetry output plugin. When I try to tail this file, traces are ingested as logs instead of traces.

Sample trace.log file

{ "resourceSpans": [ { "resource": { "attributes": [ { "key": "service.name", "value": { "stringValue": "my.service" } } ] }, "scopeSpans": [ { "scope": { "name": "my.library", "version": "1.0.0", "attributes": [ { "key": "my.scope.attribute", "value": { "stringValue": "some scope attribute" } } ] }, "spans": [ { "traceId": "5B8EFFF798038103D269B633813FC60C", "spanId": "EEE19B7EC3C1B174", "parentSpanId": "EEE19B7EC3C1B173", "name": "I'm a server span", "startTimeUnixNano": "1544712660000000000", "endTimeUnixNano": "1544712661000000000", "kind": 2, "attributes": [ { "key": "my.span.attr", "value": { "stringValue": "some value" } } ] } ] } ] } ] }

fluent-bit.conf:

[SERVICE]
    Flush                1
    Log_level            debug

[INPUT]
    Name               tail
    Path                 ./trace.log
    Tag                  app_traces
    Read_from_head       true

[OUTPUT]
    Name                                opentelemetry
    Match                               *
    Host                                127.0.0.1
    Port                                4318
    Metrics_uri                         /v1/metrics
    Logs_uri                            /v1/logs
    Traces_uri                          /v1/traces
    Log_response_payload                True
    Tls                                 Off
    Tls.verify                          Off

Otel-collector.yml

receivers:
  otlp:
    protocols:
      http:
        endpoint: "127.0.0.1:4318"
      grpc:
        endpoint: "127.0.0.1:4317"

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  debug:
    verbosity: detailed
  file:
    path: ./otel-collector.json

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, file]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug] 

Describe the solution you'd like
Able to Read traces as json and ingest them to otel-collector. Something similar to opentelemetry_envelope which support converting trace json to opentelemetry schema which can be feed to opentelemetry output plugin.

[edsiper]

done for OpenTelemetry input:

#9938

this will be available in v3.2.7 (this week) and further versions

ah! you want to read from a file, #9938 supports over the network in the Opentelemetry input plugin.

Tail generate logs only and each line is a log line.

[gaurang101197]

you want to read from a file

Yes, I want to scrape traces printed as JSON from logs and ingest them to otel collector. Idea behind this is to use log for sending traces instead of sending traces over http call from application. We don't want to add extra http call to application for sending traces, instead application can print them as json logs and fluent bit pipeline can off load ingesting those traces from logs to otel collector.

[gaurang101197]

Looks like opentelemetry_envelope don't support traces as of now. It will be great if we add support of processing trace json.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.