Split logs to accommodate exporter limitations

[brandonbirdj]

Component(s)

No response

Describe the issue you're reporting

Some exporters have different log entry size limits. Google Cloud Logging has a limit of 256KB and will issue the following error when breached.

Error

rpc error: code = InvalidArgument desc = Log entry with size 388.3K exceeds maximum size of 256.0K
error details: name = Unknown  desc = log_entry_errors:{key:16 value:{code:3 message:"Log entry with size 388.3K exceeds maximum size of 256.0K"}} log_entry_errors:{key:17 value:{code:3 message:"Log entry with size 323.0K exceeds maximum size of 256.0K"}}

To get around this error it's possible to use the filelogreceiver to split the log lines.

exporters:
  googlecloud:
    log: {}

receivers:
  filelog:
    include_file_path: true
    include:
      - /var/log/pods/*/*/*.log
    operators:
      - id: container-parser
        type: container
        max_log_size: 250KiB #limited for googlecloud logging exporter

service:
  pipelines:
    logs/googlecloud:
      exporters:
      - googlecloud #has 256 KiB log entry limit
      receivers:
      - filelog

However, there are many other receiver types which don't have the capability to split logs. The OTLP receiver allows setting max_recv_msg_size_mib however that would cause the log to simply not be sent, and doesn't allow for splitting.

exporters:
  debug:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ":4317"

service:
  pipelines:
    logs:
      exporters:
      - debug
      receivers:
      - otlp

One additional use-case could be using two exporters which allow for different sized logs. In the following example the smaller exporter googlecloud requires the logs be cut smaller than the the googlecloudpubsub exporter.

exporters:
  googlecloud:
    log: {}
  googlecloudpubsub:
    project: "${env:GOOGLE_PROJECT_ID}"
    topic: "projects/${env:GOOGLE_PROJECT_ID}/topics/my-topic"

receivers:
  filelog:
    include_file_path: true
    include:
      - /var/log/pods/*/*/*.log
    operators:
      - id: container-parser
        type: container
        max_log_size: 250KiB #limited for googlecloud logging exporter

service:
  extensions:
  - health_check
  pipelines:
    logs/googlecloud:
      exporters:
      - googlecloud #has 256 KiB log entry limit
      - googlecloudpubsub #has 10MiB limit, though can be compressed also
      receivers:
      - filelog

[atoulme]

Discussed on slack, maybe take a look at https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/logstransformprocessor