Is your feature request related to a problem? Please describe.
Currently, Fluentd is only able to process ingesting logs type of events with the following grammar I guess:
Connection ::= <<Request>>*
Request ::= Message | Forward | PackedForward | nil
Message ::= [ Tag, Time, Record, Option? ]
Forward ::= [ Tag, MultiEventStream, Option? ]
MultiEventStream ::= [ Event* ]
PackedForward ::= [ Tag, MessagePackEventStream, Option? ]
MessagePackEventStream ::= <<Event>>*
Metadata ::= integer | EventTime
Event ::= [ Metadata, Record ]
Tag ::= string
Record ::= object
Option ::= object
But I Forward Protocol v1.5 defines metadata which is inside of the events like:
Metadata ::= integer | EventTime | [ EventTime, Hash* ]
Event ::= [ Metadata, Record ]
So, the supported grammar of ingesting records with Forward protocol in Fluentd should be:
Connection ::= <<Request>>*
Request ::= Message | Forward | PackedForward | nil
Message ::= [ Tag, Time, Record, Option? ]
Forward ::= [ Tag, MultiEventStream, Option? ]
MultiEventStream ::= [ Event* ]
PackedForward ::= [ Tag, MessagePackEventStream, Option? ]
MessagePackEventStream ::= <<Event>>*
Metadata ::= integer | EventTime | [ EventTime, Hash* ] # We need to support the 3rd part of metadata BNF
Event ::= [ Metadata, Record ]
Tag ::= string
Record ::= object
Option ::= object
This is because Fluent Bit core developers decided to send metadata inside of logs type of events by default.
So, we need to improve interoperability between Fluentd and Fluent Bit.
Describe the solution you'd like
Ideally, sending with metadata should work between Fluentd and Fluent Bit.
Describe alternatives you've considered
Currently, just turning off sending metadata that is inside of events with retain_metadata_in_forward_mode=false is able to send logs type of events from Fluent Bit to Fluentd.
Additional context
This is really needed feature because just turning off sending metadata losing the metadata information which is related to OpenTelemetry attributes in Fluent Bit because we're handling them within our mechanism of metadata handling.
Is your feature request related to a problem? Please describe.
Currently, Fluentd is only able to process ingesting logs type of events with the following grammar I guess:
But I Forward Protocol v1.5 defines metadata which is inside of the events like:
So, the supported grammar of ingesting records with Forward protocol in Fluentd should be:
This is because Fluent Bit core developers decided to send metadata inside of logs type of events by default.
So, we need to improve interoperability between Fluentd and Fluent Bit.
Describe the solution you'd like
Ideally, sending with metadata should work between Fluentd and Fluent Bit.
Describe alternatives you've considered
Currently, just turning off sending metadata that is inside of events with
retain_metadata_in_forward_mode=falseis able to send logs type of events from Fluent Bit to Fluentd.Additional context
This is really needed feature because just turning off sending metadata losing the metadata information which is related to OpenTelemetry attributes in Fluent Bit because we're handling them within our mechanism of metadata handling.