Skip to content

Conversation

@edsiper
Copy link
Member

@edsiper edsiper commented Jan 29, 2025

The following PR introduces ZSTD compression support in Fluent Bit:

  • adds zstd v1.56 (bundled library)
  • new flb_zstd.c compress/uncompress interface
  • internal interface unit tests
  • opentelemetry output: add support for compress: zstd
  • opentelemetry input: add support to detect and uncompress zstd

Test: Fluent Bit --> OpenTelemetry Collector

bin/fluent-bit -i dummy -p dummy="{\"zstd\": \"ok\"}" -o opentelemetry -p port=4318 -p compress=zstd

opentelemetry-collector:

2025-01-13T18:53:30.506-0600	info	Logs	{"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2025-01-13T18:53:30.506-0600	info	ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2025-01-14 00:53:29.832229967 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Map({"zstd":"ok"})
Trace ID: 
Span ID: 
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "debug"}

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

robskillington and others added 10 commits January 29, 2025 13:05
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
This patch adds a new compression mechanism called 'zstd', this is
available through the 'compress' configuration property, in the configuration file
it can be enabled with:

  pipeline:
     outputs:
      - name: opentelemetry
        match: '*'
        compress: zstd

Signed-off-by: Eduardo Silva <[email protected]>
@edsiper edsiper added this to the Fluent Bit v4.0.0 milestone Jan 29, 2025
@patrick-stephens patrick-stephens added the ok-package-test Run PR packaging tests label Jan 29, 2025
@edsiper edsiper merged commit 8808a93 into master Feb 11, 2025
107 of 113 checks passed
@edsiper edsiper deleted the lib-zstd-cleanup branch February 11, 2025 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-required ok-package-test Run PR packaging tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants