Buffer size metrics showing negative values

[pranavmarla]

Details

I have Fluentd set up to forward logs to Kafka. In some of the Prometheus metrics for the buffers of those Kafka output plugins, I'm seeing negative values. Specifically, I'm seeing negative values for the following metrics:

• fluentd_output_status_buffer_total_bytes
• fluentd_output_status_buffer_stage_byte_size

Eg.

fluentd_output_status_buffer_total_bytes{worker_id="3",plugin_id="kafka_output",type="kafka2"} -7897834.0
fluentd_output_status_buffer_stage_byte_size{worker_id="3",plugin_id="kafka_output",type="kafka2"} -7897834.0

This appears to be the same issue as fluent/fluent-plugin-prometheus#55, open since 2018.

Prometheus Config

<source>
  @type prometheus
  port 24240
  metrics_path /metrics
  aggregated_metrics_path /aggregated_metrics
</source>

<source>
  @type prometheus_output_monitor
</source>

Env Info

• Fluentd: 1.11.2
• fluent-plugin-prometheus: 1.8.2
• fluent-plugin-prometheus_pushgateway: 0.0.2
• fluent-plugin-kafka: 0.14.1
• Note that Fluentd has multi-process workers enabled

[cristianpetrula]

I am experiencing the same issue with fluentd 1.14.6 and fluent-plugin-prometheus 2.0.2

[Watson1978]

Thank you for reporting this.

This plugin simply acts as a mirror: it reads the internal buffer metrics (such as buffer.stage_size or buffer.total_queued_size) directly from Fluentd core and exports them.
If these values are negative, it means Fluentd core's internal buffer calculations have drifted or become corrupted in memory.

[Watson1978]

Maybe, it is related to

• stage size computation in buffer plugin is not thread safe #2712