Gauge double registration warning for Kafka consumer metrics

[MichaelVilser]

Since #5688 we observe warnings being logged for the default kafka consumer gauge metrics such as 'kafka.consumer.fetch.manager.bytes.consumed.rate' in the form of "This Gauge has been already registered".

Stack trace of when a gauge metric is registered
1stTime_on_startup.txt
2ndTime_after_startup.txt

Looking at this function, it seems like this behavior is actually intended:

micrometer/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka/KafkaMetrics.java

Lines 138 to 147 in 9d650ca

	@Override
	public void bindTo(MeterRegistry registry) {
	this.registry = registry;
	commonTags = getCommonTags(registry);
	prepareToBindMetrics(registry);
	checkAndBindMetrics(registry);
	scheduler.scheduleAtFixedRate(() -> checkAndBindMetrics(registry), REFRESH_INTERVAL_MILLIS,
	REFRESH_INTERVAL_MILLIS, TimeUnit.MILLISECONDS);
	}

[oliver-schwab]

We also see these warnings in our applications, but only in the ones with multiple consumers. Is this also intended?

[jonatan-ivanov]

Thanks for the issue!
KafkaMetrics needs to re-register Meters since the Kafka client pulls the rug and drops KafkaMetric instances (where it records the data) in some cases. Re-registration is expected but meters should be removed first so the gauge double-registration might indicate a bug.

A proper way to "fix" this would be moving the instrumentation into the Kafka Client itself, so we can instrument this natively and don't need to wrestle with the Kafka Client. Please comment this if you would like to see that happen: https://issues.apache.org/jira/browse/KAFKA-15191

Normally I would say that we have a bug and we don't remove something but based on your first attached file (1stTime_on_startup.txt) it seems this happens at startup so it is also possible that Kafka publishes something twice?

Does any of you have a minimal sample project to reproduce this issue?
Does this happen every time at startup or occasionally?
Does this happen only in case of kafka.consumer.fetch.manager.bytes.consumed.rate or other metrics are also affected?