support of dynamic tags where the value can change during runtime

[jackhammer2k]

I have a feature request for dynamic tag value evaluation. Currently the values of tags must be specified at creation time. However there are cases where they change during runtime. Specifically in my case I want to identify the leader of several instances with a tag. I want to add that tag as a common tag, because it applies to all meters. Because the tag value can change during runtime I cannot specify it as common tag at creation time.

I'm thinking about something like this:

public interface Tag extends Comparable<Tag> {
[...]
    static Tag of(String key, Supplier<String> valueSupplier) {
        return new ImmutableTag(key, valueSupplier);
    }
[...]
}

public class ImmutableTag implements Tag {
[...]
    @Override
    public String getValue() {
        return valueSupplier.get();
    }
[...]
}

I would also try to implement that feature, but before I want to be sure it has good chances to be integrated.

[SimonScholz]

I'd love to have such a feature as well. Because unfortunately the WebMvcTagsProvider, which can be used to apply custom tags depending on the request and response values, only applies for timer instances and not for regular Counter instances.

[jkschneider]

@jackhammer2k Where this gets really complicated is Micrometer maintains an internal map of which meters have already been registered, so if you repeatedly call registry.timer("my.timer", "key", "value") you get the same Timer instance back.

With dynamic tagging, there is this problem that what was previously two meters suddenly becomes one (e.g. a dynamically determined tag key key now collides with a statically determined key key on another meter). Conversely, a single meter may split in two.

So the implementation isn't so straightforward. Open to ideas.

[SimonScholz]

@jkschneider Well the key itself will be static in @jackhammer2k 's proposal. It's just the value that changes.
We have some common tags we'd like to apply for each meter, but the value of the common tag would change during runtime.
Passing these common tag values all the time is really inconvenient!

[shakuzen]

We have some common tags we'd like to apply for each meter, but the value of the common tag would change during runtime.

I'm curious to understand more of these use cases. What kind of dynamic value common tags are you using?

[SimonScholz]

We have some common tags we'd like to apply for each meter, but the value of the common tag would change during runtime.

I'm curious to understand more of these use cases. What kind of dynamic value common tags are you using?

Thanks for having a look, Tommy.

We have some meta data for each and every request as request headers, which we want to add as tags to each and every metric.
The meta data is country [germany, italy, spain, etc] and captureChannel [APP, MOBILE, DESKTOP], which we receive from our clients.
In Grafana we'd like to be able to filter our dashboards by these values for each and every metric.

We also already utilize the DefaultWebMvcTagsProvider class to adjust this meta data to metrics like http_server_requests_seconds_count, but as mentioned before we want to have this meta data everywhere.

We even want it for third party libraries, which create timers and counters themselves, like resilience4j does. But resilience4j does not offer any extension to add dynamic/additional tags either. See https://github.com/resilience4j/resilience4j

Another point is that the mentioned meta data is quite new, which means that we would need to touch each and every existing counter and timer in our code, which would be a huge amount of effort compared to having dynamic common tags, which we could simply apply for each and every metric.

Does this answer your question Tommy?

Not sure whether also @jackhammer2k also want's to share his reason why he wants to have dynamic tag values.

[checketts]

Sounds like @SimonScholz usecase would need those dynamic tags calculated based on the current thread, since values would be different between requests.

Would a MeterFilter that does that dynamic lookup suffice?

[SimonScholz]

Let me check whether I can utilize a MeterFilter for my purpose by using the io.micrometer.core.instrument.Meter.Id#withTags method.

[jackhammer2k]

So MeterFilter also offers the ability to change common tags dynamically, interesting. If I understand it right the difference between the approaches is:

• MeterFilter creates a new metric (tag is part of ID) everytime the value changes and the old metrics still continue to exist, but are not changed anymore. This can cause issues with gauges when querying metrics.
• The tag supplier leads to just one metric that changes its tag value over time.

[jackhammer2k]

As I explained in my initial post we have several instances but just one of them is the leader. The leader is dynamically elected during runtime and can change during runtime. In our Grafana dashboard we're only interested in the metrics of the leader at each specific point in time. I see several solutions there:

1. manually select leader instance in dashboard and filter for that instance in each query
2. add extra metric pointing out if instance is leader (0 or 1) and combine with logical AND with each metric in graph
3. add static tag (leader=true/false) to each metric
4. add dynamic tag (leader=true/false) to each metric via common tags and filter for that tag in each query
5. add static tag (leader=true/false) to each metric via MetricFilter
6. remove all metrics from metric registry if instance looses leadership, only register metrics when is leader

Every approach has some drawbacks. I also like 6. which is quite simple as long as there is not a mix of leader-specific-metrics and common ones. However in my opinion 4. fits better into the general concept.

[tbhaagh]

I am happy to see this going on the backlog, as we would like to tag metrics with enabled feature flags, so we can observe the progressive delivery of features in a multi-tenant system.

[giteshjha]

#535 (comment)

[rborale]

Hi guys,
what if I use something like this,

    meterRegistry.counter("metric_name",
            "tag1", "val11",
            "tag2", "val2").increment();
    meterRegistry.counter("metric_name",
            "tag1", "val1",
            "tag2", "val2").increment();

Here, I will be calling meterRegistry.counter for every increment of the counter.
Whenever value of any tag changes, (like here tag1) it initiates the new counter. Otherwise increment the value of existing one.

[wentfar]

Is this feature commited? I‘ve met the same scenario processing dynamic tag values;