Global MeterFilters with dynamic content are bypassed by cache

[brunobat]

Describe the bug
We have a test that executes requests to the same endpoint but with different parameters and headers.
The parametric test runs 4 times and in the last execution it fails because the right meter, with the right tag is not present.

When executed alone, that test passes because there is no Micrometer caching involved.

The previous test will add a very similar meter that will be resolved in the cache here:

The preFilterIdToMeterMap in

micrometer/micrometer-core/src/main/java/io/micrometer/core/instrument/MeterRegistry.java

Line 627 in 9ffef08

Meter m = preFilterIdToMeterMap.get(originalId);

This gets returned and the global meter filters that would otherwise be applied in

micrometer/micrometer-core/src/main/java/io/micrometer/core/instrument/MeterRegistry.java

Line 632 in 9ffef08

Id mappedId = getMappedId(originalId);

... are never executed.

If one of those filters is a bean returning dynamic content, in this case, based on specifics of the request: https://github.com/quarkiverse/quarkus-cxf/blob/6cea9f2acb0127f3b4d17273578a3a4fc353ee27/integration-tests/hc5/src/main/java/io/quarkiverse/cxf/hc5/it/MeterFilterProducer.java#L12
Those tags are never applied and the wrong meeter is effectively returned.

See example:

Environment

x@x quarkus-cxf % mvn --version
Apache Maven 3.9.8 (36645f6c9b5079805ea5009217e36f2cffd34256)
Maven home: x/.sdkman/candidates/maven/current
Java version: 22.0.2, vendor: Eclipse Adoptium, runtime: x/.sdkman/candidates/java/22.0.2-tem
Default locale: en_PT, platform encoding: UTF-8
OS name: "mac os x", version: "13.6.9", arch: "aarch64", family: "mac"

Micromenter 1.13.3 on Quarkus main branch (unreleased).
Using the Prometheus simple client.

To Reproduce
The bug can be reproduced following the steps of this bug report: quarkusio/quarkus#42800

Expected behavior
If dynamic content can be added by global MeterFilters caching must not bypass them.

[brunobat]

Having a look at the code it seems that it's assumed that global MeterFilters only provide static data, however, that is not mentioned in the documentation: https://docs.micrometer.io/micrometer/reference/concepts/meter-filters.html
I wonder if this is a bad use of the API that seems ok if you read the docs.

Also, Wouldn't it be great to have static and dynamic meeter filters in order to better cache things?

[brunobat]

CC @jonatan-ivanov

[ppalaga]

@jonatan-ivanov it would be great if the non-volatility and cacheability requirements on MeterFilter methods could become a part of your official documentation. Otherwise it is very hard for us to explain to our users why this kind of use case stopped working after the upgrade to Micrometer 1.13

[johnnywiller]

We are running into the same issue, having HTTP headers (tenantIDs and other low cardinality IDs) that should be applied to all metrics as tags, but doing so in a decoupled way from where the metric is created.
Since what we provide is a spring boot starter for users just plug and have that behaviour.

[shakuzen]

I've opened #5809 to add the documentation mentioned in #5480, which hopefully explains the constraints that surprised users here as well as gives some direction on alternatives.

When executed alone, that test passes because there is no Micrometer caching involved.

That sounds like a combination of #5480 and #4187. Are the tests using the global registry? Maybe some of the things mentioned in there could help isolate the tests, depending on how things are setup.

Wouldn't it be great to have static and dynamic meeter filters in order to better cache things?

We could consider adding support for dynamic filters, but generally the recommendation is to make the dynamic aspects part of the instrumentation. Specific advice on doing this depends on the specific instrumentation, but we would be happy to take a look at instrumentation and advise on how that might look as much as we can.

@brunobat With the pending update to the documentation (feedback on the PR is welcome!) and the above explanation, should we consider this issue resolved, or would you like this to serve as the enhancement request for adding support for dynamic meter filters? We'd want to understand more about the use cases where it is difficult to do that as part of the instrumentation.

We are running into the same issue, having HTTP headers (tenantIDs and other low cardinality IDs) that should be applied to all metrics as tags

@johnnywiller where does the filter get the tenantID or any other dynamic tag values from?

[brunobat]

I think the explanation is good for now.
Dynamic filters should have their own issue, if you guys what to go ahead with them.
They would be interesting for data that you will only know at runtime, like instance ids or header data, and you need an already started container to get it.
Here is an example: https://github.com/quarkiverse/quarkus-cxf/pull/1497/files#diff-2e125881c34d9caae9ec8772a78fdf894eccb499718d888eedf691c3f2929429L12
Dynamic filters should not be cached, but rather than that could be exactly the same as the static ones.

[shakuzen]

They would be interesting for data that you will only know at runtime, like instance ids or header data, and you need an already started container to get it.

Header data would be dynamic per request, but instance ID is generally known on start-up and does not change over the lifetime of the instance. For those kinds of cases, they can be set with MeterFilter#commonTags (also as registry.config().commonTags).

[shakuzen]

Given the discussion and documentation update, we'll go ahead and close this issue for now. I'm still curious to hear about @johnnywiller's use case, though.