Not possible to add a metric to same namespace

[panduz86]

Hello,

we open this ticket because we noticed that after the first creation of a MetricSystem, then it's not possible anymore to add a new metric inside.

For example we have this code

val METRICS_NAMESPACE = "appnamespace"
val METRIC_COUNTER_CASES = "CounterCases"

val MetricsSystem = UserMetricsSystems.getMetricSystem(METRICS_NAMESPACE, builder => { 
  builder.registerCounter(METRIC_COUNTER_CASES)
})

MetricsSystem.counter(METRIC_COUNTER_CASES).inc(1L)

I execute the code above and then I update it for adding a new metric under the same namespace, like this:

val METRICS_NAMESPACE = "appnamespace"
val METRIC_COUNTER_CASES = "CounterCases"
val METRIC_COUNTER_ERRORS = "CounterErrors"

val MetricsSystem = UserMetricsSystems.getMetricSystem(METRICS_NAMESPACE, builder => { 
  builder.registerCounter(METRIC_COUNTER_CASES)
  builder.registerCounter(METRIC_COUNTER_ERRORS)
})

MetricsSystem.counter(METRIC_COUNTER_CASES).inc(1L)
MetricsSystem.counter(METRIC_COUNTER_ERRORS).inc(1L)

when I execute it, I get this exception:

Metric 'CounterErrors' in namespace appnamespace was not found

so basically the new counter has not been created and we think that the reason is that UserMetricsSystems.getMetricSystem doesn't execute the builder metric if the namespace already exists (because of the getOrElseUpdate method called here: https://github.com/mspnp/spark-monitoring/blob/main/src/spark-listeners/src/main/scala/org/apache/spark/metrics/UserMetricsSystem.scala#L33) .

Could you please tell us how to manage this?

Thank you

Paolo