Add "retention" feature allowing idle metrics to expire

[phihos]

As proposed in #20 I implemented a way to auto remove idle metrics at runtime without needing to restart.
Example:

<metric>
  name message_foo_counter
  type counter
  desc The total number of foo in message.
  key foo
  retention 3600 # 1h
  retention_check_interval 1800 # 30m
  <labels>
    bar ${bar}
  </labels>
</metric>

If ${bar} was baz one time but after that no records with that value were processed, then after one hour the metric
foo{bar="baz"} might be removed.
When this actually happens depends on retention_check_interval (default 60).
It causes a background thread to check every 30 minutes for expired metrics.
So worst case the metrics are removed 30 minutes after expiration.

The naming of the config keys were shamelessly stolen from inspired by grok_exporter to make this feature more familiar to people using the grok_exporter.

Additional to the implementation I had to refactor the Metrics class to directly implement instrument(record, expander) and put subclass-specific logic into value(record), set_value?(value) and set_value(value, labels).
That reduces code duplication and was necessary for not introducing further duplicates.

I also had to introduce a new data store based on the default data store of prometheus/client_ruby to allow for removal of elements.

The last thing I want to mention is that I need to use the thread helper to start cleaning expired metrics in the background. I first tried to use the timer helper but it caused a test to go into an infinite loop.

If you need more tests or need other alterations to the code please let me know. I am looking forward to your feedback 🙂

[ashie]

Sorry for my late response.
Could you resolve conflict?

[ashie]

Is it really need to use thread?
I'm not sure but timer_execute (run it in plugin thread) might be enough to do it.

[micaelmbagira]

@phihos any update?

[Watson1978]

Seems we need #237 fix to use timer plugin helper.

[Watson1978]

I replaced to timer at 0261ea6

[phihos]

Sorry I had two very busy weeks. I hope I get to it tomorrow.

[gromnsk]

@phihos any update?

[AlbusLumos]

@phihos is this still on-going?

[dkulchinsky]

Hey @phihos 👋🏼 wondering if you still have cycles to work on this? I can try and take over though not as experienced with Ruby.

Looks like there was mainly a concern whether a dedicated thread is needed, instead of running it in the plugin thread.

[Lusitaniae]

Wonder if you had the bandwidth to pick this up again @phihos

🙏

[Watson1978]

I solved the conflict.
The original PR is backed up at https://github.com/Watson1978/fluent-plugin-prometheus/pull/1/files

Can I leave this to @daipom to review?

[daipom]

Sorry for my late response.
I will review this.

[daipom]

• ref: Decision needed: Should we allow deleting a labelset from an existing metric? prometheus/client_ruby#140

[daipom]

• ref: Decision needed: Should we allow deleting a labelset from an existing metric? prometheus/client_ruby#140

In prometheus/client_ruby, a similar feature is pending.
The reasons given are that const metrics would be more appropriate, and that there are issues with the File store.
I don’t fully understand the connection with const metrics, but since this plugin doesn’t use the File store, I think it would be fine to add this feature as long as the impact on existing users and future maintenance can be kept to a minimum.