Add support for time_key and remove_keys

[mihirpatel]

• Allow using time from log record, added new config for time_key and time_format
• Any log record fields used to populate splunk metadata (time, source, sourcetype, index, host) does not need to be duplicated in record itself. Hence, support for remove_keys will allow avoid duplicates.

We use plugin v0.3.0 of the plugin, if you can open up branch from tag 0.3.0, I have patch (
support_time_key_and_remove_keys.patch.txt) ready I can apply and create a PR for this change.

[Jitsusama]

I really like the added time stuff. Instead of using a remove_keys parameter, I would recommend something like preserve_time_key boolean that would just remove that single key. This is primarily due to other plugins offering remove_keys functionality in a more flexible manner, and that particular feature doesn't seem to fall in line with this plugin, outside of removing the time key itself.

[mihir-shutterfly]

Hi @Jitsusama, the support for remove_keys is similar to other plugins, we remove multiple fields added as part of various filters and transformation steps to avoid duplication of data between what gets sent as metadata to splunk and payload. For example, we use:

remove_keys source, host, index, @timestamp

[chilicheech]

On the leaf nodes I use in_tail and out_forward. The data gets sent to an "aggregator" that uses in_forward and this plugin to send to splunk. I need to set source, sourcetype, and host on the leaf node. Once the data gets to the aggregator I use the source, sourcetype and host fields of this plugin by extracting record["host"], etc.. Now I have duplicate fields in splunk and I need to deduplicate by using something like remove_keys.