Centralised logging / metrics for k8s

[josibake]

I had to rip out fork observer, fluentd for now. They might not be the best solution going forward, too! But we definitely want some sort of logging / dashboard solution

[willcl-ark]

I also propose to rip out fluentd for docker in this pull: #173

It slows down docker too much, both on the log processing side (Ruby maxes out 1 CPU for me) and AFAIK bitcoind logs on each container going to stdout will be written to a daemon host file once (tank container), also read by fluentd, written again to another host logfile (fluentd), so doubling storage space and file read/writes. (can verify as docker logs tank_000000 will return tank 0 logs, and they will also be obn fluentd container docker logs warnet_fluentd.

Long story short we should prob just use what's best for each orchestration/container tool.

For docker I think just use default logger and the modified log_grepper fn in #173.

For k8s, which might be much larger scale, perhaps we can leverage @m3dwards expertise with an ELK stack or similar?

[willcl-ark]

I think @m3dwards is working on this