[Experimental] - Max resource lifetime controller

[chungers]

This PR adds a simple utility that sets maximum resource lifetime by scanning and finding the oldest resource. If the oldest instance's age is greater than the set max lifetime, and that the system is in steady-state (ie, the successive counts of instances are not decreasing over time), then destroy the instance. Once the instance is destroyed, the other controller (specifically the Group plugin) will detect the change and replace the instance with a new one.

This is useful for rotating compute nodes / workers in a cluster such that no instances can exceed a certain lifetime and can be subjected to attacks. This serves as a POC for combining container and infrastructure orchestration to improve security of a compute cluster.

It's possible to add interaction with a flavor plugin to force a "Drain" of the targeted resource before destruction. It will be done in a future PR.

A new subcommand x for experimental is added to the CLI so that this feature is added as a subcommand of the experimental features.

Signed-off-by: David Chung [email protected]

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "maxlife" [email protected]:chungers/infrakit.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842353806272
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[codecov]

Codecov Report

Merging #545 into master will increase coverage by 1.76%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #545      +/-   ##
==========================================
+ Coverage   56.66%   58.43%   +1.76%     
==========================================
  Files          57       32      -25     
  Lines        3969     2069    -1900     
==========================================
- Hits         2249     1209    -1040     
+ Misses       1432      717     -715     
+ Partials      288      143     -145

Impacted Files	Coverage Δ
pkg/types/spec.go
pkg/util/exec/exec.go
pkg/types/any.go
pkg/plugin/metadata/updatable.go
pkg/template/help.go
pkg/store/swarm/swarm.go
pkg/spi/event/types.go
pkg/template/funcs.go
pkg/plugin/metadata/plugin.go
pkg/template/template.go
... and 14 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 007114d...ba8fffc. Read the comment docs.