Actor lifecycle hooks

[sfunkenhauser]

This issue tracks the design and implementation of actor lifecycle hooks.

These are analogous to Kubernetes container lifecycle hooks. At a minimum we should support 2 new hooks (names TDB).

1. PostResume - Triggered right after an actor is resumed.
2. PreSuspend - Triggered right before an actor is suspended.

This area needs more investigation to determine exactly how these hooks should be implemented.

[EItanya]

I'm also definitely interested in this, but there's a bit of a wrinkle here i think we need to account for in this design. Specifically, at what point in the snapshot lifecycle does this happen. Before/after makes a very large difference for pre-warming etc.

[eliranw]

We are also interested in this.
We would need lifecycle hooks for running commands post-resume + pre-suspend for cuda-checkpointing GPU actors.
This connects to #185.

[HavenXia]

sfunkenhauser I can take this one. The upgrade workstream (#473) has to define these semantics anyway: a blue/green model that batch-suspends every running actor, so the PreSuspend and PostResume are inevitable path on the upgrade.

[sfunkenhauser]

sfunkenhauser I can take this one. The upgrade workstream (#473) has to define these semantics anyway: a blue/green model that batch-suspends every running actor, so the PreSuspend and PostResume are inevitable path on the upgrade.

Is this required for the initial release? Or is it sufficient to push this responsibility to the actor? The actor receives a SIGTERM when the pod is deleted and decides how to proceed. This was the assumption I was proceeding with for graceful termination, and it seems like upgrades would be very similar?