user defined save_best_checkpoint() hook

[jsleep]

it should be a feature of our library to keep the "best" checkpoint based on a tracked metric like the minimum value of validation loss.

this could be a semi-big change because:

• we do not have any metric tracking
• we do not have any conditional checkpointing implemented, but Eduardo make checkpointing pretty generic for it.

[ananthrs1]

We can have the current checkpointing logic as is to check point every specific interval
We can enhance the trainer code to take a custom function callback from the scenario code that dictates which is the best checkpoint to use based on operations on the custom metric relevant to that scenario. Then the trainer can overwrite the best checkpoint based on scenario owner implemented custom logic in the function provided to the trainer and by default we can have the latest checkpoint as the best checkpoint.