Issue 1986 Checkpointing write & remove order change

[devrimcavusoglu]

Fixes #1986

Description: Former "remove & write" implementation order is reversed to "write & remove", api call has been added for ClearMLSaver remove functionality.

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[vfdev-5]

@devrimcavusoglu thanks for the PR, i'll review it asap

[vfdev-5]

@devrimcavusoglu could you please explain why you are adding this code ?
I understand that the idea is to use clearml's v1.0 new feature and make remove work as expected for others.
I'm not sure how it would work with the previous logic where a queue is used...

cc @bmartinn any suggestion on how we could recode everything here ?

[devrimcavusoglu]

As you said, this would make use of clearml v1.0 new feature, and used to actually remove the remote or local file(s). For backward compatibility, I'd appreciate any suggestions.

[bmartinn]

No need to pass logger to StorageHelper.get , basically this should be fine:

helper = StorageHelper.get(filename)
helper.delete(filename)

clearml v1.0 supports older versions of clearml-server, so pushing the requirements for clearml>1.0 should not be limiting in any way for users.

[devrimcavusoglu]

Thanks for the review & comment @bmartinn. I'll update PR with that change asap. I and @vfdev-5 were mainly concerned about backward compatibility as it may negatively impact users using clearml<1.0; however, I see that it is not a concern.

[bmartinn]

@devrimcavusoglu I rechecked backwards compatibility, the only exception will be uploading models directly to clearml-server < 1.0. If this is the case the delete function might return an error as the http file server did not support the delete operation. I really think that just protecting with except ValueError should be enough. wdyt?

try:
  helper.delete(filename)
except ValueError:
  # log something ?!

[vfdev-5]

@bmartinn can you please confirm whether we still need _CallbacksContext and _checkpoint_slots code with clearml v1.X ?

I have an impression that if we could remove data from remote storage then we can drastically simplify ClearMLSaver for v1.X ?

The idea could be to setup ClearMLSaver as ClearMLSaverV1 if clearml is present and has version >= 1.0.

For clearml version < 1.0 we can keep the existing code as ClearMLSaverV0 and try to temporary hack it in order to make it work with the new write->remove logic. Later we will remove the support for clearml version < 1.0.

[vfdev-5]

@devrimcavusoglu I wonder if we could recode __call__ and remove methods for ClearMLSaverV0 to delay saving execution. We have two possibilities when saving new checkpoint:

• checkpoint is intermediate in the slots -> only save is executed
• checkpoint should replace a previous one in the slots -> save and remove are executed

Second case could be covered easily such that we call "saving" ops in remove method:

    ...
    slots[slots.index(filename)] = None
    ...
    try:
        super(ClearMLSaver, self).__call__(checkpoint, filename, metadata)
    finally:
        WeightsFileHandler.remove_pre_callback(pre_cb_id)
        WeightsFileHandler.remove_post_callback(post_cb_id)

The first case is complicated as remove method is not called. Maybe we could schedule a thread to execute saving ops after a delay of 5-10 seconds ?

What do you think ?

[devrimcavusoglu]

@vfdev-5 Hi there, sorry for delayed reply (was in tight schedule). Option 2 seems more solid to me, and also more practical. Regarding the class seperation (former comment) for different versions, I think rather than pointing to V0 and V1, handling it under ClearMLSaver would be more concrete, wdyt ?

[vfdev-5]

Regarding the class seperation (former comment) for different versions, I think rather than pointing to V0 and V1, handling it under ClearMLSaver would be more concrete, wdyt ?

I was thinking that doing things into two separate classes will help us later to quickly remove V0 when no one is using it anymore...

[jkhenning]

Hi @vfdev-5,

@bmartinn can you please confirm whether we still need _CallbacksContext and _checkpoint_slots code with clearml v1.X ?

I think that if you handle storing a new checkpoint to a temporary-named newly uploaded, file, removing the previously uploaded file and renaming the temporary-named file to the previous name, you won't need the checkpoints 🙂