[core] avoid etcd updates when AWs do not have preemption condition generated #632
Description
Describe the Bug
AWs were sending etcd updates for every run of the preemption thread. The update should only be send the AWs have preempt condition. the side effect of this is that the current AW will be sent to back-off queue and wont be dispatched for next 20 seconds (default) backoff time.
Codeflare Stack Component Versions
Please specify the component versions in which you have encountered this bug.
Codeflare SDK:
MCAD: Release 1.34.1 and main
Steps to Reproduce the Bug
Submit an AW that takes a very long time to spawn pods, the AW will get preempted and will be sent to the backoff queue.
What Have You Already Tried to Debug the Issue?
Same as above
Expected Behavior
AWs should only be preempted and backed off when they have a valid preemption condition.
Screenshots, Console Output, Logs, etc.
NA
Affected Releases
Release 1.34.1
Additional Context
NA