Add signal handling during shutdown operations #232
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements signal handling during shutdown operations. This should mitigate issues with commands run from
sorunlibthat error out and leave either SMuRF streaming or ACU motion (or both) on.Some things I want to mention:
SIGTERMis now always handled just likeSIGINT. This makes "Interrupt" and "Terminate" behave the same from nextline, both raising aKeyboardInterrupt. In my opinion, both signals should result in a clean shutdown. This was the simplest way to accomplish that. I may change howSIGTERMis handled in the future to try to shutdown things on its own (likely after Check stream status and stop running streams before operations #231 or similar gets implemented), and thensys.exit(0), but this should work for now.run.smurf.stream('off')in modules that called it. This was done so that these modules also benefit from the signal handling addition.protect_shutdowndecorator should be used on any functions that perform shutdown actions moving forward. This catches the signals and just prints them to stdout while shutdown occurs. If something isn't stopping properly aSIGKILLcan still be sent to exit (likely with streams/ACU motion still occurring).I tested this in a local environment with the following elements:
I tested sending
SIGINT,SIGTERM, andSIGKILLat various points during an example scan block that looked like this:I am able to interrupt during the
run.seq.scan()line and watch the scan stop gracefully. Additional signals sent (with the exception ofSIGKILL) are simply printed tostdout.SIGKILLends things abruptly with streams still running.This does not handle the case of streams left running while a schedule is started. That will still run into the issues described in #229.
Resolves #166.