Added watchdog timeout when stop is issued for stuck actors. #336
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 25, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
d4f7b17 to
96bbded
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 25, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
96bbded to
baf8c7a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 25, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
baf8c7a to
c848e18
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 25, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
2 times, most recently
from
2dd8505 to
9bc4b2c
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 25, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 25, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
9bc4b2c to
e9be98d
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
force-pushed
the
export-D77303504
branch
from
e9be98d to
79511d0
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 26, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 26, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
79511d0 to
c99a6f5
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
force-pushed
the
export-D77303504
branch
from
c99a6f5 to
c41bd9c
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 26, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
force-pushed
the
export-D77303504
branch
from
c41bd9c to
e9dd94f
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 26, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
e9dd94f to
7809d28
Compare
vidhyav
force-pushed
the
export-D77303504
branch
from
29f2e92 to
b96f22e
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Reviewed By: suo Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
b96f22e to
b28f15d
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Reviewed By: suo Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
b28f15d to
0e93560
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Reviewed By: suo Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
0e93560 to
2ed2110
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Reviewed By: suo Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
2ed2110 to
4a0646e
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
force-pushed
the
export-D77303504
branch
from
4a0646e to
fe0a267
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Reviewed By: suo Differential Revision: D77303504
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
force-pushed
the
export-D77303504
branch
from
fe0a267 to
875e191
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Here is the reason why we needed in the first place: While running an xlformers test, we encountered a problem wherein the process was getting stuck upon issuing an exit. The root cause for the blocking was the cuda unregistry routine getting stuck blocking the exit() call and all other calls. In order to simulate the same, we added here an exit handler that loops forever. Reviewed By: suo, technicianted Differential Revision: D77303504
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Here is the reason why we needed in the first place: While running an xlformers test, we encountered a problem wherein the process was getting stuck upon issuing an exit. The root cause for the blocking was the cuda unregistry routine getting stuck blocking the exit() call and all other calls. In order to simulate the same, we added here an exit handler that loops forever. Reviewed By: suo, technicianted Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
875e191 to
f82e669
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jun 27, 2025
…-labs#336) Summary: Pull Request resolved: pytorch-labs#336 Allocators have a timeout to kill unresponsive procs when stop() is issued. Here is the reason why we needed in the first place: While running an xlformers test, we encountered a problem wherein the process was getting stuck upon issuing an exit. The root cause for the blocking was the cuda unregistry routine getting stuck blocking the exit() call and all other calls. In order to simulate the same, we added here an exit handler that loops forever. Reviewed By: suo, technicianted Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
f82e669 to
73af9fe
Compare
…-labs#336) Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued. Here is the reason why we needed in the first place: While running an xlformers test, we encountered a problem wherein the process was getting stuck upon issuing an exit. The root cause for the blocking was the cuda unregistry routine getting stuck blocking the exit() call and all other calls. In order to simulate the same, we added here an exit handler that loops forever. Reviewed By: suo, technicianted Differential Revision: D77303504
vidhyav
force-pushed
the
export-D77303504
branch
from
73af9fe to
32e8f6b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D77303504 |
|
This pull request has been merged in 96cc0a7. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Allocators have a timeout to kill unresponsive procs when stop() is issued.
Differential Revision: D77303504