Add more explanation to missing python actor object error #366
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D77446127 |
pzhan9
added a commit
to pzhan9/monarch-1
that referenced
this pull request
Jun 27, 2025
…bs#366) Summary: I hit this error when plumbing `cast` to the python side (D77386342, v1). Briefly, I use `cast` [when mesh is available, and fall back on direct send if not](https://www.internalfb.com/code/fbsource/[D77386342-V1]/fbcode/monarch/python/monarch/actor_mesh.py?lines=192-216). This lead to a mixture usage of `cast` and `direct send` in this test https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=256-263 Briefly, `proc.spawn` uses cast, but inside `f.get.stream(t)`, it uses direct send: https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=250-253 That is because when `t` gets pickle, `PythonActorMesh` is [not included](https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/monarch/actor_mesh.py?lines=153-163). This is very painful to debug. Let us beef up the error message so hopefully I can remember what happened when I run into it next time. Differential Revision: D77446127
pzhan9
added a commit
to pzhan9/monarch-1
that referenced
this pull request
Jun 27, 2025
…bs#366) Summary: I hit this error when plumbing `cast` to the python side (D77386342, v1). Briefly, I use `cast` [when mesh is available, and fall back on direct send if not](https://www.internalfb.com/code/fbsource/[D77386342-V1]/fbcode/monarch/python/monarch/actor_mesh.py?lines=192-216). This lead to a mixture usage of `cast` and `direct send` in this test https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=256-263 Briefly, `proc.spawn` uses cast, but inside `f.get.stream(t)`, it uses direct send: https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=250-253 That is because when `t` gets pickle, `PythonActorMesh` is [not included](https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/monarch/actor_mesh.py?lines=153-163). This is very painful to debug. Let us beef up the error message so hopefully I can remember what happened when I run into it next time. Differential Revision: D77446127
|
This pull request was exported from Phabricator. Differential Revision: D77446127 |
pzhan9
added a commit
to pzhan9/monarch-1
that referenced
this pull request
Jun 27, 2025
…bs#366) Summary: Pull Request resolved: pytorch-labs#366 I hit this error when plumbing `cast` to the python side (D77386342, v1). Briefly, I use `cast` [when mesh is available, and fall back on direct send if not](https://www.internalfb.com/code/fbsource/[D77386342-V1]/fbcode/monarch/python/monarch/actor_mesh.py?lines=192-216). This lead to a mixture usage of `cast` and `direct send` in this test https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=256-263 Briefly, `proc.spawn` uses cast, but inside `f.get.stream(t)`, it uses direct send: https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=250-253 That is because when `t` gets pickle, `PythonActorMesh` is [not included](https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/monarch/actor_mesh.py?lines=153-163). This is very painful to debug. Let us beef up the error message so hopefully I can remember what happened when I run into it next time. Differential Revision: D77446127
|
This pull request was exported from Phabricator. Differential Revision: D77446127 |
pzhan9
added a commit
to pzhan9/monarch-1
that referenced
this pull request
Jun 27, 2025
…bs#366) Summary: Pull Request resolved: pytorch-labs#366 I hit this error when plumbing `cast` to the python side (D77386342, v1). Briefly, I use `cast` [when mesh is available, and fall back on direct send if not](https://www.internalfb.com/code/fbsource/[D77386342-V1]/fbcode/monarch/python/monarch/actor_mesh.py?lines=192-216). This lead to a mixture usage of `cast` and `direct send` in this test https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=256-263 Briefly, `proc.spawn` uses cast, but inside `f.get.stream(t)`, it uses direct send: https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=250-253 That is because when `t` gets pickle, `PythonActorMesh` is [not included](https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/monarch/actor_mesh.py?lines=153-163). This is very painful to debug. Let us beef up the error message so hopefully I can remember what happened when I run into it next time. Differential Revision: D77446127
…bs#366) Summary: I hit this error when plumbing `cast` to the python side (D77386342, v1). Briefly, I use `cast` [when mesh is available, and fall back on direct send if not](https://www.internalfb.com/code/fbsource/[D77386342-V1]/fbcode/monarch/python/monarch/actor_mesh.py?lines=192-216). This lead to a mixture usage of `cast` and `direct send` in this test https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=256-263 Briefly, `proc.spawn` uses cast, but inside `f.get.stream(t)`, it uses direct send: https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=250-253 That is because when `t` gets pickle, `PythonActorMesh` is [not included](https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/monarch/actor_mesh.py?lines=153-163). This is very painful to debug. Let us beef up the error message so hopefully I can remember what happened when I run into it next time. Reviewed By: mariusae Differential Revision: D77446127
|
This pull request was exported from Phabricator. Differential Revision: D77446127 |
|
This pull request has been merged in cc8630b. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
I hit this error when plumbing
castto the python side (D77386342, v1). Briefly, I usecastwhen mesh is available, and fall back on direct send if not. This lead to a mixture usage ofcastanddirect sendin this testhttps://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=256-263
Briefly,
proc.spawnuses cast, but insidef.get.stream(t), it uses direct send:https://www.internalfb.com/code/fbsource/[5ef41e9fdc95c3a2f4f9135244dbb990d0818d0d]/fbcode/monarch/python/meta/tests/test_python_actors.py?lines=250-253
That is because when
tgets pickle,PythonActorMeshis not included.This is very painful to debug. Let us beef up the error message so hopefully I can remember what happened when I run into it next time.
Differential Revision: D77446127