Add G1 Wuji teleop example - #699
Conversation
|
📝 Docs preview is not auto-deployed for fork PRs. A maintainer with write access to |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR introduces a complete G1-Wuji AVP+MANUS teleoperation example for Isaac Teleop. It adds a new Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@avp_manus.sh`:
- Around line 170-172: The final demo launch command uses an invalid config path
that will not resolve from a normal repo checkout. Update the config argument in
the main teleop command in avp_man.sh to use a repo-relative path or the correct
full IsaacTeleop path, and make sure the g1_wuji_teleop_main.py invocation still
points to the avp_manus.yml config correctly.
In `@examples/g1_wuji_teleop/python/g1_wuji_teleop/devices/avp_manus_stream.py`:
- Around line 53-58: The retargeter state path in
FIXED_G1_WUJI_RETARGET_DEFAULTS is hardcoded to a shared /tmp filename, which
can collide across runs and be abused via pre-created symlinks. Update the
left/right parameter_config_path entries in avp_manus_stream.py to generate a
unique file per process in a private runtime directory instead of using fixed
/tmp/...json names, and route the code that consumes these defaults to use the
generated path via the existing default config structure.
In `@examples/g1_wuji_teleop/python/g1_wuji_teleop/paths.py`:
- Around line 32-37: resolve_repo_relative_path currently checks app_root() but
never actually falls back to repo_root(), so repo-relative paths are resolved
from the wrong base. Update this function to first try the app_root()-based path
and, if it does not exist, resolve the same relative_path against repo_root()
instead; keep the behavior centered on resolve_repo_relative_path, app_root(),
and repo_root() so the fallback is clear and correct.
In `@examples/g1_wuji_teleop/python/g1_wuji_teleop/robots/g1_wuji/debug_viz.py`:
- Around line 350-378: The frame marker updater only handles "left" and "right",
so head markers are skipped and never refreshed. Update
update_frame_axis_markers() to also process the "head" key (or otherwise make
the side iteration configurable) so update_head_marker() can pass {"head":
head_pose} and still drive frame_markers.visualize for handles.head_link_frame.
In `@examples/g1_wuji_teleop/python/g1_wuji_teleop/robots/g1_wuji/runtime.py`:
- Around line 1002-1017: The early return in runtime.py’s hand retargeting path
is too strict: the current left_input/right_input check in the logic around
_last_left_skeleton, _last_right_skeleton, and _waiting_announced blocks updates
for one side when only the other side is missing. Change the fallback flow so
each hand is processed independently using its own skeleton or last-known value,
and only skip the side that truly has no usable input instead of returning both
_last_left_targets and _last_right_targets together. Keep the waiting message in
the same retargeting helper, but gate it on both sides being unavailable rather
than one side missing.
In `@examples/g1_wuji_teleop/python/g1_wuji_teleop/viz/teleop_visualizer.py`:
- Line 24: The teleop visualizer is hardcoding a shared MPLCONFIGDIR under /tmp,
which should be replaced with a uniquely created per-user/per-process cache
directory. Update teleop_visualizer.py where
os.environ.setdefault("MPLCONFIGDIR", ...) is set so it points to a freshly
created private temp directory for each run, using a location derived from the
current process/user rather than a fixed /tmp/matplotlib path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a2eb2022-8fea-477e-9c9e-c6423d590ccb
📒 Files selected for processing (147)
.gitattributes.gitignore.gitmodulesAGENTS.mdREUSE.tomlavp_manus.shdeps/cloudxr/webxr_client/helpers/react/CloudXRComponent.tsxdeps/cloudxr/webxr_client/src/App.tsxdeps/cloudxr/webxr_client/src/CloudXRUI.tsxexamples/g1_wuji_teleop/README.mdexamples/g1_wuji_teleop/assets/g1_wuji/g1_three_fingers.usdexamples/g1_wuji_teleop/assets/g1_wuji/g1_wuji.usdexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger1_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger2_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger3_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger4_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/finger5_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/palm_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/left/palm_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger1_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger2_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger3_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger4_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link1.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link1_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link2.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link2_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link3.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link3_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link4.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_link4_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_tip_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/finger5_tip_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/palm_link.STLexamples/g1_wuji_teleop/assets/wuji_hand/meshes/right/palm_link_collision.STLexamples/g1_wuji_teleop/assets/wuji_hand/urdf/left.urdfexamples/g1_wuji_teleop/assets/wuji_hand/urdf/right.urdfexamples/g1_wuji_teleop/config/avp_manus.ymlexamples/g1_wuji_teleop/config/local/g1_wuji/hand_left_config.ymlexamples/g1_wuji_teleop/config/local/g1_wuji/hand_right_config.ymlexamples/g1_wuji_teleop/config/official/g1_wuji/manus_wuji_left.yamlexamples/g1_wuji_teleop/config/official/g1_wuji/manus_wuji_right.yamlexamples/g1_wuji_teleop/python/g1_wuji_teleop/__init__.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/cli.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/devices/avp_manus_stream.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/hand_retargeting/__init__.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/hand_retargeting/base.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/hand_retargeting/wuji_official_adapter.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/paths.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/robots/g1_wuji/__init__.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/robots/g1_wuji/debug_viz.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/robots/g1_wuji/runtime.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/robots/g1_wuji/scene.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/session.pyexamples/g1_wuji_teleop/python/g1_wuji_teleop/viz/teleop_visualizer.pyexamples/g1_wuji_teleop/scripts/g1_wuji_teleop_main.pyexamples/g1_wuji_teleop/scripts/g1_wuji_teleop_visualizer.pyexamples/teleop/CMakeLists.txtexamples/teleop/python/pyproject.tomlscripts/setup_v2d_src.shsrc/plugins/manus/app/main.cppsrc/plugins/manus/core/inc/manus/manus_hand_tracking_plugin.hppsrc/plugins/manus/core/manus_hand_tracking_plugin.cppsrc/viz/python/layers_bindings.cppsrc/viz/python_tests/test_offscreen_session.pythird_party/wuji-retargeting
6047d99 to
99c3146
Compare
|
Updated the PR to a single example-only commit. The current diff is limited to Most CodeRabbit inline comments now refer to files that no longer exist in the current PR version:
|
Signed-off-by: lei.hu <lei.hu@git.lightwheel.ai>
99c3146 to
8816f3f
Compare
| init_state=ArticulationCfg.InitialStateCfg( | ||
| pos=config.initial_world_position, | ||
| # Config stores xyzw; IsaacLab InitialStateCfg.rot expects wxyz. | ||
| rot=_xyzw_to_wxyz(config.initial_world_orientation_xyzw), |
There was a problem hiding this comment.
Please fix: This conversion is inverted against Isaac Lab 3.x, masked by a config value that's wrong the opposite way. InitialStateCfg.rot is documented (x, y, z, w) — the identity default (0,0,0,1) confirms the order — as of current Isaac Lab (checked against 3.0.0 / isaaclab 7.4.0; the field moved to xyzw in the 2.x→3.x migration), so _xyzw_to_wxyz (line 24) reorders it wrongly. It only looks right because initial_world_orientation_xyzw: [0, 1, 0, 0] is itself wrong (180° about +Y, upside-down) and the bad conversion happens to land on the 180° yaw you want. The trap: anyone editing this documented-as-xyzw key with a real xyzw value (e.g. identity [0,0,0,1]) gets a silently rotated robot, no error. Please delete _xyzw_to_wxyz, pass the normalized quat straight to rot=, and set initial_world_orientation_xyzw to [0, 0, 1, 0] in both avp_manus.yaml and vr_manus.yaml.
|
|
||
| # G1-Wuji AVP/VR + MANUS Setup | ||
|
|
||
| This runbook assumes an Isaac Lab Python environment and an IsaacTeleop build |
There was a problem hiding this comment.
Please fix: The example's largest dependency, and the runbook only "assumes an Isaac Lab Python environment" — no how-to, no required version, no mention that Isaac Sim (and the omni.* / pxr / torch modules the code imports) comes with it. This isn't just convenience: orientation correctness flips across Isaac Lab's wxyz→xyzw migration (see the scene.py note), so the wrong generation gives a subtly wrong robot, not an error. Please (1) add an "Install Isaac Lab" step with a concrete command and minimum version (e.g. pip install "isaaclab[isaacsim,all]" --extra-index-url https://pypi.nvidia.com, or link the official docs), noting every later step runs in that same env; (2) state the supported version in pyproject.toml/README; (3) add a startup assertion (e.g. matrix_from_quat([0,0,0,1]) is identity) so a mismatched env fails loudly.
| ) | ||
|
|
||
|
|
||
| class WujiHandTargetBackend: |
There was a problem hiding this comment.
Please fix: WujiHandTargetBackend (and _wuji_skeleton21_from_openxr below) run retargeting outside the pipeline on a bespoke per-frame dict format built in input_stream.py, rather than as a retargeting-engine node. The canonical pattern for wrapping an external retargeter as a BaseRetargeter via OutputCombiner is src/retargeters/dex_hand_retargeter.py, and src/retargeters/sharpa_hand_retargeter.py already has the OpenXR-26→21 mapping this file re-derives. Since examples teach the integration path, please restructure the hand side as a retargeter node (example-local is fine), leaving only sim-coupled logic (arm IK, frame binding) in the Isaac Lab loop — that should also shed a lot of glue in robot.py.
| plugin_name: manus_hand_plugin | ||
| plugin_root_id: manus | ||
| search_paths: | ||
| - ../../../install/plugins |
There was a problem hiding this comment.
Please fix: search_paths: [../../../install/plugins] is resolved relative to the config file, so it only works from a source tree. In the install tree from this example's CMakeLists.txt (which installs config docs assets under <prefix>/examples/g1_wuji_teleop) it points at <prefix>/install/plugins, which doesn't exist — and input_stream.py hard-fails on a missing search path, so the installed example can't run. Sibling examples anchor on __file__ to work from both trees; please default the search path in code relative to the installed package and keep the YAML key as an optional override. Same in vr_manus.yaml:11.
| def start_openxr_runtime() -> None: | ||
| nonlocal teleop, teleop_started | ||
| teleop = TeleopMain(teleop_config, config_dir=config_path.parent) | ||
| teleop.__enter__() |
There was a problem hiding this comment.
Please fix: TeleopMain is entered with a bare __enter__() and only exited on a clean loop finish. Retargeting raises on transient bad input (e.g. an invalid/collapsed skeleton), so one bad glove frame unwinds out of the loop without __exit__, leaking the MANUS/OpenXR session docs/setup.md notes is single-owner — recovery then needs a kill and replug. Please wrap the lifecycle in try/finally (or use TeleopMain as a real with), and consider degrading a transient invalid skeleton to "hold last target" (as you already do for a missing frame) instead of aborting.
| ) | ||
| self._startup_ref_announced.add(side) | ||
|
|
||
| def _compute_delta_joint_pos(self, delta_pose: Any, jacobian: Any) -> Any: |
There was a problem hiding this comment.
Please fix: _compute_delta_joint_pos implements pinv/svd/trans solvers (plus matching _ik_params branches and command_type == "position" paths), but _arm_ik_runtime_config hardcodes ik_method="dls" / command_type="pose" and nothing feeds these from config — so ~60 lines can never run, partly re-implementing DifferentialIKController internals. Relatedly, the ArmIkRuntimeConfig dataclass defaults (orientation_error_scale=0.35, smoothing_alpha=0.35, deadbands, enabled, pose_scale/pose_z_offset) are all overridden by different hardcoded values, so the defaults are dead and misleading to anyone tuning. Please keep only the dls/pose path and collapse the config to the fields that vary.
| third_viewpoint: | ||
| relative_position_m: [0.0, -1.2, 0.55] | ||
|
|
||
| avp: |
There was a problem hiding this comment.
Please fix: In the VR/PICO config, this avp: section configures controllers, not an Apple Vision Pro — and the AVP branding runs through the code it maps onto: avp_head / avp_controller_* outputs, the AvpRobotFrameBinding class and its robot.avp_frame_binding key, and an "AVP head-hand frame calibrated" log even in VR mode. The underlying library names these device-agnostically (HeadSource, ControllersSource, HandsSource). YAML keys are a public surface users copy, so better to rename now — e.g. xr_input: / robot.xr_frame_binding / HeadHandFrameBinding and head / controller_* outputs. Genuinely AVP-specific comments can stay.
| return rotation | ||
|
|
||
|
|
||
| def _head_hand_frame_rotation( |
There was a problem hiding this comment.
Please fix: _head_hand_frame_rotation builds the operator anchor frame from the full 3D head-forward vector, but _world_z_yaw_rotation_from_forward later decomposes yaw about world-Z, which only holds if that frame is horizontal. If the operator looks down during calibration (natural when checking their hands), the frame pitches and operator-vertical leaks into robot-forward. Please project head-forward onto the horizontal plane before building the frame so both paths share one assumption.
| name.replace("left_", "right_", 1) for name in LEFT_WUJI_JOINTS | ||
| ) | ||
|
|
||
| WUJI_SKELETON21_OPENXR_NAMES = ( |
There was a problem hiding this comment.
Please fix: This 26→21 remap and the hand-off to wuji-retargeting are the key convention for anyone adapting a different glove or tracker, and it's all implicit — the order is MediaPipe hand-landmark order (wrist; thumb CMC/MCP/IP/TIP; then MCP/PIP/DIP/TIP per finger, index-MCP and middle-MCP at indices 5 and 9), and landmarks go to the retargeter in raw OpenXR world coordinates (Y-up, meters). Please add a short docstring stating order, units, and frame. Also worth noting whether mediapipe_rotation z: ±15° in the retargeting configs is a MANUS sensor offset or frame-convention compensation — that's the first thing that breaks when the input source changes.
| return frame @ quat_xyzw_to_matrix(orientation_xyzw) @ frame.T | ||
|
|
||
|
|
||
| def normalize_quat_xyzw(quat: Sequence[float]) -> np.ndarray: |
There was a problem hiding this comment.
Please fix: normalize_quat_xyzw (here) and quat_xyzw_to_matrix (line 433) duplicate same-named functions in transforms.py, with different behavior — the transforms.py versions raise on a zero-norm quaternion and return float64; these silently return identity/default and use float32. Same names, opposite failure behavior, imported both ways (input_stream.py from transforms, app.py from robot) — a latent bug. Please keep one copy in transforms.py (I'd keep the raising behavior) and import from there; consolidating the quat math there also gives the package one convention boundary, pairing well with the scene.py fix.
2 participants
Description
Add a self-contained G1-Wuji teleoperation example under
examples/g1_wuji_teleop.The example wires AVP + MANUS and VR/PICO + MANUS input into the G1-Wuji robot and Wuji hand
retargeting workflow without modifying the underlying CloudXR, WebXR, MANUS plugin, or Televiz
runtime code.
Changes include:
examples/g1_wuji_teleopexample package and CMake entry.and scene setup.
examples/g1_wuji_teleop/docs/setup.md.Fixes: N/A
Type of change
Testing
Tested locally on Linux.
Commands run: