Add Wuji glove input plugin and Wuji hand retargeter by wuji-tech-dev · Pull Request #791 · NVIDIA/IsaacTeleop

wuji-tech-dev · 2026-07-17T07:54:27Z

Summary

This PR adds Wuji glove and dexterous-hand support to IsaacTeleop:

Add an optional out-of-process Wuji glove plugin based on Wuji SDK
v2026.7.14.
Map the glove's 21-joint skeleton to the existing OpenXR 26-joint
HandPose representation. Unavailable joints are marked invalid.
Reuse the existing HandTracker → HandsSource → HandInput
pipeline without introducing a new schema.
Support left/right gloves, stale-data handling, disconnection and
automatic reconnection.
Add WujiHandRetargeter to convert hand-tracking input into 20 Wuji
hand joint commands.
Add a teleoperation example supporting both Wuji Hand and Wuji Hand 2,
along with synthetic and replay modes.

Testing

Verified plugin installation and hand skeleton streaming.
Verified glove disconnection and automatic reconnection.
Verified end-to-end glove-to-hand teleoperation.
Verified synthetic and replay modes without physical devices.
Passed pre-commit and C++ formatting checks.

Known limitation

The glove skeleton is currently wrist-relative and provides 21 of the 26
OpenXR hand joints. A future update will add wrist-pose fusion and complete
26-joint hand tracking.

Add an optional out-of-process OpenXR plugin that reads 21-joint skeleton data from a Wuji glove and injects it through the existing push-device hand-tracking interface. The plugin maps the glove skeleton to the OpenXR 26-joint layout, routes left and right hands, handles stale data and device reconnection, and supports TeleopSession-managed startup and shutdown. It reuses the existing HandTracker, HandsSource, and HandInput pipeline without introducing a new schema. Add WujiHandRetargeter to convert OpenXR hand input into 20-DoF Wuji hand joint commands. The hardware demo supports Wuji Hand and Wuji Hand 2, while synthetic and replay modes can run without hardware. Add CMake integration, the isaacteleop[wuji] dependency extra, an installation script, and plugin usage documentation. Pin the Wuji SDK dependencies to version 2026.7.14. The injected glove skeleton remains wrist-relative. Fusion with an external wrist-pose source and completion of the five unavailable OpenXR joints are outside the scope of this change. Signed-off-by: wuji-tech-dev <dev@wuji.tech>

github-actions · 2026-07-17T07:54:35Z

📝 Docs preview is not auto-deployed for fork PRs.

A maintainer with write access to NVIDIA/IsaacTeleop can deploy a preview by
commenting /preview-docs on this PR. Once deployed, the preview
will live at:

https://nvidia.github.io/IsaacTeleop/preview/pr-791/

jiwenc-nv · 2026-07-17T15:12:01Z

@@ -0,0 +1,65 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 Wuji Technology. All rights reserved.


@wuji-tech-dev please turn this md file into a rst under docs/source/devices

jiwenc-nv · 2026-07-22T03:25:22Z

+    XR_HAND_JOINT_LITTLE_TIP_EXT,
+};
+
+constexpr XrSpaceLocationFlags kValidFlags =


Please fix: IIUC, this PR doesn't provide 6DoF pose for the hand posision? Can you do something similar to what Manus has?
That make the pose fushion for all the hand joints?

A bit more details: Because the plugin injects through the typed push-device hand channel (HandInjector → xrPushDevicePushHandTrackingNV), the injected hand is indistinguishable from a native XR_EXT_hand_tracking hand to every consumer of /hand/left|right — recorders, visualizers, and wrist-consuming retargeters will trust these bits. Please fuse a device wrist pose, following the manus plugin's pattern: wrist source from XDEV hand tracking with controller-aim fallback (src/plugins/manus/core/manus_hand_tracking_plugin.cpp:800-830), skeleton placed at the fused wrist, and the TRACKED bits set only while the wrist source is actually tracked (:886-892). Ideally factor the wrist-source machinery into plugin_utils, since two glove plugins now need it. If no wrist source is available at runtime, degrade to honest flags: VALID without TRACKED.

jiwenc-nv · 2026-07-22T03:25:22Z

+    for (int mp = 0; mp < 21; ++mp)
+    {
+        const WujiSkeletonJoint& src = frame->joints[mp];
+        XrHandJointLocationEXT& dst = out[kMpToXr[mp]];


Please fix: the SDK joint orientations are copied verbatim and stamped ORIENTATION_VALID, but nothing states the Wuji skeleton's per-joint axis convention. XR_EXT_hand_tracking prescribes a specific basis (−Z along the bone toward the wrist, +Y through the back of the hand), and SharpaHandRetargeter consumes JOINT_ORIENTATIONS through the same HandsSource path, so it will trust whatever basis these quaternions are in. Please verify the SDK basis against the OpenXR convention, remap at conversion if it differs, and document it here. This also gates the wrist-fusion request above: composing wrist_pose × wrist-relative joint pose requires knowing the SDK's local frame. Note that clearing ORIENTATION_VALID instead won't work — live_hand_tracker_impl.cpp:430 requires position and orientation valid per joint, which would zero out your own pipeline.

jiwenc-nv · 2026-07-22T03:25:22Z

+                while not stop_requested:
+                    t0 = time.monotonic()
+                    result = session.step()
+                    send(list(result["hand_joints"]))  # 20 values, firmware order


Please fix: on any tracking dropout (hand absent, or a single flickering joint tripping the retargeter's all-or-nothing validity gate), WujiHandRetargeter emits all-zero joint angles, and this loop forwards them straight to the hardware at 120 Hz.

Am I understand this correctly?

On the Wuji Hand 2 path that's a raw MIT-mode JointCommand with kp=3.0 and no filtering (the Hand 1 path at least has the 5 Hz LowPass), so a 200 ms glove dropout mid-grasp becomes an instantaneous snap-to-zero on a real hand. Please hold the last command (or ramp) on the loss transition, and run Hand 2 through the same low-pass as Hand 1.

jiwenc-nv · 2026-07-22T03:25:22Z

+    esac
+    name="wuji-sdk-c-${WUJI_SDK_C_VERSION}-${arch}"
+    url="${WUJI_SDK_C_URL_BASE}/v${WUJI_SDK_C_VERSION}/${name}.tar.gz"
+    work="$(mktemp -d)"


Please fix: the SDK extracts into this mktemp -d directory, which the EXIT trap deletes — but WUJI_SDK_INCLUDE_DIR/WUJI_SDK_LIB pointing into it are baked into the persistent CMake cache. The first build works; every later reconfigure/rebuild fails on a missing wuji_sdk.h (and cmake --install would try to install the vendor libs from the deleted dir), and since the cache variables are still set, the friendly FATAL_ERROR guard never fires. Please extract to a persistent location instead (e.g. under the build tree — install_manus.sh sets the precedent by extracting next to the script).

jiwenc-nv · 2026-07-22T03:25:23Z

+    url="${WUJI_SDK_C_URL_BASE}/v${WUJI_SDK_C_VERSION}/${name}.tar.gz"
+    work="$(mktemp -d)"
+    echo "==> Downloading ${name}.tar.gz"
+    curl -fL --retry 3 "$url" -o "$work/${name}.tar.gz" \


Please fix: this downloads a binary SDK over the network and links it into the build with no integrity check. install_manus.sh:184-195 already sets the house pattern: pin per-arch SHA-256s and hard-fail on mismatch. Please do the same for the wuji-sdk-c tarball.

jiwenc-nv · 2026-07-22T03:25:23Z

+
+using namespace plugins::wuji_glove;
+
+static_assert(ATOMIC_BOOL_LOCK_FREE, "lock-free atomic bool is required for signal safety");


Nit: static_assert(ATOMIC_BOOL_LOCK_FREE, ...) passes when the macro is 1 ("sometimes lock-free"), so it doesn't guarantee what its message claims. Either compare == 2 or drop it (haptikos/main.cpp:16 carries the identical assert — if you keep a corrected form here, the sibling deserves the same fix in a follow-up).

jiwenc-nv · 2026-07-22T03:25:23Z

+    if (wuji_scan(&list, &count) != WUJI_STATUS_OK)
+    {
+        std::cerr << "WujiGlovePlugin: wuji_scan failed: " << safe_err() << std::endl;
+        wuji_discovered_free(list, count);


Nit: two small teardown hardenings. (1) wuji_discovered_free(list, count) is also reached on the wuji_scan failure path with whatever the failed call left in list/count — guard with if (list) unless the SDK documents zero-init on failure. (2) In disconnect_glove, invalidate_hand runs before wuji_sub_close joins the callback thread, so an in-flight callback can re-validate the slot; invalidating again after the close would make teardown airtight (the 200 ms staleness gate bounds the damage meanwhile).

jiwenc-nv · 2026-07-22T03:25:23Z

+    model = "wuji_hand_2" if is_hand2 else "wuji_hand"
+    if is_hand2:
+        hand.effort_limit().set(1.5)
+        hand.mit_params().set((3.0, 0.05))


Nit: effort_limit(1.5) and mit_params((3.0, 0.05)) are unitless magic gains in an example people will copy verbatim — please state units and provenance ("vendor-recommended safe defaults" would do). Related: drive mode never checks that the connected hand's side matches --hand/the glove's side, so a left glove driving a right hand mirrors silently; a one-line warning on mismatch would help.

jiwenc-nv · 2026-07-22T03:25:23Z

+    DESTINATION plugins/wuji_glove
+)
+
+install(FILES "${WUJI_SDK_LIB}" "${WUJI_HAND_CPP_LIB}"


Nit: installing the vendor .sos into the shared ${CMAKE_INSTALL_LIBDIR} follows the haptikos precedent, so no change requested on location — but flagging for an explicit decision: these are proprietary Wuji binaries landing in the common libdir next to Apache-2.0 artifacts (redistribution question + soname-collision surface). If that's not intended, installing them under plugins/wuji_glove/ next to the binary (the $ORIGIN build RPATH already proves the layout) is the alternative, with INSTALL_RPATH adjusted to match.

jiwenc-nv · 2026-07-22T03:27:31Z

+# wuji_sdk.retargeting). The wuji-sdk[retarget] extra pulls the retargeting
+# engine's own deps (numpy, scipy, nlopt, pin, pyyaml).
+#
+# Heads-up for aarch64: wuji-sdk[retarget] drags in nlopt. PyPI ships modern nlopt


Please fix: This just happens recently: nlopt now has a ARM64 wheel prebuilt. So the comments where about nlopt is no longer needed.

Can you help validate and update the docs if that's the case already? Thanks!

jiwenc-nv requested review from jiwenc-nv, lotusl-code and rwiltz July 17, 2026 17:00

jiwenc-nv added the Community label Jul 17, 2026

jiwenc-nv reviewed Jul 22, 2026

View reviewed changes

jiwenc-nv requested changes Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Wuji glove input plugin and Wuji hand retargeter#791

Add Wuji glove input plugin and Wuji hand retargeter#791
wuji-tech-dev wants to merge 1 commit into
NVIDIA:mainfrom
wuji-tech-dev:feat/wuji-glove-plugin-and-hand-retargeter

wuji-tech-dev commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

jiwenc-nv Jul 17, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

jiwenc-nv Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,65 @@
		<!--
		SPDX-FileCopyrightText: Copyright (c) 2026 Wuji Technology. All rights reserved.


		using namespace plugins::wuji_glove;

		static_assert(ATOMIC_BOOL_LOCK_FREE, "lock-free atomic bool is required for signal safety");

Uh oh!

Conversation

wuji-tech-dev commented Jul 17, 2026

Summary

Testing

Known limitation

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants