xr_skill

Overview

This skill integrates Vuer's XR features for VR/AR/WebXR applications, exposing hand tracking, motion controllers, AR mesh access, gripper control, and haptics. It provides simple Python APIs to stream input data and send actuator events for interactive robotics and immersive visualization workflows. Use it to prototype hand-driven interactions, controller mappings, haptic feedback, and capture AR meshes for spatial understanding.

How this skill works

The skill registers XR entities (Hands, MotionController, WebXRMesh) in a DefaultScene and streams events to your Python app. Hand and controller events arrive through handlers (HAND_MOVE, CONTROLLER_MOVE) with joint positions, rotations, button states, and pinch strength. You can upsert scene objects (e.g., Gripper) from sensor data, request AR mesh snapshots, and trigger device haptics with structured actuator pulses.

When to use it

• Prototype hand-based manipulation or teleoperation using tracked joints and pinch strength.
• Map VR motion controller inputs to robot arms, tools, or UI controls.
• Capture AR mesh data from a device for environment modeling or obstacle detection.
• Drive haptic feedback from simulation events or user interactions.
• Integrate spatial sensing into visualization dashboards for robotics and XR research.

Best practices

• Stream only required channels (hands, controller, mesh) to minimize bandwidth and latency.
• Smooth or filter raw joint positions before driving actuators or robot controllers.
• Use pinchStrength and tip joint distances for reliable grasp detection rather than single thresholds.
• Request AR mesh snapshots intermittently; avoid continuous polling for large meshes.
• Coordinate haptic timing with visual and physics events to reinforce perception without overwhelming users.

Example use cases

• Real-time hand teleoperation: map wrist and finger joints to a remote gripper and update Gripper object with computed pinchWidth.
• Controller-driven interface: read controller position, rotation, and buttons to manipulate 3D objects or navigate scenes.
• Environment scanning: capture WebXRMesh snapshots, extract vertices/indices and semantic labels for obstacle mapping.
• Haptic cues: trigger left/right HapticActuatorPulse events when collisions occur or tasks complete.
• Hybrid demos: combine hand tracking with controller input and AR mesh geometry to build interactive training simulations.