Interactive Robot Control with Gizmo

Interactive Robot Control with Gizmo#

This tutorial demonstrates native DexSim and browser-based Viser Gizmo control. DexSim owns native entity and robot IK controllers; EmbodiChain keeps only the robot control-part adapter, Viser commands, and controller lifecycle management.

For the cross-frontend capability summary, supported targets, lifecycle rules, and security boundary, see Interactive Gizmos.

The Code#

The tutorial corresponds to the gizmo_robot.py script in the scripts/tutorials/sim directory.

Code for gizmo_robot.py
  1# ----------------------------------------------------------------------------
  2# Copyright (c) 2021-2026 DexForce Technology Co., Ltd.
  3#
  4# Licensed under the Apache License, Version 2.0 (the "License");
  5# you may not use this file except in compliance with the License.
  6# You may obtain a copy of the License at
  7#
  8#     http://www.apache.org/licenses/LICENSE-2.0
  9#
 10# Unless required by applicable law or agreed to in writing, software
 11# distributed under the License is distributed on an "AS IS" BASIS,
 12# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 13# See the License for the specific language governing permissions and
 14# limitations under the License.
 15# ----------------------------------------------------------------------------
 16"""Control a UR10 end effector with a Gizmo and manual physics stepping."""
 17
 18from __future__ import annotations
 19
 20import time
 21import torch
 22import numpy as np
 23import argparse
 24
 25from embodichain.lab.sim import SimulationManager, SimulationManagerCfg
 26from embodichain.lab.sim.objects import GizmoCfg
 27from embodichain.lab.visualization import visualization_cfg_from_args
 28from embodichain.lab.gym.utils.gym_utils import add_env_launcher_args_to_parser
 29from embodichain.lab.sim.cfg import (
 30    RenderCfg,
 31    RobotCfg,
 32    URDFCfg,
 33    JointDrivePropertiesCfg,
 34)
 35
 36from embodichain.lab.sim.motion.solvers import PinkSolverCfg
 37from embodichain.data import get_data_path
 38from embodichain.utils import logger
 39
 40
 41def main():
 42    """Main function to create and run the simulation scene."""
 43
 44    # Parse command line arguments
 45    parser = argparse.ArgumentParser(
 46        description="Create a simulation scene with SimulationManager"
 47    )
 48    add_env_launcher_args_to_parser(parser)
 49    args = parser.parse_args()
 50
 51    # Configure the simulation
 52    sim_cfg = SimulationManagerCfg(
 53        width=1920,
 54        height=1080,
 55        headless=True,
 56        physics_dt=1.0 / 100.0,
 57        sim_device=args.device,
 58        render_cfg=RenderCfg(renderer=args.renderer),
 59        visualization=visualization_cfg_from_args(args),
 60        robot_ik_gizmo=GizmoCfg(ik_start_enabled=True),
 61    )
 62
 63    sim = SimulationManager(sim_cfg)
 64
 65    # Get UR10 URDF path
 66    urdf_path = get_data_path("UniversalRobots/UR10/UR10.urdf")
 67
 68    # Create UR10 robot
 69    robot_cfg = RobotCfg(
 70        uid="ur10_gizmo_test",
 71        urdf_cfg=URDFCfg(
 72            components=[{"component_type": "arm", "urdf_path": urdf_path}]
 73        ),
 74        control_parts={"arm": ["Joint[1-6]"]},
 75        solver_cfg={
 76            "arm": PinkSolverCfg(
 77                urdf_path=urdf_path,
 78                end_link_name="ee_link",
 79                root_link_name="base_link",
 80                pos_eps=1e-2,
 81                rot_eps=5e-2,
 82                max_iterations=300,
 83                dt=0.1,
 84            )
 85        },
 86        drive_pros=JointDrivePropertiesCfg(
 87            stiffness={"Joint[1-6]": 1e4},
 88            damping={"Joint[1-6]": 1e3},
 89        ),
 90    )
 91    robot = sim.add_robot(cfg=robot_cfg)
 92    if sim.is_use_gpu_physics:
 93        sim.init_gpu_physics()
 94
 95    # Set initial joint positions
 96    initial_qpos = torch.tensor(
 97        [[0, -np.pi / 2, np.pi / 2, 0.0, np.pi / 2, 0.0]],
 98        dtype=torch.float32,
 99        device=sim.device,
100    )
101    joint_ids = robot.get_joint_ids("arm")
102    robot.set_qpos(qpos=initial_qpos, joint_ids=joint_ids, target=False)
103    robot.set_qpos(qpos=initial_qpos, joint_ids=joint_ids)
104
105    sim.update(step=1)  # Refresh link poses before creating the IK target.
106
107    native_window_opened = False
108    if not args.headless:
109        native_window_opened = sim.open_window()
110
111    if not native_window_opened and not args.viser:
112        logger.log_warning(
113            "Gizmo interaction is disabled in headless mode without Viser."
114        )
115
116    logger.log_info("Gizmo-Robot example started!")
117    if native_window_opened or args.viser:
118        logger.log_info("Use the gizmo to drag the robot end-effector (EE)")
119    if native_window_opened:
120        logger.log_info(
121            "Native robot IK Gizmo starts enabled; press I to show or hide it"
122        )
123    logger.log_info("Press Ctrl+C to stop the simulation")
124
125    run_simulation(sim)
126
127
128def run_simulation(sim: SimulationManager) -> None:
129    """Advance physics; the manager owns native and Viser IK interaction."""
130    step_count = 0
131    try:
132        last_time = time.perf_counter()
133        last_step = 0
134        while True:
135            frame_start = time.perf_counter()
136            # update() owns IK interaction, physics stepping, and Viser capture.
137            sim.update(step=1)
138            step_count += 1
139
140            if step_count % 100 == 0:
141                current_time = time.perf_counter()
142                elapsed = current_time - last_time
143                fps = (
144                    sim.num_envs * (step_count - last_step) / elapsed
145                    if elapsed > 0
146                    else 0
147                )
148                logger.log_info(f"Simulation step: {step_count}, FPS: {fps:.2f}")
149                last_time = current_time
150                last_step = step_count
151
152            elapsed = time.perf_counter() - frame_start
153            time.sleep(max(0.0, sim.sim_config.physics_dt - elapsed))
154    except KeyboardInterrupt:
155        logger.log_info("\nStopping simulation...")
156    finally:
157        sim.destroy()
158        logger.log_info("Simulation terminated successfully")
159
160
161if __name__ == "__main__":
162    main()

The Code Explained#

Similar to the previous tutorial on robot simulation, we use the SimulationManager class to set up the simulation environment. If you haven’t read that tutorial yet, please refer to Simulating a Robot first.

Important: Gizmo supports a single environment (num_envs=1). Automatic registration is skipped for multi-environment simulations.

Robot Gizmo registration, updates, visibility, and destruction are managed by SimulationManager:

# Toggle visibility for a gizmo
sim.toggle_gizmo_visibility("ur10_gizmo_test", control_part="arm")

# Set visibility explicitly
sim.set_gizmo_visibility("ur10_gizmo_test", visible=False, control_part="arm")

Native interaction uses DexSim controllers. The standard Viser mode includes interactive Gizmo control:

python scripts/tutorials/sim/gizmo_robot.py --viser

Only expose the Viser endpoint to trusted browser clients because dragging a Gizmo mutates simulation targets.

Click-to-Pick in Viser#

Unlike the native DexSim window, the browser does not ray-cast the scene for you, so EmbodiChain performs the click hit-test against the published scene geometry. Enable it explicitly in the browser panel:

  1. Toggle the Enable click-to-pick Gizmo checkbox under the Interaction folder.

  2. Click a rigid object or robot link in the 3D view. A transform control is attached to it (replacing any previously picked Gizmo); drag it to move the target. Robot IK is solved with DexSim Newton IK, just as in the native window.

  3. Click empty space, or uncheck the checkbox, to detach the picker-owned Gizmo.

The picker manages at most one Gizmo at a time and never touches Gizmos you created yourself through sim.enable_gizmo(...). Only rigid objects and robots are pickable; articulations, soft bodies, and cameras are ignored by the picker.

What is a Gizmo?#

A Gizmo is an interactive visual tool that allows users to manipulate simulation objects in real-time through mouse interactions. In robotics applications, gizmos are particularly useful for:

  • Interactive Robot Control: Drag the robot’s end-effector to desired positions

  • Inverse Kinematics: Automatically solve joint angles to reach target poses

  • Real-time Manipulation: Provide immediate visual feedback during robot motion planning

  • Debugging and Visualization: Test robot reachability and workspace limits

The objects.Gizmo class manages native robot controllers and Viser targets. Native entity manipulation remains owned by DexSim.

Setting up Robot Configuration#

First, we configure a UR10 robot with an IK solver for end-effector control:

    # Create UR10 robot
    robot_cfg = RobotCfg(
        uid="ur10_gizmo_test",
        urdf_cfg=URDFCfg(
            components=[{"component_type": "arm", "urdf_path": urdf_path}]
        ),
        control_parts={"arm": ["Joint[1-6]"]},
        solver_cfg={
            "arm": PinkSolverCfg(
                urdf_path=urdf_path,
                end_link_name="ee_link",
                root_link_name="base_link",
                pos_eps=1e-2,
                rot_eps=5e-2,
                max_iterations=300,
                dt=0.1,
            )
        },
        drive_pros=JointDrivePropertiesCfg(
            stiffness={"Joint[1-6]": 1e4},
            damping={"Joint[1-6]": 1e3},
        ),
    )
    robot = sim.add_robot(cfg=robot_cfg)

Key components of the robot configuration:

  • URDF Configuration: Loads the robot’s kinematic and visual model

  • Control Parts: Defines which joints can be controlled ("Joint[1-6]" for UR10)

  • IK Solver: solvers.PinkSolverCfg supplies chain metadata and an optional solver override

  • Drive Properties: Sets stiffness and damping for joint control

The configured EmbodiChain solver is optional: it supplies default IK-chain metadata (root link, end link, and TCP transform). IK itself is solved by DexSim Newton IK. Applications may instead set this metadata directly in objects.GizmoCfg.

Automatic Robot Controls#

SimulationManager discovers each control part with existing root-link and end-link metadata and uses its configured TCP transform. This tutorial explicitly activates native IK at startup with one setting:

sim_cfg = SimulationManagerCfg(
    robot_ik_gizmo=GizmoCfg(ik_start_enabled=True),
)

The first update after opening the window creates and displays the controller. The tutorial sets current joint positions and drive targets before opening the window; activation initializes the controller from that pose.

  • In this tutorial’s native window, IK targets start visible. Press I to hide or show them. Ordinary simulations retain ik_start_enabled=False and wait for the first I press to activate.

  • In Viser, the TCP controls are available automatically when commands are allowed. The solver is constructed on the first drag.

  • Pure headless, read-only Viser, and multi-environment simulations skip automatic registration.

Registration alone does not initialize another IK solver or overwrite drive targets. Explicit startup activation initializes the controller once; closing and reopening the window preserves its later visibility state. DexSim Newton IK is the default. To use the robot’s configured solver instead:

from embodichain.lab.sim.objects import GizmoCfg

sim_cfg = SimulationManagerCfg(
    robot_ik_gizmo=GizmoCfg(ik_solver="embodichain"),
)

Set robot_ik_gizmo=None to disable automatic setup. Robots without chain metadata can still use sim.enable_gizmo(...) with explicit objects.GizmoCfg link settings. Advanced callers can use objects.create_robot_ik_gizmo_controller() and manage its updates directly; SimulationManager will not create a duplicate native controller for that part.

How Gizmo-Robot Interaction Works#

The gizmo-robot interaction follows this workflow:

  1. Target Update: DexSim or Viser records the requested TCP transform

  2. Deferred Solve: the native controller or sim.update_gizmos() invokes Newton IK only when needed

  3. State Bridge: Newton IK reads and writes the selected EmbodiChain control-part joints through an adapter

  4. Drive Target: Both paths use Robot.set_qpos(..., target=True)

  5. Robot Motion: Joint drives move the robot toward the target without teleporting its current state

The Simulation Loop#

The tutorial uses manual physics only. After setting initial joint positions and drive targets, each iteration advances one physics step:

def run_simulation(sim: SimulationManager) -> None:
    """Advance physics; the manager owns native and Viser IK interaction."""
    step_count = 0
    try:
        last_time = time.perf_counter()
        last_step = 0
        while True:
            frame_start = time.perf_counter()
            # update() owns IK interaction, physics stepping, and Viser capture.
            sim.update(step=1)

sim.update() processes native and Viser interaction, advances physics, and publishes visualization state. No separate controller update is required. The tutorial paces the loop using physics_dt and releases resources with sim.destroy() on Ctrl+C.

Gizmo Lifecycle Management#

SimulationManager handles automatic robot control registration and cleanup. Closing a native window detaches input handlers; reopening it reuses existing controllers and preserves their visibility without writing new drive targets. Removing a robot also removes its managed controls.

For explicit overrides:

  • sim.enable_gizmo(uid, control_part, gizmo_cfg) replaces that part’s settings.

  • sim.disable_gizmo(uid, control_part) disables one part and prevents automatic recreation; omitting the part disables every part of the robot.

  • sim.toggle_gizmo_visibility(uid, control_part) and sim.set_gizmo_visibility(uid, visible, control_part) control visibility.

Running the Tutorial#

To run the gizmo robot tutorial:

cd scripts/tutorials/sim
python gizmo_robot.py --device cpu

Command-line options:

  • --device cpu|cuda: Choose simulation device

  • --num_envs N: Number of parallel environments

  • --headless: Run without GUI for automated testing

  • --renderer auto|hybrid|fast-rt|rt: Select the renderer

  • --viser: Use browser-based interaction

Once running:

  1. Open: The native IK target starts visible, or open the Viser page

  2. Mouse Interaction: Click and drag the gizmo to move the robot

  3. Real-time IK: Watch the robot joints automatically adjust to follow the gizmo

  4. Workspace Limits: Observe how the robot behaves at workspace boundaries

  5. Performance: Monitor FPS in the console output

Tips and Best Practices#

Performance optimization:

  • Use sim.update(step=1) to service interaction and advance manual physics

  • Reduce IK solver iterations for better real-time performance if needed

  • Pace manual steps using physics_dt

Debugging tips:

  • Check console output for IK solver success/failure messages

  • Inspect the robot TCP or Viser target pose when debugging alignment

  • Monitor FPS to identify performance bottlenecks

Robot compatibility:

  • Set the IK chain (root link and end-effector link) in objects.GizmoCfg, or configure an EmbodiChain solver to supply them as defaults

  • Check the end-effector (EE) link name

  • Test joint limits and workspace boundaries

Visualization customization:

  • Adjust Viser axis lengths, ring radius, and line width through objects.GizmoCfg

  • Adjust gizmo scale according to robot size

  • Enable collision for debugging if needed

Next Steps#

After mastering basic gizmo usage, you can explore:

  • Multi-robot Gizmos: Attach gizmos to multiple robots simultaneously

  • Gizmo with Rigid Objects: Use gizmos for interactive object manipulation

  • Advanced IK Configuration: Fine-tune solver parameters for specific robots

For more advanced robot control and simulation features, refer to the complete Simulating a Robot tutorial and the API documentation for objects.Gizmo and solvers.PinkSolverCfg.

Contents