Is this page helpful?

Quick Start#

Tutorial

Get a robot moving with a ready-made controller. Pick the path that matches your robot — both are complete, runnable examples you can drive interactively.

Path

What you get

Mobile robot

A Jetbot you steer with sliders for speed and yaw rate.

Robot arm

A Franka reaching for a target you drag around the viewport, avoiding an obstacle on the way.

Every controller in the API shares one interface — reset and forward — so the shape of the loop below is the same whichever you use, and the same whether the controller ships with Isaac Sim or you write it yourself.

Mobile Robot#

./python.sh standalone_examples/api/isaacsim.robot_motion.controllers/differential_drive.py

Construct DifferentialDriveController once, giving it the robot’s joint space and the wheel geometry:

controller = ctrl.DifferentialDriveController(
    robot_joint_space=robot.dof_names,  # every joint, not just the wheels
    left_wheel_joint=LEFT_WHEEL_JOINT,
    right_wheel_joint=RIGHT_WHEEL_JOINT,
    wheel_radius=WHEEL_RADIUS,  # metres
    wheel_base=WHEEL_BASE,  # metres, centre-to-centre
    max_linear_speed=MAX_SPEED,  # optional clamp [m/s]
    max_angular_speed=MAX_YAW_RATE,  # optional clamp [rad/s]
)

Then, each physics step, say how fast you want the robot to go:

# Both linear and angular velocities are required on the control point site.
setpoint = mg.RobotState(
    sites=mg.SpatialState.from_name(
        spatial_space=["control_point"],
        linear_velocities=(["control_point"], wp.array([[v, 0.0, 0.0]], dtype=wp.float32)),
        angular_velocities=(["control_point"], wp.array([[0.0, 0.0, omega]], dtype=wp.float32)),
    )
)

…and apply what comes back:

desired_state = controller.forward(mg.RobotState(), setpoint, 0.0)

if desired_state is not None and desired_state.joints is not None:
    if desired_state.joints.velocities is not None:
        robot.set_dof_velocity_targets(
            desired_state.joints.velocities,
            dof_indices=robot.get_dof_indices(desired_state.joints.velocity_names),
        )

You never compute a wheel speed. You state the motion you want a point on the robot to have, and the controller works out what the joints must do to produce it.

That point is a site — a named frame on the robot. Here the site is "control_point", and what you give it is a twist: a linear and an angular velocity. The arm example below commands a site too, just with a pose instead of a twist.

The output carries the joint names it applies to, so get_dof_indices(...) maps them onto the articulation for you.

Drag the two sliders to change the speed and yaw rate while it drives.

Robot Arm#

./python.sh standalone_examples/api/isaacsim.robot_motion.cumotion/rmpflow_follow_target.py

Setting Up the Scene#

The goal the arm chases is nothing special — just a cube you can select and move with the translate gizmo.

# The goal is an ordinary visual cube — drag it in the viewport to move the goal.
target = Cube(paths=TARGET_PRIM_PATH, sizes=0.04, positions=[0.5, 0.0, 0.7], colors=(1.0, 0.0, 0.0))
# Point the end effector down at the target rather than at it side-on.
target.set_world_poses(orientations=np.array([transform_utils.euler_angles_to_quaternion([0, np.pi, 0]).numpy()]))

The obstacle is an ordinary cube too. What makes it an obstacle is that it has a collision API applied, which is the marker the planner searches for.

# Also an ordinary cube. What makes it an obstacle is the collision API, which is
# what the scene search below looks for.
Cube(OBSTACLE_PRIM_PATH, sizes=0.05, positions=[np.array([0.4, 0.0, 0.45])], colors=(0.4, 0.4, 0.4))
GeomPrim(OBSTACLE_PRIM_PATH, apply_collision_apis=True)

Building the Planning World#

A planner needs to know two things: what the robot is, and what is around it.

The first is a robot configuration — the arm’s kinematics and the collision spheres used to check it against the world. cuMotion ships tuned configurations for supported robots, so for a Franka you just ask for one by name.

# cuMotion ships tuned configurations for supported robots, so you do not have to
# describe the arm's kinematics or collision spheres yourself.
robot_config = load_cumotion_supported_robot("franka")

The second comes from the stage. SceneQuery searches for prims the planner should care about — here, everything carrying a collision API within a box around the robot.

# SceneQuery finds prims on the stage that the planner should treat as obstacles.
# Here: everything with a collision API in a 20 m box around the robot, minus the robot.
robot_positions, robot_orientations = robot.get_world_poses()
obstacles = SceneQuery().get_prims_in_aabb(
    search_box_origin=robot_positions.numpy()[0],
    search_box_minimum=[-10.0, -10.0, -10.0],
    search_box_maximum=[10.0, 10.0, 10.0],
    tracked_api=TrackableApi.PHYSICS_COLLISION,
    exclude_prim_paths=[ROBOT_PRIM_PATH],
)

Those prims still have to be turned into something a planner can test against. ObstacleStrategy decides how, per shape type. Our obstacle is a cube, so that is the one to configure — here setting the safety padding the arm keeps around it.

# ObstacleStrategy says how each kind of shape is represented for planning, and how
# much clearance to keep around it. Our obstacle is a Cube, so configure that: keep
# its cube shape, and hold the arm 2 cm clear of it.
obstacle_strategy = ObstacleStrategy()
obstacle_strategy.set_default_configuration(Cube, ObstacleConfiguration("cube", 0.02))

WorldBinding ties those pieces together and keeps them live. It builds the planner’s collision world once, then tracks the prims so that moving one on the stage moves it for the planner too.

# WorldBinding is the live link between the USD stage and the planner's collision
# world: it builds that world once, then keeps it in step as prims move.
world_binding = WorldBinding(
    world_interface=CumotionWorldInterface(device="cpu"),
    obstacle_strategy=obstacle_strategy,
    tracked_prims=obstacles,
    tracked_collision_api=TrackableApi.PHYSICS_COLLISION,
)

# populate the cuMotion planning world:
world_binding.initialize()

# update the robot position in the planning world:
world_binding.get_world_interface().update_world_to_robot_root_transforms(poses=(robot_positions, robot_orientations))

# Run this whenever you want to synchronize
# the poses of obstacles in the planning world
# to the USD scene.
world_binding.synchronize_transforms()

Building the Controller#

RmpFlowController takes the robot configuration and that world, and turns a desired pose into joint position targets.

It is commanded through a site, exactly like the mobile robot. The difference is which site and what you give it: for an arm the site of interest is the tool frame at the end effector, and you give it a pose rather than a twist.

# Sites are the named frames you can command. The robot configuration lists the ones
# cuMotion knows about; the first is the arm's tool frame (its end effector).
robot_joint_space = robot.dof_names
robot_site_space = robot_config.robot_description.tool_frame_names()
tool_frame = robot_site_space[0]

controller = RmpFlowController(
    cumotion_robot=robot_config,
    cumotion_world_interface=world_binding.get_world_interface(),
    robot_joint_space=robot_joint_space,
    robot_site_space=robot_site_space,
    tool_frame=tool_frame,
)

Running It#

A setpoint is a RobotState describing what you want the site to do — read straight off the cube each step.

def read_target_setpoint() -> mg.RobotState:
    """Wrap the target cube's current pose as a setpoint on the tool frame."""
    positions, orientations = target.get_world_poses()
    return mg.RobotState(
        sites=mg.SpatialState.from_name(
            spatial_space=robot_site_space,
            positions=([tool_frame], positions),
            orientations=([tool_frame], orientations),
        )
    )

reset() tells the controller where the robot currently is, so it starts from the real configuration rather than guessing. This is the one place the measured joint state is needed.

# reset() seeds the controller with where the robot actually is. It is the only place
# the measured joint state is needed — forward() runs open-loop from there.
measured_state = mg.RobotState(
    joints=mg.JointState.from_name(
        robot_joint_space=robot_joint_space,
        positions=(robot_joint_space, robot.get_dof_positions()),
        velocities=(robot_joint_space, robot.get_dof_velocities()),
    )
)
if not controller.reset(measured_state, read_target_setpoint(), 0.0):
    raise RuntimeError("RmpFlowController failed to reset.")

The loop is then the same three beats as the mobile robot — setpoint, forward(), apply — plus one line to keep the planning world in step with the stage.

sim_time = 0.0
frame_count = 0

while simulation_app.is_running():
    simulation_app.update()

    # Push any prim movement into the planning world, then solve for this step.
    world_binding.synchronize_transforms()
    desired_state = controller.forward(None, read_target_setpoint(), sim_time)

    if desired_state is not None and desired_state.joints is not None and desired_state.joints.positions is not None:
        robot.set_dof_position_targets(
            positions=desired_state.joints.positions,
            dof_indices=robot.get_dof_indices(desired_state.joints.position_names),
        )

    sim_time += PHYSICS_DT
    frame_count += 1
    if args.test and frame_count >= 10:
        break

Drag the target around the scene and the tool follows, avoiding the obstacle. Move the obstacle and the robot reacts; WorldBinding.synchronize_transforms() keeps the cuMotion planning world updated to the latest pose of any obstacle it was initialized with.

Next Steps#

Both examples above are a few dozen lines because the hard parts are already written. Where to go next depends on what you need:

  • Controllers — ready-made mobile-base controllers: differential drive, Ackermann, and holonomic.

  • cuMotion — beyond reactive RMPflow, cuMotion also does collision-free path planning and trajectory optimization.

  • PINK — differential IK with weighted tasks and safety barriers, for when you want to specify a pose and a posture and a constraint at once.

  • Motion Generation API — the framework itself: combining controllers into larger behaviors, driving them from a state machine, populating a planning world from USD, and writing your own controller when nothing off the shelf fits.

Back to Gallery View