Adding a Manipulator Robot#

Tutorial

Learning Objectives#

This tutorial introduces a manipulator robot to the simulation, a Franka Panda. It describes how to add the robot to the scene and execute a pick-and-place operation. After this tutorial, you will have more experience using manipulator robots and controlling them with the Robot Motion API in NVIDIA Isaac Sim.

15-20 Minute Tutorial

Getting Started#

Prerequisites

Review Hello Robot prior to beginning this tutorial.

This tutorial uses standalone Python scripts. Run them with a Python environment where Isaac Sim is installed:

Creating the Scene with a Franka Robot#

Use ManipulationScenario to add the ground and a configured Franka asset. The scenario owns a plain Articulation; robot-specific USD variants, named joint defaults, controller frame, measured link, and grasp offset are declarative data. Scene setup is therefore separate from control logic.

"""Create a scene with ground, Franka robot, and blue cube."""

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--test", action="store_true")
args, _ = parser.parse_known_args()

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": False})

import isaacsim.core.experimental.utils.app as app_utils

app_utils.enable_extension("isaacsim.robot_motion.examples")

from isaacsim.core.experimental.objects import Cube
from isaacsim.core.experimental.prims import GeomPrim, RigidPrim
from isaacsim.core.simulation_manager import SimulationManager
from isaacsim.robot_motion.examples.manipulation import ManipulationScenario

DEVICE = "cpu"

# Robot-specific defaults and tool transforms come from the named configuration.
scenario = ManipulationScenario("franka")
articulation = scenario.setup_scene()

# Create a blue cube for the robot to pick up
cube_shape = Cube(
    paths="/World/Cube",
    positions=[0.5, 0.0, 0.0258],
    sizes=1.0,
    scales=[0.0515, 0.0515, 0.0515],
    colors="blue",
)
GeomPrim(paths=cube_shape.paths, apply_collision_apis=True)
RigidPrim(paths=cube_shape.paths)

SimulationManager.setup_simulation(dt=1.0 / 60.0, device=DEVICE)
physics_scene = SimulationManager.get_physics_scenes()[0]
physics_scene.set_enabled_gpu_dynamics(False)
app_utils.play()
app_utils.update_app(steps=20)

step_count = 0
max_test_steps = 60
while simulation_app.is_running():
    simulation_app.update()
    step_count += 1
    if args.test and step_count >= max_test_steps:
        break

scenario.cleanup()
app_utils.stop()
simulation_app.close()

Run the script. A window opens with the Franka robot and cube in the scene; the simulation runs until you close the window.

The scenario provides the simulation boundary:

read_robot_state() reads named joints and the measured controller site
apply_robot_state() applies controller output to the articulation
sync_world() updates the RMPFlow collision world
articulation.reset_to_default_state() restores the configured robot defaults

Using the Pick-and-Place Task#

PickPlaceTask creates the scene and constructs the controllers directly. An RMPFlow arm controller runs with a robot-independent PickPlaceController and an observable gripper controller. They communicate through RobotState; no robot subclass or inverse-kinematics wrapper is required.

"""Pick-and-place using the generic motion-generation controller."""

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--test", action="store_true")
args, _ = parser.parse_known_args()

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": False})

import isaacsim.core.experimental.utils.app as app_utils

app_utils.enable_extension("isaacsim.robot_motion.examples")

from isaacsim.core.simulation_manager import SimulationManager
from isaacsim.robot_motion.examples.manipulation import PickPlaceTask

DEVICE = "cpu"

# The task composes RMPFlow, the feedback-driven state machine, and the gripper.
task = PickPlaceTask()
task.setup_scene()

SimulationManager.setup_simulation(dt=1.0 / 60.0, device=DEVICE)
physics_scene = SimulationManager.get_physics_scenes()[0]
physics_scene.set_enabled_gpu_dynamics(False)
app_utils.play()
# Run a few steps so the articulation's physics tensor entity is valid before `task.initialize()`
app_utils.update_app(steps=20)
task.initialize()
task.reset()

# Main loop: run one pick-place step each physics frame until done
step_count = 0
max_test_steps = 600
while simulation_app.is_running():
    simulation_app.update()
    step_count += 1
    if app_utils.is_playing():
        if not task.is_done:
            task.step(1.0 / 60.0)
            if task.failed:
                raise RuntimeError(str(task.status()["failure_reason"]))
        else:
            print("Pick-and-place completed")
            app_utils.pause()
            if args.test:
                break
    if args.test and step_count >= max_test_steps:
        raise RuntimeError("Pick-and-place did not complete within the test step budget")

task.cleanup()
app_utils.stop()
simulation_app.close()

Run the script. The robot automatically executes all phases of picking up and placing the cube.

Understanding the Pick-and-Place State Machine#

PickPlaceController generates controller-frame targets for these phases:

Pick-and-Place Phases#
Phase	Description	Completion condition
`APPROACH_PICK`	Move above the pick goal	Measured tool pose is stable at the target
`DESCEND_PICK`	Move to the pick goal	Measured tool pose is stable at the target
`GRASP`	Close the gripper	Tool pose is stable and the gripper reports completion
`LIFT`	Lift the object	Measured tool pose is stable at the target
`APPROACH_PLACE`	Move above the place goal	Measured tool pose is stable at the target
`DESCEND_PLACE`	Move to the place goal	Measured tool pose is stable at the target
`RELEASE`	Open the gripper	Tool pose is stable and the gripper reports completion
`RETREAT`	Move away from the placed object	Measured tool pose is stable at the target

Each active phase has a timeout that produces a failure; time alone never marks a motion phase complete. Pick and place goals are captured when the controller resets. Customize the task by changing its cube and place positions before scene setup:

# Goals are captured at reset; configure task positions before scene setup.
task = PickPlaceTask(cube_positions=[(0.4, 0.2, 0.0258)])
task.place_position = np.asarray([-0.4, 0.2, 0.0258], dtype=np.float32)
task.setup_scene()

Complete code:

"""Customize the generic pick-and-place task."""

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--test", action="store_true")
args, _ = parser.parse_known_args()

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": False})

import isaacsim.core.experimental.utils.app as app_utils

app_utils.enable_extension("isaacsim.robot_motion.examples")

import numpy as np
from isaacsim.core.simulation_manager import SimulationManager
from isaacsim.robot_motion.examples.manipulation import PickPlaceTask

DEVICE = "cpu"

# -- Begin custom setup -- #
# Goals are captured at reset; configure task positions before scene setup.
task = PickPlaceTask(cube_positions=[(0.4, 0.2, 0.0258)])
task.place_position = np.asarray([-0.4, 0.2, 0.0258], dtype=np.float32)
task.setup_scene()
# -- End of custom setup -- #

SimulationManager.setup_simulation(dt=1.0 / 60.0, device=DEVICE)
physics_scene = SimulationManager.get_physics_scenes()[0]
physics_scene.set_enabled_gpu_dynamics(False)
app_utils.play()
# Run a few steps so the articulation's physics tensor entity is valid before `task.initialize()`
app_utils.update_app(steps=20)
task.initialize()
task.reset()

# Main loop: run one pick-place step each physics frame until done
step_count = 0
max_test_steps = 600
while simulation_app.is_running():
    simulation_app.update()
    step_count += 1
    if app_utils.is_playing():
        if not task.is_done:
            task.step(1.0 / 60.0)
            if task.failed:
                raise RuntimeError(str(task.status()["failure_reason"]))
        else:
            print("Pick-and-place completed")
            app_utils.pause()
            if args.test:
                break
    if args.test and step_count >= max_test_steps:
        raise RuntimeError("Pick-and-place did not complete within the test step budget")

task.cleanup()
app_utils.stop()
simulation_app.close()

Summary#

This tutorial covered the following topics:

Adding a configured Franka as a plain Articulation with ManipulationScenario
Constructing RMPFlow, pick-and-place, and gripper controllers explicitly
Exchanging measured and desired state through RobotState
Using feedback-driven pick-and-place phases with explicit failure semantics

Next Steps#

Continue to the next tutorial in our Essential Tutorials series, Adding Multiple Robots, to learn how to add multiple robots to the simulation.

Back to Gallery View