Multiple Robot Scenarios#

Learning Objectives#

This tutorial describes how to create and manage multiple robot scenarios in NVIDIA Isaac Sim. It explains how to use parameterization and Python classes to scale your simulations with multiple instances of robots performing similar tasks. After this tutorial, you will have more experience building scalable multi-robot simulations in NVIDIA Isaac Sim.

15-20 Minute Tutorial

Getting Started#

Prerequisites

Begin with the source code open from the previous tutorial, Adding Multiple Robots.

Note

Pressing STOP, then PLAY in this workflow might not reset the world properly. Use the RESET button instead.

Organizing Robot Scenarios with Classes#

When working with multiple robots performing similar tasks, it’s helpful to encapsulate the robot setup and control logic into reusable classes. This approach allows you to easily create multiple instances with different parameters (like position offsets).

Create a RobotScenario class that manages a Jetbot pushing a cube to a Franka:

  1import isaacsim.core.experimental.utils.stage as stage_utils
  2from isaacsim.core.experimental.prims import Articulation, XformPrim, GeomPrim, RigidPrim
  3from isaacsim.core.experimental.objects import Cube
  4from isaacsim.core.experimental.materials import PreviewSurfaceMaterial
  5from isaacsim.examples.base.base_sample_experimental import BaseSample
  6from isaacsim.core.simulation_manager import SimulationManager
  7from isaacsim.robot.manipulators.examples.franka import FrankaExperimental
  8from isaacsim.storage.native import get_assets_root_path
  9import numpy as np
 10
 11
 12class RobotScenario:
 13    """Encapsulates a Jetbot + Franka + Cube scenario with an offset."""
 14
 15    def __init__(self, name: str, offset: np.ndarray = np.array([0.0, 0.0, 0.0])):
 16        self.name = name
 17        self.offset = offset
 18        self.state = 0
 19        self.step_counter = 0
 20        self.pick_phase = 0
 21        self.jetbot = None
 22        self.franka = None
 23        self.cube = None
 24        self.cube_goal = np.array([1.2, 0.0, 0.0]) + offset
 25
 26    def setup_scene(self):
 27        """Create the robots and cube for this scenario."""
 28        assets_root_path = get_assets_root_path()
 29        base_path = f"/World/{self.name}"
 30
 31        # Add Jetbot
 32        stage_utils.add_reference_to_stage(
 33            usd_path=assets_root_path + "/Isaac/Robots/NVIDIA/Jetbot/jetbot.usd",
 34            path=f"{base_path}/Jetbot",
 35        )
 36        jetbot_xform = XformPrim(f"{base_path}/Jetbot")
 37        jetbot_xform.reset_xform_op_properties()
 38        jetbot_xform.set_world_poses(positions=[self.offset.tolist()])
 39
 40        # Add cube in front of Jetbot
 41        cube_pos = self.offset + np.array([0.15, 0.0, 0.025])
 42        visual_material = PreviewSurfaceMaterial(f"{base_path}/Materials/red")
 43        visual_material.set_input_values("diffuseColor", [1.0, 0.0, 0.0])
 44        cube_shape = Cube(
 45            paths=f"{base_path}/Cube",
 46            positions=np.array([cube_pos]),
 47            sizes=[1.0],
 48            scales=np.array([[0.05, 0.05, 0.05]]),
 49            reset_xform_op_properties=True,
 50        )
 51        GeomPrim(paths=cube_shape.paths, apply_collision_apis=True)
 52        RigidPrim(paths=cube_shape.paths)
 53        cube_shape.apply_visual_materials(visual_material)
 54
 55        # Add Franka
 56        franka_pos = self.offset + np.array([0.8, -0.3, 0.0])
 57        self.franka = FrankaExperimental(
 58            robot_path=f"{base_path}/Franka", create_robot=True
 59        )
 60        franka_xform = XformPrim(f"{base_path}/Franka")
 61        franka_xform.reset_xform_op_properties()
 62        franka_xform.set_world_poses(positions=[franka_pos.tolist()])
 63
 64    def initialize(self):
 65        """Initialize articulation handles after scene load."""
 66        base_path = f"/World/{self.name}"
 67        self.jetbot = Articulation(f"{base_path}/Jetbot")
 68        self.cube = RigidPrim(f"{base_path}/Cube")
 69
 70    def reset(self):
 71        """Reset the scenario state."""
 72        self.state = 0
 73        self.step_counter = 0
 74        self.pick_phase = 0
 75        self.franka.reset_to_default_pose()
 76
 77    def step(self):
 78        """Execute one step of the scenario logic."""
 79        if self.state == 0:
 80            # Jetbot pushes cube
 81            cube_pos = self.cube.get_world_poses()[0].numpy()[0]
 82            if np.linalg.norm(cube_pos[:2] - self.cube_goal[:2]) > 0.05:
 83                self.jetbot.set_dof_velocity_targets([[10.0, 10.0]])
 84            else:
 85                self.jetbot.set_dof_velocity_targets([[0.0, 0.0]])
 86                self.state = 1
 87                self.step_counter = 0
 88
 89        elif self.state == 1:
 90            # Jetbot backs up
 91            self.jetbot.set_dof_velocity_targets([[-8.0, -8.0]])
 92            self.step_counter += 1
 93            if self.step_counter > 100:
 94                self.jetbot.set_dof_velocity_targets([[0.0, 0.0]])
 95                self.state = 2
 96                self.step_counter = 0
 97                self.franka.open_gripper()
 98
 99        elif self.state == 2:
100            # Franka pick-and-place
101            self._franka_pick_place()
102
103    def _franka_pick_place(self):
104        """Execute Franka pick-and-place state machine."""
105        cube_pos = self.cube.get_world_poses()[0].numpy()[0]
106        down_orient = self.franka.get_downward_orientation()
107        self.step_counter += 1
108
109        if self.pick_phase == 0:
110            self.franka.set_end_effector_pose(
111                np.array([[cube_pos[0], cube_pos[1], cube_pos[2] + 0.2]]), down_orient)
112            if self.step_counter > 120:
113                self.pick_phase = 1
114                self.step_counter = 0
115        elif self.pick_phase == 1:
116            self.franka.set_end_effector_pose(
117                np.array([[cube_pos[0], cube_pos[1], cube_pos[2] + 0.1]]), down_orient)
118            if self.step_counter > 100:
119                self.franka.close_gripper()
120                self.pick_phase = 2
121                self.step_counter = 0
122        elif self.pick_phase == 2:
123            self.franka.close_gripper()
124            if self.step_counter > 50:
125                self.pick_phase = 3
126                self.step_counter = 0
127        elif self.pick_phase == 3:
128            self.franka.set_end_effector_pose(
129                np.array([[cube_pos[0], cube_pos[1], cube_pos[2] + 0.25]]), down_orient)
130            if self.step_counter > 100:
131                self.pick_phase = 4
132                self.step_counter = 0
133        elif self.pick_phase == 4:
134            target = self.offset + np.array([0.3, 0.3, 0.15])
135            self.franka.set_end_effector_pose(np.array([target]), down_orient)
136            if self.step_counter > 150:
137                self.franka.open_gripper()
138                self.step_counter = 0
139                self.pick_phase = 5
140        elif self.pick_phase == 5:
141            # Lift the arm from target position (don't use cube_pos - cube was dropped)
142            target = self.offset + np.array([0.3, 0.3, 0.4])  # Lift above drop location
143            self.franka.set_end_effector_pose(np.array([target]), down_orient)
144            if self.step_counter > 150:
145                self.step_counter = 0
146                self.state = 5  # Done
147
148
149class HelloWorld(BaseSample):
150    def __init__(self) -> None:
151        super().__init__()
152        self._physics_callback_id = None
153        self._scenario = None
154
155    def setup_scene(self):
156        # Add ground plane
157        stage_utils.add_reference_to_stage(
158            usd_path=get_assets_root_path() + "/Isaac/Environments/Grid/default_environment.usd",
159            path="/World/ground",
160        )
161        # Create a single scenario
162        self._scenario = RobotScenario(name="scenario_0", offset=np.array([0.0, 0.0, 0.0]))
163        self._scenario.setup_scene()
164
165    async def setup_post_load(self):
166        self._scenario.initialize()
167
168        from isaacsim.core.simulation_manager.impl.isaac_events import IsaacEvents
169        self._physics_callback_id = SimulationManager.register_callback(
170            self.physics_step, IsaacEvents.POST_PHYSICS_STEP
171        )
172
173    def physics_step(self, dt, context):
174        self._scenario.step()
175
176    async def setup_post_reset(self):
177        self._scenario.reset()
178
179    def physics_cleanup(self):
180        if self._physics_callback_id is not None:
181            SimulationManager.deregister_callback(self._physics_callback_id)
182            self._physics_callback_id = None
../_images/core_api_tutorials_6_1.webp

Scaling to Multiple Scenarios#

The power of the class-based approach becomes clear when scaling to multiple scenarios. Simply create multiple RobotScenario instances with different offsets:

 1import isaacsim.core.experimental.utils.stage as stage_utils
 2from isaacsim.core.experimental.prims import Articulation, XformPrim, GeomPrim, RigidPrim
 3from isaacsim.core.experimental.objects import Cube
 4from isaacsim.core.experimental.materials import PreviewSurfaceMaterial
 5from isaacsim.examples.base.base_sample_experimental import BaseSample
 6from isaacsim.core.simulation_manager import SimulationManager
 7from isaacsim.robot.manipulators.examples.franka import FrankaExperimental
 8from isaacsim.storage.native import get_assets_root_path
 9import numpy as np
10
11# RobotScenario class definition (same as above)
12# ... (include the full RobotScenario class from the previous example)
13
14
15class HelloWorld(BaseSample):
16    def __init__(self) -> None:
17        super().__init__()
18        self._physics_callback_id = None
19        self._scenarios = []
20        self._num_scenarios = 2  # Number of parallel scenarios
21
22    def setup_scene(self):
23        # Add ground plane
24        stage_utils.add_reference_to_stage(
25            usd_path=get_assets_root_path() + "/Isaac/Environments/Grid/default_environment.usd",
26            path="/World/ground",
27        )
28
29        # Create multiple scenarios with Y-axis offsets
30        for i in range(self._num_scenarios):
31            offset = np.array([0.0, (i - 1) * 2.0, 0.0])  # Spread along Y-axis
32            scenario = RobotScenario(name=f"scenario_{i}", offset=offset)
33            scenario.setup_scene()
34            self._scenarios.append(scenario)
35
36    async def setup_post_load(self):
37        # Initialize all scenarios
38        for scenario in self._scenarios:
39            scenario.initialize()
40
41        from isaacsim.core.simulation_manager.impl.isaac_events import IsaacEvents
42        self._physics_callback_id = SimulationManager.register_callback(
43            self.physics_step, IsaacEvents.POST_PHYSICS_STEP
44        )
45
46    def physics_step(self, dt, context):
47        # Step all scenarios
48        for scenario in self._scenarios:
49            scenario.step()
50
51    async def setup_post_reset(self):
52        # Reset all scenarios
53        for scenario in self._scenarios:
54            scenario.reset()
55
56    def physics_cleanup(self):
57        if self._physics_callback_id is not None:
58            SimulationManager.deregister_callback(self._physics_callback_id)
59            self._physics_callback_id = None
60        self._scenarios = []
../_images/core_api_tutorials_6_2.webp

Adding Randomization#

To make simulations more interesting, you can add randomization to the scenario parameters. Modify the RobotScenario class to accept randomization options:

 1class RobotScenario:
 2    """Encapsulates a Jetbot + Franka + Cube scenario with randomization."""
 3
 4    def __init__(self, name: str, offset: np.ndarray = np.array([0.0, 0.0, 0.0]),
 5                 randomize: bool = False):
 6        self.name = name
 7        self.offset = offset
 8        self.randomize = randomize
 9        self.state = 0
10        self.step_counter = 0
11        self.pick_phase = 0
12
13        # Randomize cube goal position if enabled
14        if randomize:
15            random_x = np.random.uniform(1.1, 1.4)
16            self.cube_goal = np.array([random_x, 0.0, 0.0]) + offset
17        else:
18            self.cube_goal = np.array([1.2, 0.0, 0.0]) + offset
19
20        # ... rest of the class remains the same

Then create scenarios with randomization enabled:

 1# Create multiple scenarios with randomization
 2for i in range(self._num_scenarios):
 3    offset = np.array([0.0, (i - 1) * 2.0, 0.0])
 4    scenario = RobotScenario(
 5        name=f"scenario_{i}",
 6        offset=offset,
 7        randomize=True  # Enable randomization
 8    )
 9    scenario.setup_scene()
10    self._scenarios.append(scenario)

Best Practices for Scaling#

When creating large-scale multi-robot simulations:

  1. Use unique paths: Each scenario should use unique USD prim paths to avoid conflicts. The RobotScenario class uses the scenario name to create unique paths like /World/scenario_0/Jetbot.

  2. Manage state independently: Each scenario instance maintains its own state variables, allowing scenarios to progress independently.

  3. Clean up properly: The physics_cleanup method ensures callbacks are deregistered and scenario lists are cleared when the simulation is stopped.

  4. Consider performance: With many scenarios, consider reducing physics step frequency or using GPU-accelerated simulation for better performance.

Summary#

This tutorial covered the following topics:

  1. Organizing robot scenarios into reusable Python classes

  2. Using the offset parameter to position multiple scenarios in the world

  3. Scaling to multiple parallel scenarios with a simple loop

  4. Adding randomization to scenario parameters

  5. Best practices for managing multiple robot instances