Multiple Robot Scenarios#

Learning Objectives#

This tutorial describes how to create and manage multiple robot scenarios in NVIDIA Isaac Sim. It explains how to use parameterization and Python classes to scale your simulations with multiple instances of robots performing similar tasks. After this tutorial, you will have more experience building scalable multi-robot simulations in NVIDIA Isaac Sim.

15-20 Minute Tutorial

Getting Started#

Prerequisites

Begin with the source code open from the previous tutorial, Adding Multiple Robots.

Note

Pressing STOP, then PLAY in this workflow might not reset the world properly. Use the RESET button instead.

Organizing Robot Scenarios with Classes#

When working with multiple robots performing similar tasks, it’s helpful to encapsulate the robot setup and control logic into reusable classes. This approach allows you to easily create multiple instances with different parameters (like position offsets).

Create a RobotScenario class that manages a Jetbot pushing a cube to a Franka:

  1import isaacsim.core.experimental.utils.stage as stage_utils
  2import numpy as np
  3from isaacsim.core.experimental.materials import PreviewSurfaceMaterial
  4from isaacsim.core.experimental.objects import Cube
  5from isaacsim.core.experimental.prims import Articulation, GeomPrim, RigidPrim, XformPrim
  6from isaacsim.core.simulation_manager import SimulationManager
  7from isaacsim.examples.base.base_sample_experimental import BaseSample
  8from isaacsim.robot.manipulators.examples.franka import FrankaExperimental
  9from isaacsim.storage.native import get_assets_root_path
 10
 11
 12class RobotScenario:
 13    """Encapsulates a Jetbot + Franka + Cube scenario with an offset."""
 14
 15    def __init__(self, name: str, offset: np.ndarray = np.array([0.0, 0.0, 0.0])):
 16        self.name = name
 17        self.offset = offset
 18        self.state = 0
 19        self.step_counter = 0
 20        self.pick_phase = 0
 21        self.jetbot = None
 22        self.franka = None
 23        self.cube = None
 24        self.cube_goal = np.array([1.2, 0.0, 0.0]) + offset
 25
 26    def setup_scene(self):
 27        """Create the robots and cube for this scenario."""
 28        assets_root_path = get_assets_root_path()
 29        base_path = f"/World/{self.name}"
 30
 31        # Add Jetbot
 32        stage_utils.add_reference_to_stage(
 33            usd_path=assets_root_path + "/Isaac/Robots/NVIDIA/Jetbot/jetbot.usd",
 34            path=f"{base_path}/Jetbot",
 35        )
 36        jetbot_xform = XformPrim(f"{base_path}/Jetbot")
 37        jetbot_xform.reset_xform_op_properties()
 38        jetbot_xform.set_world_poses(positions=[self.offset.tolist()])
 39
 40        # Add cube in front of Jetbot
 41        cube_pos = self.offset + np.array([0.15, 0.0, 0.025])
 42        visual_material = PreviewSurfaceMaterial(f"{base_path}/Materials/red")
 43        visual_material.set_input_values("diffuseColor", [1.0, 0.0, 0.0])
 44        cube_shape = Cube(
 45            paths=f"{base_path}/Cube",
 46            positions=np.array([cube_pos]),
 47            sizes=[1.0],
 48            scales=np.array([[0.05, 0.05, 0.05]]),
 49            reset_xform_op_properties=True,
 50        )
 51        GeomPrim(paths=cube_shape.paths, apply_collision_apis=True)
 52        RigidPrim(paths=cube_shape.paths)
 53        cube_shape.apply_visual_materials(visual_material)
 54
 55        # Add Franka
 56        franka_pos = self.offset + np.array([0.8, -0.3, 0.0])
 57        self.franka = FrankaExperimental(robot_path=f"{base_path}/Franka", create_robot=True)
 58        franka_xform = XformPrim(f"{base_path}/Franka")
 59        franka_xform.reset_xform_op_properties()
 60        franka_xform.set_world_poses(positions=[franka_pos.tolist()])
 61
 62    def initialize(self):
 63        """Initialize articulation handles after scene load."""
 64        base_path = f"/World/{self.name}"
 65        self.jetbot = Articulation(f"{base_path}/Jetbot")
 66        self.cube = RigidPrim(f"{base_path}/Cube")
 67
 68    def reset(self):
 69        """Reset the scenario state."""
 70        self.state = 0
 71        self.step_counter = 0
 72        self.pick_phase = 0
 73        self.franka.reset_to_default_pose()
 74
 75    def step(self):
 76        """Execute one step of the scenario logic."""
 77        if self.state == 0:
 78            # Jetbot pushes cube
 79            cube_pos = self.cube.get_world_poses()[0].numpy()[0]
 80            if np.linalg.norm(cube_pos[:2] - self.cube_goal[:2]) > 0.05:
 81                self.jetbot.set_dof_velocity_targets([[10.0, 10.0]])
 82            else:
 83                self.jetbot.set_dof_velocity_targets([[0.0, 0.0]])
 84                self.state = 1
 85                self.step_counter = 0
 86
 87        elif self.state == 1:
 88            # Jetbot backs up
 89            self.jetbot.set_dof_velocity_targets([[-8.0, -8.0]])
 90            self.step_counter += 1
 91            if self.step_counter > 100:
 92                self.jetbot.set_dof_velocity_targets([[0.0, 0.0]])
 93                self.state = 2
 94                self.step_counter = 0
 95                self.franka.open_gripper()
 96
 97        elif self.state == 2:
 98            # Franka pick-and-place
 99            self._franka_pick_place()
100
101    def _franka_pick_place(self):
102        """Execute Franka pick-and-place state machine."""
103        cube_pos = self.cube.get_world_poses()[0].numpy()[0]
104        down_orient = self.franka.get_downward_orientation()
105        self.step_counter += 1
106
107        if self.pick_phase == 0:
108            self.franka.set_end_effector_pose(np.array([[cube_pos[0], cube_pos[1], cube_pos[2] + 0.2]]), down_orient)
109            if self.step_counter > 120:
110                self.pick_phase = 1
111                self.step_counter = 0
112        elif self.pick_phase == 1:
113            self.franka.set_end_effector_pose(np.array([[cube_pos[0], cube_pos[1], cube_pos[2] + 0.1]]), down_orient)
114            if self.step_counter > 100:
115                self.franka.close_gripper()
116                self.pick_phase = 2
117                self.step_counter = 0
118        elif self.pick_phase == 2:
119            self.franka.close_gripper()
120            if self.step_counter > 50:
121                self.pick_phase = 3
122                self.step_counter = 0
123        elif self.pick_phase == 3:
124            self.franka.set_end_effector_pose(np.array([[cube_pos[0], cube_pos[1], cube_pos[2] + 0.25]]), down_orient)
125            if self.step_counter > 100:
126                self.pick_phase = 4
127                self.step_counter = 0
128        elif self.pick_phase == 4:
129            target = self.offset + np.array([0.3, 0.3, 0.15])
130            self.franka.set_end_effector_pose(np.array([target]), down_orient)
131            if self.step_counter > 150:
132                self.franka.open_gripper()
133                self.step_counter = 0
134                self.pick_phase = 5
135        elif self.pick_phase == 5:
136            # Lift the arm from target position (don't use cube_pos - cube was dropped)
137            target = self.offset + np.array([0.3, 0.3, 0.4])  # Lift above drop location
138            self.franka.set_end_effector_pose(np.array([target]), down_orient)
139            if self.step_counter > 150:
140                self.step_counter = 0
141                self.state = 5  # Done
142
143
144class HelloWorld(BaseSample):
145    def __init__(self) -> None:
146        super().__init__()
147        self._physics_callback_id = None
148        self._scenario = None
149
150    def setup_scene(self):
151        # Add ground plane
152        stage_utils.add_reference_to_stage(
153            usd_path=get_assets_root_path() + "/Isaac/Environments/Grid/default_environment.usd",
154            path="/World/ground",
155        )
156        # Create a single scenario
157        self._scenario = RobotScenario(name="scenario_0", offset=np.array([0.0, 0.0, 0.0]))
158        self._scenario.setup_scene()
159
160    async def setup_post_load(self):
161        self._scenario.initialize()
162
163        from isaacsim.core.simulation_manager.impl.isaac_events import IsaacEvents
164
165        self._physics_callback_id = SimulationManager.register_callback(
166            self.physics_step, IsaacEvents.POST_PHYSICS_STEP
167        )
168
169    def physics_step(self, dt, context):
170        self._scenario.step()
171
172    async def setup_post_reset(self):
173        self._scenario.reset()
174
175    def physics_cleanup(self):
176        if self._physics_callback_id is not None:
177            SimulationManager.deregister_callback(self._physics_callback_id)
178            self._physics_callback_id = None
../_images/core_api_tutorials_6_1.webp

Scaling to Multiple Scenarios#

 1import isaacsim.core.experimental.utils.stage as stage_utils
 2import numpy as np
 3from isaacsim.core.experimental.materials import PreviewSurfaceMaterial
 4from isaacsim.core.experimental.objects import Cube
 5from isaacsim.core.experimental.prims import Articulation, GeomPrim, RigidPrim, XformPrim
 6from isaacsim.core.simulation_manager import SimulationManager
 7from isaacsim.examples.base.base_sample_experimental import BaseSample
 8from isaacsim.robot.manipulators.examples.franka import FrankaExperimental
 9from isaacsim.storage.native import get_assets_root_path
10
11# RobotScenario class definition (same as above)
12# ... (include the full RobotScenario class from the previous example)
13
14
15class HelloWorld(BaseSample):
16    def __init__(self) -> None:
17        super().__init__()
18        self._physics_callback_id = None
19        self._scenarios = []
20        self._num_scenarios = 2  # Number of parallel scenarios
21
22    def setup_scene(self):
23        # Add ground plane
24        stage_utils.add_reference_to_stage(
25            usd_path=get_assets_root_path() + "/Isaac/Environments/Grid/default_environment.usd",
26            path="/World/ground",
27        )
28
29        # Create multiple scenarios with Y-axis offsets
30        for i in range(self._num_scenarios):
31            offset = np.array([0.0, (i - 1) * 2.0, 0.0])  # Spread along Y-axis
32            scenario = RobotScenario(name=f"scenario_{i}", offset=offset)
33            scenario.setup_scene()
34            self._scenarios.append(scenario)
35
36    async def setup_post_load(self):
37        # Initialize all scenarios
38        for scenario in self._scenarios:
39            scenario.initialize()
40
41        from isaacsim.core.simulation_manager.impl.isaac_events import IsaacEvents
42
43        self._physics_callback_id = SimulationManager.register_callback(
44            self.physics_step, IsaacEvents.POST_PHYSICS_STEP
45        )
46
47    def physics_step(self, dt, context):
48        # Step all scenarios
49        for scenario in self._scenarios:
50            scenario.step()
51
52    async def setup_post_reset(self):
53        # Reset all scenarios
54        for scenario in self._scenarios:
55            scenario.reset()
56
57    def physics_cleanup(self):
58        if self._physics_callback_id is not None:
59            SimulationManager.deregister_callback(self._physics_callback_id)
60            self._physics_callback_id = None
61        self._scenarios = []
../_images/core_api_tutorials_6_2.webp

Adding Randomization#

To make simulations more interesting, you can add randomization to the scenario parameters. Modify the RobotScenario class to accept randomization options:

 1class RobotScenario:
 2    """Encapsulates a Jetbot + Franka + Cube scenario with randomization."""
 3
 4    def __init__(self, name: str, offset: np.ndarray = np.array([0.0, 0.0, 0.0]), randomize: bool = False):
 5        self.name = name
 6        self.offset = offset
 7        self.randomize = randomize
 8        self.state = 0
 9        self.step_counter = 0
10        self.pick_phase = 0
11
12        # Randomize cube goal position if enabled
13        if randomize:
14            random_x = np.random.uniform(1.1, 1.4)
15            self.cube_goal = np.array([random_x, 0.0, 0.0]) + offset
16        else:
17            self.cube_goal = np.array([1.2, 0.0, 0.0]) + offset
18
19        # ... rest of the class remains the same

Then create scenarios with randomization enabled:

1# Create multiple scenarios with randomization
2for i in range(self._num_scenarios):
3    offset = np.array([0.0, (i - 1) * 2.0, 0.0])
4    scenario = RobotScenario(name=f"scenario_{i}", offset=offset, randomize=True)  # Enable randomization
5    scenario.setup_scene()
6    self._scenarios.append(scenario)

Best Practices for Scaling#

When creating large-scale multi-robot simulations:

  1. Use unique paths: Each scenario should use unique USD prim paths to avoid conflicts. The RobotScenario class uses the scenario name to create unique paths like /World/scenario_0/Jetbot.

  2. Manage state independently: Each scenario instance maintains its own state variables, allowing scenarios to progress independently.

  3. Clean up properly: The physics_cleanup method ensures callbacks are deregistered and scenario lists are cleared when the simulation is stopped.

  4. Consider performance: With many scenarios, consider reducing physics step frequency or using GPU-accelerated simulation for better performance.

Summary#

This tutorial covered the following topics:

  1. Organizing robot scenarios into reusable Python classes

  2. Using the offset parameter to position multiple scenarios in the world

  3. Scaling to multiple parallel scenarios with a simple loop

  4. Adding randomization to scenario parameters

  5. Best practices for managing multiple robot instances