Actor Simulation and Synthetic Data Generation#

Detecting and tracking people in diverse environments offers significant value across industries like retail, manufacturing, and logistics. It helps optimize layouts, improve safety, and enhance efficiency. However, collecting real-world data to train detection models is often costly and unscalable.

Synthetic data generation offers a flexible, scalable solution. The Isaacsim.Replicator.Agent (IRA) extension generates synthetic data for human characters and robots in 3D environments. It provides control over environments, cameras, and actor behaviors via configuration file, supporting sensors like RGB and stereo cameras. IRA aims to provide a GPU-accelerated solution for training computer vision models and testing software-in-the-loop systems. In IRA, human characters and robots are referred to as agents or actors.

IRA simplifies simulation customization with features like:

  • Simplified Setup: Included in Isaac Sim, it offers both GUI and scripting interfaces for interactive and headless workflows.

  • High-Fidelity Data: Leverages Omniverse’s SimReady assets, physics, and rendering to produce realistic imagery and accurate annotations essential for AI training.

  • Seamless Integration: As a Kit extension, it works natively with omni.anim.graph, omni.anim.navigation, and omni.replicator.core.

Before enabling this extension, read What Is Isaac Sim? to learn about Isaac Sim and follow Installation to install Isaac Sim.

characters and robots move around a warehouse 3D environment.

Note

Isaacsim.Replicator.Agent is in beta. For more information on pre-release and beta terms, refer to the Omniverse License Agreement.

Enable Isaacsim.Replicator.Agent#

  1. Follow the Omniverse Extension Manager guide to enable the isaacsim.replicator.agent.core and isaacsim.replicator.agent.ui extensions.

    • The extension fetches sample assets from Isaac Sim Assets during start. Refer to Isaac Sim Assets if you encounter issues for loading assets.

    • If loading the UI appears to be hanging, try starting Isaac Sim with the flag --/persistent/isaac/asset_root/timeout=1.0.

  2. The IRA UI panel is accessible by Tools > Action and Event Data Generation > Actor SDG and it opens on the right side of the screen.

Note

  • To have the extension auto-loaded on startup, check the autoload checkbox in the extension manager.

  • Because of extension dependencies, a restart of the Isaac Sim app might be required.

Getting Started#

To launch data generation with IRA, load a YAML configuration file. Or use the default configuration file that comes with the extension and follow the steps below. For a more detailed walkthrough, refer to the Walkthrough section.

  1. Enable the IRA extension and open the UI panel.

  2. Load the configuration file or use the default configuration file.

    • Some sample config files are provided in [Isaac Sim App Path]/extscache/isaacsim.replicator.agent.core-[current-version]/data/sample_configs/. When the extension starts, it will load the minimal config file.

  3. Modify the configuration file to your needs.

  4. Click the Set Up Simulation button from the top of the UI and it will start loading simulation assets, which can take a while.

  5. Click the Start Data Generation button from the top of the UI and the simulation and data generation will start. It can take a while to finish.

  6. The output data can be found from the Output Directory according to the Replicator panel.

Configuration File#

The configuration file is the central place to define your simulation. It controls everything from the environment and characters to the sensors and data output. The file uses the YAML format.
The configuration file is organized into these top-level sections:

  • environment: Defines the simulation environment and assets.

  • character: Configures human characters.

  • robot: Configures robots.

  • sensor: Configures RTX sensors.

  • replicator: Configures data generation and output.

For detailed configuration instructions, parameter lists, and examples, refer to the following document:

Walkthrough#

Tip

If you encounter unexpected errors, try launching Isaac Sim with the --reset-user flag to clear previous user settings.

./isaac-sim.sh --reset-user

Data Generation from UI#

The IRA UI provides control for all the attributes of the simulation and data generation configuration. The attributes provide you with fine-grained control over the simulation result. The following steps launch the data generation with simulation configurations:

  1. Enable the IRA extension to open the Actor SDG window.

    • A set of sample config files are provided in [Isaac Sim App Path]/extscache/isaacsim.replicator.agent.core-[current-version]/data/sample_configs/. When the extension starts, it will load the minimal config file.

    • To load a different configuration file, click the folder icon next to the config file path text field on the top panel or directly enter the path to the config file in the text field.

  2. Modify other simulation properties in the UI. All the fields in the configuration file can be directly edited through the UI. All the UI fields have their respective fields in the configuration file. Refer to the Configuration File to review what each field means.

    • When there are unsaved changes, Set Up Simulation and Start Data Generation buttons run according to the information displayed in the UI rather than from the configuration file on disk.

    • Assets (environment, characters, cameras) are not loaded until the Set Up Simulation button is clicked.

  3. Click the Set Up Simulation button to load the simulation assets and wait for loading to finish. Loading can take a while, depending on the complexity of the scene.

    Note

    • Before setting up the simulation, IRA requires a NavMesh in the stage to spawn actors and control them correctly. Learn how to create your NavMesh from Navigation Mesh.

    • You can also go to Window > Navigation > NavMesh and turn off Auto-Bake in the NavMesh settings. Turning it off can increase the performance.

  4. [Optional] Click the Save or Save As icon from the top UI panel, if you want the configuration for this simulation to be saved.

  5. After the assets have been loaded and modified, press the Start Data Generation button from the top UI panel to start the simulation and data generation.

    • We recommend clicking the “Play” button to test the simulation before the actual data generation starts.

    • The simulation automatically stops when enough data has been generated to cover simulation_duration. The simulation currently operates at a fixed rate of 30 frames per second.

  6. The output data can be found from the Output Directory according to the Replicator panel.

Data Generation from Script#

For large-scale data generation, it can be more efficient to launch it from script. IRA provides an automatic script (actor_sdg.py) to run offline data generation.

To run from script, open a terminal from where Isaac Sim is installed and run the following commands.

  • For Linux:

    ./python.sh tools/actor_sdg/actor_sdg.py -c [config file path]

  • For Windows:

    .\python.bat tools\actor_sdg\actor_sdg.py -c [config file path]

Note

  • [config file path] is the path to the IRA configuration file.

  • You must use the python.sh or python.bat bundled with Isaac Sim to run the script.

  • An example config file is also provided in the /tools/actor_sdg folder. For a sample Linux run, execute: ./python.sh tools/actor_sdg/actor_sdg.py -c tools/actor_sdg/sample_config.yaml

Actor Controllers#

omni.behavior.composer and isaacsim.anim.robot are the two controllers that are used to control the actors in the simulation.

  • omni.behavior.composer is a controller that is used to control the character’s behaviors.

  • isaacsim.anim.robot is a controller that is used to control the robot’s behaviors.

For more information about the controllers, please refer to the following documents:

Terminology#

Isaacsim.Replicator.Agent.Core

The core extension that manages the simulation state. It contains the essential API and modules for setting up the simulation and capturing the synthetic data. Its modules can be called independently.

Isaacsim.Replicator.Agent.UI

The UI extension for IRA. When this extension loads, the core extension is loaded automatically. This extension contains the UI components for easy interaction with the extension.

Configuration File

A .yaml file that contains configuration data that defines the key components of a simulation, including the randomization seed, duration of the simulation, number of the actors, and output format. To use the extension, you must load a configuration file or use the UI to generate a YAML file first.

Actor

Actors are controlled by the respective controllers (omni.behavior.composer and isaacsim.anim.robot) and perform actions in the simulation. The extension supports human characters and robots (Nova Carter, iw.hub) as actors. The terms “actor” and “agent” are used interchangeably in this documentation.

Seed

Randomization seed. Given the same seed, the extension can generate the same randomized result for camera and agent location and agent behaviors. With the same seed and the same sequence of operations, the same data is guaranteed to be generated.

Replicator (Omni.Replicator.Core)

The data capturing extension that our extension is based on. More information about the Replicator extension can be found in Replicator Official Documentation.