Configuration File Guide#

This guide describes how to configure the Isaac Sim Replicator Agent (IRA) for simulation and synthetic data generation. The configuration controls the environment, sensor generation and placement, character/robot agents, behaviors, and data generation.

Concepts & Workflow#

Before diving into detailed configuration, it is helpful to understand the general workflow and key concepts of an IRA simulation.

Workflow Overview#

  1. Environment Setup: Define the static 3D environment where the simulation takes place.

  2. Agent & Sensor Definition: Configure characters, robots, and cameras (sensors) to populate the environment.

  3. Behavior Configuration: Assign routines (weighted random actions like walking, idling) and triggers (reactive behaviors like when a collision occurs) to actors.

  4. Data Generation: Configure the Replicator writers to generate ground-truth data (RGB, segmentation, etc.).

Key Concepts#

  • Environment: The static 3D world (USD stage) loaded for the simulation. It also defines the NavMesh for navigation.

  • Agents (Actors): Dynamic entities in the scene, which can be Characters (humans) or Robots.

  • Behaviors: Atomic actions an actor can perform, such as wander, patrol, or idle.

  • Routines: A collection of behaviors assigned to an actor group. Actors randomly select behaviors from this pool based on assigned weights.

  • Triggers: Conditional logic that interrupts normal routines. When a condition is met (for example, a specific time or event), the trigger executes its defined list of behaviors in sequence. Once the trigger sequence is complete, the agent resumes its standard routine until another trigger activates.

  • Sensors: Cameras placed in the scene to observe the simulation.

  • Replicator: The system responsible for rendering frames and writing annotated data (ground truth) to disk or cloud storage.

Top-level Structure#

Configs are YAML files with a single root key isaacsim.replicator.agent:

isaacsim.replicator.agent:
  version: 1.0.0
  environment: { ... }            # required
  seed: 123456789                 # optional; 32-bit (0..4294967295); autogenerated if omitted
  simulation_duration: 60.0       # optional; defaults to 60.0
  character: { ... }              # optional
  robot: { ... }                  # optional
  sensor: { ... }                 # optional
  replicator: { ... }             # optional

Root Parameters#

  • version (required): Semantic version of the configuration schema (for example, “1.0.0”).

  • environment (required): Defines the simulation world.

  • seed (optional): A 32-bit unsigned integer (0..4,294,967,295). - Used to initialize random number generators for deterministic simulations (for example, character spawn locations, routine variations). - If omitted, a seed is generated based on the current system time.

  • simulation_duration (optional): The total run time of the simulation in seconds (must be >= 0). - The simulation runs at a fixed time step corresponding to 30 FPS. - Defaults to 60.0 seconds.

  • character (optional): Configures human agents (appearance, behaviors like wander/patrol, and triggers).

  • robot (optional): Configures robot agents (config path, behaviors/commands, data collection).

  • sensor (optional): Configures static cameras using placement strategies (for example, aim at targets, coverage).

  • replicator (optional): Configures data writers (for example, output directory, annotators like RGB/segmentation).

Quick Start (Minimal)#

isaacsim.replicator.agent:
  version: 1.0.0
  environment:
    base_stage_asset_path: "Isaac/Environments/Simple_Warehouse/full_warehouse.usd"

Sections#

Environment#

Defines the static 3D environment and additional assets to load.

  • base_stage_asset_path (required): Path or URL to the main USD stage. - Supports http(s):// (including S3 presigned URLs), Windows/UNC paths, and local filesystem paths. - Also supports paths relative to the Isaac Sim Assets root.

  • prop_asset_paths (optional): A list of additional USD assets to load as sublayers into the stage. This is useful for adding props or lighting to a base environment without modifying it. Supports paths relative to the Isaac Sim Assets root.

Example:

environment:
  base_stage_asset_path: "Isaac/Environments/Simple_Warehouse/full_warehouse.usd"
  prop_asset_paths:
    - "Isaac/Props/Conveyors/ConveyorBelt_A08.usd"

Character#

Defines groups of human characters, their appearance, and their behavior.

  • root_prim_path (optional): Root path for spawning characters (default: /World/Characters).

  • groups: Dictionary of character groups.

Character Group Parameters#

  • num (required): Number of characters to spawn (>= 0).

  • asset_path (optional): USD path to character assets. Supports paths relative to the Isaac Sim Assets root. Default: Isaac/People/Characters/.

  • spawn_areas (optional): List of NavMesh area names where characters can spawn. If empty, spawns anywhere on the NavMesh.

  • semantic_labels (optional): List of [type, data] pairs for semantic segmentation. Default: [["class", "character"]].

  • motion_library_path (optional): Path to a custom motion library file. Supports paths relative to the Isaac Sim Assets root. Default: Isaac/People/MotionLibrary/HumanMotionLibrary.usd.

  • routines (optional): List of behaviors the characters will execute. Default: [{ wander: {} }].

  • triggers (optional): List of event-based triggers that interrupt routines.

Behaviors#

Behaviors are defined in the routines list. Common fields:

  • weight (default 1): Probability weight for selecting this behavior.

  • repeat (default 1): How many times to repeat this behavior before choosing a new one.

Supported Behaviors:

  1. wander: Randomly walk and idle.

    • walk: - speed_range: [min, max] m/s (default [1.0, 1.0]). - distance_range: [min, max] distance to travel per walk leg (default [5.0, 15.0]). - navigation_areas: List of allowed NavMesh area tags.

    • idle: Array of idle options. - animation: Name of the animation (must exist in motion lib). - time_range: [min, max] duration in seconds (default [2.0, 5.0]). - weight: Selection probability.

  2. patrol: Follow a specific path.

    • speed_range: [min, max] m/s (default [1.0, 1.0]).

    • One of: - path_points: List of 3D points [[x,y,z], [x,y,z], ...]. - target_prims: List of prim paths to visit.

      Note

      Both path_points and target_prims must be on the NavMesh and reachable by the actors.

  3. stop: Stop and idle in place for a random duration.

    • time_range: [min, max] duration in seconds (default [5.0, 5.0]).

Triggers#

Triggers define events that interrupt normal routines to execute a list of reaction behavior.

Each actor can specify a list of triggers to listen to.

  • priority (default 1): Higher priority triggers override lower ones.

  • behavior: List of behaviors to execute in sequence when triggered.

Tip

Authoring a specific sequence of actions

Routines select behaviors randomly based on weights, so they are not suited for deterministic sequences. If you need actors to perform actions in a specific order (for example, walk to point A, idle for 5 seconds, then walk to point B), use a trigger instead. A trigger’s behavior list is always executed in order, making it the right tool for scripted sequences. Use a time_trigger with time: 0 to start the sequence immediately when the simulation begins.

Trigger Types:

  • event_trigger: Fires on a named carb event.

  • time_trigger: Fires after a specific duration during play (in seconds).

    Tip

    Dispatching Events from Python

    # Send out a custom event named 'my_test_event'
import carb
carb.eventdispatcher.get_eventdispatcher().dispatch_event(event_name="my_test_event")

# Actors spawned by trigger setting will react to `my_test_event`.
# triggers:
#   - event_trigger:
#     event: my_test_event
#     priority: 10
#     behavior: [...]

Example:

character:
  root_prim_path: "/World/Characters"
  groups:
    warehouse_workers:
      asset_path: "Isaac/People/Characters/"
      num: 10
      spawn_areas: ["warehouse_floor"]
      routines:
        - wander:
            walk:
              speed_range: [0.8, 1.5]
              distance_range: [5.0, 10.0]
            idle:
              - animation: look_around
                time_range: [2.0, 5.0]
      triggers:
        - time_trigger:
            time: 30.0
            priority: 10
            behavior:
              - patrol: # Move to break room
                  speed_range: [1.0, 1.2]
                  target_prims: ["/World/BreakRoom"]
        - event_trigger:
            event: test_event
            priority: 10
            behavior:
              - patrol:
                  speed_range: [5.0, 5.5]
                  path_points:
                    - [0, 0, 0]
                    - [0, -5, 0]

Robot#

Defines robot agents.

  • root_prim_path (optional): Root path for robots (default: /World/Robots).

  • groups: Dictionary of robot groups.

Robot Group Parameters#

  • num (required): Number of robots (>= 1).

  • config_file_path (required): Path to the robot agent YAML configuration file for this robot type. Supports absolute paths or paths relative to the built-in sample config folder (data/sample_configs/ within the isaacsim.anim.robot.core extension).

  • spawn_areas (optional): NavMesh areas for spawning.

  • agent_radius (optional): Radius in meters used for NavMesh queries. Must be > 0 when set. If omitted, defaults to 0.3 at runtime.

  • write_data (optional): If true, enables data collection from the robot’s onboard cameras.

  • camera_prim_paths (optional): List of specific camera prims on the robot to use. If empty and write_data is true, all cameras on the robot are used. Requires write_data to be true. Not available in Isaac Sim 6.0 EA yet. (Coming in 6.0 GA)

  • semantic_labels (optional): Default [["class", "robot"]].

  • semantic_label_path (optional): Relative path under the robot prim to apply semantics.

  • routines (optional): List of robot behaviors. Default: [{ wander: {} }].

  • triggers (optional): List of triggers that interrupt routines. Robots support event_trigger and time_trigger.

Note

write_data has a behavior regression (comparing to Isaac Sim 6.0 EA) and enabling it may lead to program hang. It is recommended to disable it until next release.

Robot Behaviors#

  • wander: - move: { distance_range: [min, max] (default [10.0, 15.0]), navigation_areas: list of allowed NavMesh area tags (default []) } - idle: { time_range: [min, max] (default [2.0, 5.0]) }

  • patrol (exactly one must be provided): - path_points: List of 3D points [[x,y,z], [x,y,z], ...]. - target_prims: List of prim paths to visit.

    Note

    Both path_points and target_prims must be on the NavMesh and reachable by the robots.

  • halt: - time_range: [min, max] seconds to remain halted (default [5.0, 5.0]).

Sensor#

Defines static cameras in the scene. Cameras are organized into named groups.

  • root_prim_path (optional): Absolute prim path where all camera groups will be created (default: /World/Cameras).

  • groups: A dictionary where keys are group names and values define the camera configuration.

Group Configuration#

Each group must specify num (number of cameras) and one placement strategy (aim_at_targets OR maximum_coverage). - For aim_at_targets, num must be >= 0. - For maximum_coverage, num can be >= -1. If -1, the number of cameras is automatically calculated based on the grid resolution and coverage ratio.

1. Placement Strategy: aim_at_targets

Places cameras to look at specific targets.

  • targets (optional): List of target prim paths (for example, /World/Characters) or identifiers.

  • raycast_density (optional): Density of rays used to find valid camera positions. Higher values are more precise but slower.

  • yaw_range (optional): [min, max] degrees (0..360) for the camera’s rotation around the target.

  • occlusion_threshold (optional): Threshold for filtering occluded views (-1 to disable).

2. Placement Strategy: maximum_coverage

Places cameras to maximize visual coverage of the environment.

  • target_coverage_ratio (optional): Desired coverage ratio (0.0 to 1.0). Default 0.9.

  • grid_resolution (optional): Size of the grid cells (in meters) used for coverage calculation. Default 1.0.

Shared Parameters

These apply to all placement strategies:

  • height_range: [min, max] height in meters (Z-axis).

  • look_down_angle_range: [min, max] pitch angle in degrees (0 = horizontal, 90 = straight down).

  • focal_length_range: [min, max] focal length in millimeters.

  • distance_range: [min, max] distance from the camera to its target/interest point in meters.

Example:

sensor:
  root_prim_path: "/World/Cameras"
  groups:
    ceiling_cameras:
      num: 20
      aim_at_targets:
        targets: ["/World/Characters"]
        distance_range: [5, 10]
        height_range: [7, 10]
        focal_length_range: [10, 15]
        look_down_angle_range: [30, 45]
    coverage_cameras:
      num: 5
      maximum_coverage:
        target_coverage_ratio: 0.8
        height_range: [2, 5]

Replicator#

Controls the generation of synthetic data (images, annotations) using Omniverse Replicator.

  • writers (required): Dictionary of writer configurations.

  • hide_debug_visualization (optional, default true): Hides debug visualizations (NavMesh, skeletons, lights) during data capture.

Writer Configuration#

Supported writers: BasicWriter, IRABasicWriter, CosmosIRAWriter, RTSPWriter, CustomWriter.

Common Settings per Writer:

  • Timing: - start_frame / end_frame: Frame-based control (inclusive/exclusive). Defaults to start_frame: 30 if not specified. - start_time / end_time: Time-based control (seconds).

  • Sensors: - sensor_prim_list: Optional list of cameras to use. If omitted, uses all cameras defined in sensor.root_prim_path.

Output Settings:

  • output_dir: Local directory for output. Defaults to ~/IRA_output if not set.

  • s3_bucket, s3_region, s3_endpoint: For direct S3 upload.

Common Annotators (Parameters):

  • rgb: RGB Image.

  • bounding_box_2d_tight / loose: 2D Bounding boxes.

  • bounding_box_3d: 3D Bounding boxes.

  • semantic_segmentation: Pixel-wise semantic class IDs.

  • instance_segmentation: Pixel-wise instance IDs.

  • distance_to_camera: Depth map.

  • normals: Surface normals.

  • motion_vectors: Pixel motion.

  • colorize_*: for example, colorize_semantic_segmentation (save as visible color map vs raw ID).

Specialized Writers#

  1. IRABasicWriter:

    The foundational writer for Agent simulations, derived from Replicator’s BasicWriter. It organizes output into separate folders per annotator and consolidates object/agent metadata into object_detection.json.

    • Key Features:

      • Folder Structure: Outputs each annotator’s data into separate folders for better readability.

      • Object Detection: Consolidates bounding box and skeleton data into a single file named object_detection.json.

      • Default Semantic Filter: class:character|robot;id:* (captures characters and robots).

      • Action data output: When object detection is enabled (object_info or agent_info annotators are on), the action data for each IRA actor will be included as well.

    • Overwritten Annotators: The following standard annotators are replaced by specialized object_info_* versions and written to object_detection.json:

      • bounding_box_2d_tight

      • bounding_box_2d_loose

      • bounding_box_3d

      • skeleton_data (replaced by agent_info_skeleton_data)

    • Defaults:

      • rgb: Enabled.

      • camera_params: Enabled.

      • S3-related parameters: Disabled.

    • Special Parameters:

      • video_rendering_annotator_list: Generates .mp4 videos for specified annotators (for example, ["rgb", "semantic_segmentation"]).

      • agent_info_skeleton_data: Exports 2D/3D skeleton joints for characters.

    • Action data:

      Action data is for describing the current behavior of each actor. It is in object_detection.json under the metro_agent_data section. Data includes:

      • prim path: The prim path of the actor schema is applied.

      • agent_type: The actor schema type.

      • agent_name: The name of the actor.

      • agent_group: The group name of the actor.

      • world_position: Actor position (vec3) in world space.

      • world_rotation: Actor rotation (quaternion) in world space.

      • world_moving_direction: Actor moving direction (vec3) in world space. null means actor is not moving.

      • world_facing_direction: Actor facing direction (vec3) in world space.

      • speed: The actor moving speed.

      • current_task_name: Actor’s current action name (not behavior name).

      • asset_url: The asset URL of the actor.

  2. CosmosIRAWriter:

    • Adds “Cosmos” specific post-processing.

    • shaded_seg: Shaded segmentation visualization.

    • canny_edge: Canny edge detection filter (with canny_threshold_low/high).

  3. RTSPWriter:

    Streams selected annotators live over RTSP instead of saving frames to disk. It spins up an ffmpeg process per camera and annotator and pushes raw frame buffers into that stream.

    • Key Features:

      • Live RTSP Streaming: Publishes each camera and annotator to rtsp://<host>:<port>/<topic>_<camera>_<annotator>.

      • Per-annotator Streams: Each enabled annotator gets its own RTSP endpoint.

      • Hardware/Software Encoding: NVENC for supported 8-bit RGBA annotators with a software fallback for others.

      • Automatic GPU Distribution: Multiple GPUs are automatically distributed across available render products.

    RTSP setup (FFmpeg)

    Before streaming from Isaac Sim using RTSPWriter, install FFmpeg.

    Run the following command on Linux:

    Install FFmpeg on Linux#
    sudo apt update && sudo apt install -y ffmpeg

    Run the following command on Windows 10/11:

    Install FFmpeg on Windows 10/11#
    winget install ffmpeg
    RTSP parameters and annotators

    Each of these parameters is represented as a boolean toggle in the UI. Enabling any of them will include that data type in the RTSP output stream.

    • rtsp_stream_url: Base RTSP server URL.

    • rtsp_rgb: Toggle RGB stream (LdrColor).

    • rtsp_semantic_segmentation: Toggle semantic segmentation stream.

    • rtsp_instance_id_segmentation: Toggle instance ID segmentation stream.

    • rtsp_instance_segmentation: Toggle instance segmentation stream.

    • rtsp_normals: Toggle normals stream.

    • rtsp_distance_to_image_plane: Toggle distance to image plane stream.

    • rtsp_distance_to_camera: Toggle distance to camera stream.

    • device: NVENC GPU index.

    Supported RTSP annotators:

    • rgb (LdrColor)

    • semantic_segmentation

    • instance_id_segmentation

    • instance_segmentation

    • normals

    • distance_to_camera

    • distance_to_image_plane

    RTSP stream URL format

    Each stream URL follows this format:

    [rtsp_stream_url]/RTSPWriter[camera_prim_path_with_underscores]_[annotator_name]

    Where:

    • rtsp_stream_url: Base RTSP server URL (for example, rtsp://localhost:8554/RTSPWriter).

    • camera_prim_path_with_underscores: The camera prim path with forward slashes replaced by underscores.

    • annotator_name: The original annotator name (for example, rgb or distance_to_camera).

    Examples (rtsp_stream_url = rtsp://localhost:8554/RTSPWriter):

    • RGB stream for /World/Cameras/Camera_01: rtsp://localhost:8554/RTSPWriter_World_Cameras_Camera_01_rgb

    • Distance-to-camera stream for /World/Cameras/Camera: rtsp://localhost:8554/RTSPWriter_World_Cameras_Camera_distance_to_camera

    RTSP defaults

    • rtsp_rgb: Enabled.

    • All other rtsp_* annotators: Disabled.

    • rtsp_stream_url: rtsp://localhost:8554/RTSPWriter.

    • device: 0 (NVENC GPU index).

    RTSP runtime notes

    Warning messages are posted to the console to show the match between annotator names and RTSP stream URLs. Initializing RTSP streaming may take a while. During RTSP stream initialization, the first few frames may exhibit visual artifacts or corruption. This is expected behavior and resolves after the stream is fully established.

Note

In Isaac Sim 6.0 EA release, we support only the IRABasicWriter. Other writers will be supported in the Isaac Sim 6.0 GA release.

Example:

replicator:
  writers:
    IRABasicWriter:
      # Output Config
      output_dir: "/home/IRA_Output"
      s3_post_upload: false

      # Timing
      start_frame: 0
      end_frame: 300

      # Annotators
      rgb: true
      camera_params: true
      bounding_box_2d_tight: true
      semantic_segmentation: true
      agent_info_skeleton_data: true

      # Video
      video_rendering_annotator_list: ["rgb", "semantic_segmentation"]