Configuration File Guide#

This guide describes how to configure the Isaac Sim Replicator Agent (IRA) for simulation and synthetic data generation. The configuration controls the environment, sensor generation and placement, character/robot agents, behaviors, and data generation.

Concepts & Workflow#

Before diving into detailed configuration, it is helpful to understand the general workflow and key concepts of an IRA simulation.

Workflow Overview#

  1. Environment Setup: Define the static 3D environment where the simulation takes place.

  2. Agent & Sensor Definition: Configure characters, robots, and cameras (sensors) to populate the environment.

  3. Behavior Configuration: Assign routines (weighted random actions like walking, idling) and triggers (reactive behaviors like when a collision occurs) to actors.

  4. Data Generation: Configure the Replicator writers to generate ground-truth data (RGB, segmentation, etc.).

Key Concepts#

  • Environment: The static 3D world (USD stage) loaded for the simulation. It also defines the NavMesh for navigation.

  • Agents (Actors): Dynamic entities in the scene, which can be Characters (humans) or Robots.

  • Behaviors: Atomic actions an actor can perform, such as wander, patrol, or idle.

  • Routines: A collection of behaviors assigned to an actor group. Actors randomly select behaviors from this pool based on assigned weights.

  • Triggers: Conditional logic that interrupts normal routines. When a condition is met (e.g., a specific time or event), the trigger executes its defined list of behaviors in sequence. Once the trigger sequence is complete, the agent resumes its standard routine until another trigger activates.

  • Sensors: Cameras placed in the scene to observe the simulation.

  • Replicator: The system responsible for rendering frames and writing annotated data (ground truth) to disk or cloud storage.

Top-level Structure#

Configs are YAML files with a single root key isaacsim.replicator.agent:

isaacsim.replicator.agent:
  version: 1.0.0
  environment: { ... }            # required
  seed: 123456789                 # optional; 32-bit (0..4294967295); autogenerated if omitted
  simulation_duration: 60.0       # optional; defaults to 60.0
  character: { ... }              # optional
  robot: { ... }                  # optional
  sensor: { ... }                 # optional
  replicator: { ... }             # optional

Root Parameters#

  • version (required): Semantic version of the configuration schema (e.g., “1.0.0”).

  • environment (required): Defines the simulation world.

  • seed (optional): A 32-bit unsigned integer (0..4,294,967,295). - Used to initialize random number generators for deterministic simulations (e.g., character spawn locations, routine variations). - If omitted, a seed is generated based on the current system time.

  • simulation_duration (optional): The total run time of the simulation in seconds. - The simulation runs at a fixed time step corresponding to 30 FPS. - Defaults to 60.0 seconds.

  • character (optional): Configures human agents (appearance, behaviors like wander/patrol, and triggers).

  • robot (optional): Configures robot agents (config path, behaviors/commands, data collection).

  • sensor (optional): Configures static cameras using placement strategies (e.g., aim at targets, coverage).

  • replicator (optional): Configures data writers (e.g., output directory, annotators like RGB/segmentation).

Quick Start (Minimal)#

isaacsim.replicator.agent:
  version: 1.0.0
  environment:
    base_stage_asset_path: "Isaac/Environments/Simple_Warehouse/full_warehouse.usd"

Sections#

Environment#

Defines the static 3D environment and additional assets to load.

  • base_stage_asset_path (required): Path or URL to the main USD stage. - Supports http(s):// (including S3 presigned URLs), Windows/UNC paths, and local filesystem paths. - Also supports paths relative to the Isaac Sim Assets root.

  • prop_asset_paths (optional): A list of additional USD assets to load as sublayers into the stage. This is useful for adding props or lighting to a base environment without modifying it. Supports paths relative to the Isaac Sim Assets root.

Example:

environment:
  base_stage_asset_path: "Isaac/Environments/Simple_Warehouse/full_warehouse.usd"
  prop_asset_paths:
    - "Isaac/Props/Conveyors/ConveyorBelt_A08.usd"

Character#

Defines groups of human characters, their appearance, and their behavior.

  • root_prim_path (optional): Root path for spawning characters (default: /World/Characters).

  • groups: Dictionary of character groups.

Character Group Parameters#

  • num (required): Number of characters to spawn.

  • asset_path (optional): USD path to character assets. Supports paths relative to the Isaac Sim Assets root. Default: Isaac/People/Characters/.

  • spawn_areas (optional): List of NavMesh area names where characters can spawn. If empty, spawns anywhere on the NavMesh.

  • semantic_labels (optional): List of [type, data] pairs for semantic segmentation. Default: [["class", "character"]].

  • motion_library_path (optional): Path to a custom motion library file. Supports paths relative to the Isaac Sim Assets root. Default: Isaac/People/MotionLibrary/HumanMotionLibrary.usd.

  • routines (optional): List of behaviors the characters will execute. Default: [{ wander: {} }].

  • triggers (optional): List of event-based triggers that interrupt routines.

Behaviors#

Behaviors are defined in the routines list. Common fields:

  • weight (default 1): Probability weight for selecting this behavior.

  • repeat (default 1): How many times to repeat this behavior before choosing a new one.

Supported Behaviors:

  1. wander: Randomly walk and idle.

    • walk: - speed_range: [min, max] m/s (default [1.0, 1.0]). - distance_range: [min, max] distance to travel per walk leg (default [5.0, 15.0]). - navigation_areas: List of allowed NavMesh area tags.

    • idle: Array of idle options. - animation: Name of the animation (must exist in motion lib). - time_range: [min, max] duration in seconds (default [2.0, 5.0]). - weight: Selection probability.

  2. patrol: Follow a specific path.

    • speed_range: [min, max] m/s (default [1.0, 1.0]).

    • One of: - path_points: List of 3D points [[x,y,z], [x,y,z], ...]. - target_prims: List of prim paths to visit.

      Note

      Both path_points and target_prims must be on the NavMesh and reachable by the actors.

  3. custom_behavior: Execute a Behavior Tree from a file.

    • behavior_name: Identifier.

    • behavior_subtree: Path to the behavior JSON file.

    • params: Dictionary of arbitrary parameters passed to the tree.

Triggers#

Triggers define events that interrupt normal routines to execute a specific reaction behavior.

  • priority (default 1): Higher priority triggers override lower ones.

  • behavior: List of behaviors to execute when triggered.

Trigger Types:

  • event_trigger: Fires on a named event (e.g., custom simulation events).

  • time_trigger: Fires after a specific duration (in seconds). Not available in Isaac Sim 6.0 EA yet. (Coming in 6.0 GA)

    Tip

    Dispatching Events from Python

    The method for dispatching events differs between Characters and Robots.

    For Characters: Use the AgentManager to update the behavior tree blackboard directly.

    from isaacsim.replicator.agent.core.agent_manager import AgentManager

# Arguments:
# - prim: The prim path of the specific character instance
# - key: The internal blackboard key for events (always "event_trigger_key")
# - value: The name of the event to trigger (matches the YAML 'event' field)
AgentManager._set_tree_blackboard(
    prim="/World/Characters/warehouse_workers/Character",
    key="event_trigger_key",
    value="test_event"
)

    For Robots: Use the carb.eventdispatcher system.

    import carb

# Arguments:
# - event_name: The name of the event to trigger (matches the YAML 'event' field, e.g., "my_test_event")
# - payload: A dictionary containing:
#     - "prim_path": The prim path of the specific robot instance
carb.eventdispatcher.get_eventdispatcher().dispatch_event(
    "my_test_event",
    payload={"prim_path": "/World/Robots/iw_hub/iw_hub"}
)

Example:

character:
  root_prim_path: "/World/Characters"
  groups:
    warehouse_workers:
      asset_path: "Isaac/People/Characters/"
      num: 10
      spawn_areas: ["warehouse_floor"]
      routines:
        - wander:
            walk:
              speed_range: [0.8, 1.5]
              distance_range: [5.0, 10.0]
            idle:
              - animation: look_around
                time_range: [2.0, 5.0]
      triggers:
        - time_trigger:
            time: 30.0
            priority: 10
            behavior:
              - patrol: # Move to break room
                  speed_range: [1.0, 1.2]
                  target_prims: ["/World/BreakRoom"]
        - event_trigger:
            event: test_event
            priority: 10
            behavior:
              - patrol:
                  speed_range: [5.0, 5.5]
                  path_points:
                    - [0, 0, 0]
                    - [0, -5, 0]

Robot#

Defines robot agents. Robots can be controlled via behaviors OR an external command file.

  • root_prim_path (optional): Root path for robots (default: /World/Robots).

  • groups: Dictionary of robot groups.

Robot Group Parameters#

  • num (required): Number of robots.

  • config_file_path (required): Path to the IAR (Isaac Agent) configuration file for this robot type. Supports absolute paths or paths relative to the IAR built-in robot config sample folder (located in isaacsim/anim/robot/agent/configs within the isaacsim.anim.robot extension).

  • command_file_path (optional): Path to a global command file. Mutually exclusive with routines.

  • spawn_areas (optional): NavMesh areas for spawning.

  • agent_radius (optional): Radius in meters (default 0.5) used for NavMesh queries.

  • write_data (optional): If true, enables data collection from the robot’s onboard cameras. Not available in Isaac Sim 6.0 EA yet. (Coming in 6.0 GA)

  • camera_prim_paths (optional): List of specific camera prims on the robot to use. If empty and write_data is true, all cameras on the robot are used. Not available in Isaac Sim 6.0 EA yet. (Coming in 6.0 GA)

  • semantic_labels (optional): Default [["class", "robot"]].

  • semantic_label_path (optional): Relative path under the robot prim to apply semantics.

  • routines / triggers: Similar to characters but with robot-specific behaviors.

Robot Behaviors#

  • wander: - move: { distance_range: [min, max] (default [10.0, 15.0]), navigation_areas: [tags] } - idle: { time_range: [min, max] (default [2.0, 5.0]) }

  • patrol: - path_points: List of 3D points.

  • stop: - time_range: [min, max] seconds to remain stopped (default [5.0, 5.0]).

Sensor#

Defines static cameras in the scene. Cameras are organized into named groups.

  • root_prim_path (optional): Absolute prim path where all camera groups will be created (default: /World/Cameras).

  • groups: A dictionary where keys are group names and values define the camera configuration.

Group Configuration#

Each group must specify num (number of cameras) and one placement strategy (aim_at_targets OR maximum_coverage). - For aim_at_targets, num must be >= 0. - For maximum_coverage, num can be >= -1. If -1, the number of cameras is automatically calculated based on the grid resolution and coverage ratio.

1. Placement Strategy: aim_at_targets

Places cameras to look at specific targets.

  • targets (optional): List of target prim paths (e.g., /World/Characters) or identifiers.

  • raycast_density (optional): Density of rays used to find valid camera positions. Higher values are more precise but slower.

  • yaw_range (optional): [min, max] degrees (0..360) for the camera’s rotation around the target.

  • occlusion_threshold (optional): Threshold for filtering occluded views (-1 to disable).

2. Placement Strategy: maximum_coverage

Places cameras to maximize visual coverage of the environment.

  • target_coverage_ratio (optional): Desired coverage ratio (0.0 to 1.0). Default 0.9.

  • grid_resolution (optional): Size of the grid cells (in meters) used for coverage calculation. Default 1.0.

Shared Parameters

These apply to all placement strategies:

  • height_range: [min, max] height in meters (Z-axis).

  • look_down_angle_range: [min, max] pitch angle in degrees (0 = horizontal, 90 = straight down).

  • focal_length_range: [min, max] focal length in millimeters.

  • distance_range: [min, max] distance from the camera to its target/interest point in meters.

Example:

sensor:
  root_prim_path: "/World/Cameras"
  groups:
    ceiling_cameras:
      num: 20
      aim_at_targets:
        targets: ["/World/Characters"]
        distance_range: [5, 10]
        height_range: [7, 10]
        focal_length_range: [10, 15]
        look_down_angle_range: [30, 45]
    coverage_cameras:
      num: 5
      maximum_coverage:
        target_coverage_ratio: 0.8
        height_range: [2, 5]

Replicator#

Controls the generation of synthetic data (images, annotations) using Omniverse Replicator.

  • writers (required): Dictionary of writer configurations.

  • hide_debug_visualization (optional, default true): Hides debug visualizations (NavMesh, skeletons, lights) during data capture.

Writer Configuration#

Supported writers: BasicWriter, IRABasicWriter, CosmosIRAWriter, CustomWriter.

Common Settings per Writer:

  • Timing: - start_frame / end_frame: Frame-based control (inclusive/exclusive). Defaults to start_frame: 30 if not specified. - start_time / end_time: Time-based control (seconds).

  • Sensors: - sensor_prim_list: Optional list of cameras to use. If omitted, uses all cameras defined in sensor.root_prim_path.

Output Settings:

  • output_dir: Local directory for output. Defaults to ~/IRA_output if not set.

  • s3_bucket, s3_region, s3_endpoint: For direct S3 upload.

Common Annotators (Parameters):

  • rgb: RGB Image.

  • bounding_box_2d_tight / loose: 2D Bounding boxes.

  • bounding_box_3d: 3D Bounding boxes.

  • semantic_segmentation: Pixel-wise semantic class IDs.

  • instance_segmentation: Pixel-wise instance IDs.

  • distance_to_camera: Depth map.

  • normals: Surface normals.

  • motion_vectors: Pixel motion.

  • colorize_*: e.g., colorize_semantic_segmentation (save as visible color map vs raw ID).

Specialized Writers#

  1. IRABasicWriter:

    The foundational writer for Agent simulations, derived from Replicator’s BasicWriter. It organizes output into separate folders per annotator and consolidates object/agent metadata into object_detection.json.

    • Key Features: - Folder Structure: Outputs each annotator’s data into separate folders for better readability. - Object Detection: Consolidates bounding box and skeleton data into a single file named object_detection.json. - Default Semantic Filter: class:character|robot;id:* (captures characters and robots).

    • Overwritten Annotators: The following standard annotators are replaced by specialized object_info_* versions and written to object_detection.json:

      • bounding_box_2d_tight

      • bounding_box_2d_loose

      • bounding_box_3d

      • skeleton_data (replaced by agent_info_skeleton_data)

    • Defaults: - rgb: Enabled. - camera_params: Enabled. - S3-related parameters: Disabled.

    • Special Parameters: - video_rendering_annotator_list: Generates .mp4 videos for specified annotators (e.g., ["rgb", "semantic_segmentation"]). - agent_info_skeleton_data: Exports 2D/3D skeleton joints for characters.

  2. CosmosIRAWriter:

    • Adds “Cosmos” specific post-processing.

    • shaded_seg: Shaded segmentation visualization.

    • canny_edge: Canny edge detection filter (with canny_threshold_low/high).

Note

In Isaac Sim 6.0 EA release, we support only the IRABasicWriter. Other writers will be supported in the Isaac Sim 6.0 GA release.

Example:

replicator:
  writers:
    IRABasicWriter:
      # Output Config
      output_dir: "/home/IRA_Output"
      s3_post_upload: false

      # Timing
      start_frame: 0
      end_frame: 300

      # Annotators
      rgb: true
      camera_params: true
      bounding_box_2d_tight: true
      semantic_segmentation: true
      agent_info_skeleton_data: true

      # Video
      video_rendering_annotator_list: ["rgb", "semantic_segmentation"]