Sensor Rig

MobilityGenSensorRig is a Python Module container that groups all sensors on a robot so that cascade operations — update_state(), enable_rgb_rendering(), state_dict_rgb() — propagate to all sensors automatically without manual loops.

Each sensor is attached as a named attribute on the rig (e.g. rig.front_hawk_left). The sensor name comes from the robot YAML and becomes the key in output data (e.g. "sensor_rig.front_hawk_left.rgb_image").

Currently supported: type: camera — full support via MobilityGenCamera. Logged and skipped: type: lidar, imu, radar.

Sensor attachment

sensor_prim_path in the YAML is relative to the robot root prim. At build time it is joined with robot_root_path to form an absolute stage path. For cameras, rep.create.render_product(abs_path, resolution) attaches Replicator to the existing UsdGeom.Camera prim. If the prim is missing, the sensor is skipped with a carb.log_error.

Intrinsics (cameras)

USD is the source of truth. focal_length, aperture, and other optical parameters live in the UsdGeom.Camera prim baked into the robot asset. They are intentionally absent from the YAML — duplicating them would create a divergence risk with no benefit. The YAML only carries sensor_prim_path and pixel resolution.

YAML format

sensor_rig:
  sensors:
    - name: front_hawk_left
      type: camera
      sensor_prim_path: chassis_link/sensors/front_hawk/left/camera_left
      width_px: 960
      height_px: 600

    - name: front_hawk_right
      type: camera
      sensor_prim_path: chassis_link/sensors/front_hawk/right/camera_right
      width_px: 960
      height_px: 600

    - name: chassis_imu        # unsupported — logged and skipped
      type: imu
      sensor_prim_path: chassis_link/sensors/chassis_imu/Imu_Sensor

Scaffolding the YAML

scripts/generate_sensor_rigs.py opens a robot USD, discovers all sensor prims, and writes sensor_prim_path entries with # TODO comments on fields requiring human review (name, resolution). Unsupported sensor types are included as comments for inventory. Run once per robot, edit the output, then commit.

BUILD=./_build/linux-x86_64/release
SCRIPT=source/extensions/isaacsim.replicator.mobility_gen.examples/scripts/generate_sensor_rigs.py
DATA=source/extensions/isaacsim.replicator.mobility_gen.examples/isaacsim/replicator/mobility_gen/examples/data/robots

"$BUILD/python.sh" "$SCRIPT" --list "$DATA/carter.yaml"   # inspect only
"$BUILD/python.sh" "$SCRIPT" "$DATA/carter.yaml"          # write in-place

Robot integration

MobilityGenRobot.__init_subclass__ reads the YAML at class-definition time and stores cls.sensor_configs. build() calls cls.build_sensor_rig(prim_path) after spawning the articulation; build_sensor_rig calls MobilityGenSensorRig.build_mounted. Robots with no sensor_configs (e.g. H1Robot) get sensor_rig = None.

robot = CarterRobot.build("/World/carter")
robot.sensor_rig.enable_rgb_rendering()
robot.sensor_rig.finalize_rendering()   # re-enables Hydra — must be called after all enable_*

# each simulation step
robot.update_state()
images = robot.state_dict_rgb()
# → {"sensor_rig.front_hawk_left.rgb_image": np.ndarray,
#    "sensor_rig.front_hawk_right.rgb_image": np.ndarray}

Footgun: enable_rgb_rendering() disables Hydra texture updates while annotators are attached to avoid OmniGraph crashes on partially-constructed nodes. finalize_rendering() re-enables Hydra. Omitting it causes cameras to silently produce no frames.

Recording and replay

The UI recording callback writes pose/state only — no sensor readings. Sensor data is generated at replay time by replay_scenario() in build.py, which restores recorded poses to the stage, steps the renderer, and writes annotator outputs. Render modalities (RGB, depth, semantic and instance ID segmentation, and normals) are selected at replay time via ReplayConfig.

Multi-sensor output is automatic: state_dict_rgb() traverses the Module tree recursively, keying each sensor buffer by its full dotted path. The writer uses the key as a folder name; the reader discovers folders via glob. Adding sensors to the YAML requires no changes to writer, reader, or scenario code.