RTX Sensor Annotators#
The isaacsim.sensors.rtx extension uses Omniverse Replicator to provide Annotators for RTX Lidar and Radar data collection.
Annotators can be attached to render products, attached to OmniSensor prims (for example, OmniLidar or OmniRadar); for example,
when run in the Script Editor, the following snippet creates an OmniLidar prim at /lidar, a render product for the sensor,
and attaches an IsaacExtractRTXSensorPointCloudNoAccumulator annotator to the render product.
import omni
import omni.replicator.core as rep
from pxr import Gf
# Create an OmniLidar prim at prim path /lidar
_, sensor = omni.kit.commands.execute(
"IsaacSensorCreateRtxLidar",
translation=Gf.Vec3d(0.0, 0.0, 0.0),
orientation=Gf.Quatd(1.0, 0.0, 0.0, 0.0),
path="/lidar",
)
# Create a render product for the sensor.
render_product = rep.create.render_product(sensor.GetPath(), resolution=(1024, 1024))
# Create an annotator
annotator = rep.AnnotatorRegistry.get_annotator("IsaacExtractRTXSensorPointCloudNoAccumulator")
# Attach the render product after the annotator is initialized.
annotator.attach([render_product.path])
Alternatively, the LidarRtx class offers a single API for attaching any annotator to an OmniLidar prim
and collecting data. For example, in a standalone Python workflow, the following snippet creates an OmniLidar prim at /lidar,
creates a render product for the sensor, and attaches an IsaacExtractRTXSensorPointCloudNoAccumulator annotator to it, then collects
data from the annotator on each simulation frame.
from isaacsim import SimulationApp
kit = SimulationApp()
import numpy as np
import omni
from isaacsim.sensors.rtx import LidarRtx
# Create the RTX Lidar with the specified attributes.
sensor = LidarRtx(
prim_path="/lidar",
translation=np.array([0.0, 0.0, 1.0]),
orientation=np.array([1.0, 0.0, 0.0, 0.0]),
config_file_name="Example_Rotary",
)
# Initialize the LidarRtx object, which creates a render product for the sensor.
sensor.initialize()
# Attach an annotator to the sensor.
sensor.attach_annotator("IsaacExtractRTXSensorPointCloudNoAccumulator")
# Play the timeline to initialize the OmniGraph associated with the annotator and render product,
# and begin collecting data.
timeline = omni.timeline.get_timeline_interface()
timeline.play()
# Collect data from the annotator on each simulation frame.
while kit.is_running():
# Step the simulation
kit.update()
# Print data collected by each annotator attached to the sensor as a Python dict
print(sensor.get_current_frame())
timeline.stop()
kit.close()
Annotators#
Each isaacsim.sensors.rtx Annotator is associated with a specific isaacsim.sensors.rtx OmniGraph node, which is linked
in that Annotator’s subsection below. The inputs and outputs of the Annotator are the same as the inputs and outputs of the
corresponding OmniGraph node.
Note
In Isaac Sim 5.0, several existing isaacsim.sensors.rtx annotators were removed in favor of simpler
annotators that can handle output from the new OmniLidar or OmniRadar prims, in addition to the
deprecated Camera-prim-based workflows. See Deprecated Annotators for details.
If the Lidar rotation rate is slower than the frame rate, data from Annotators for accumulated Lidar scans will contain returns from multiple frames. If the Lidar prim moves between frames, or objects
move in the scene, the buffer might contain returns from before the Lidar or objects moved, causing points to appear as though they are “dragging” behind objects when
viewed with the DebugDrawPointCloud or DebugDrawPointCloudBuffer writers.
isaacsim.sensors.rtx annotators rely on the GenericModelOutput AOV from the OmniLidar prim being
provided on device. If --/app/sensors/nv/lidar/outputBufferOnGPU or --/app/sensors/nv/radar/outputBufferOnGPU is
set to false, the annotators will not function correctly.
IsaacCreateRTXLidarScanBuffer#
The IsaacCreateRTXLidarScanBuffer Annotator accumulates frames of data from an OmniLidar prim into a single scan,
and provides the accumulated scan data as outputs. It is associated with the IsaacCreateRTXLidarScanBuffer node.
By default the node outputs a 3D Cartesian point cloud, and
can optionally the following data if the user sets the corresponding input flag on the node to true.
The node outputs data as pointers to buffers and the table below specifies the data type of each buffer.
Output |
Type |
Description |
Notes |
|---|---|---|---|
|
|
3D Cartesian point cloud. |
Always provided. |
|
|
Azimuth of each return, in degrees. |
Provided if |
|
|
Elevation of each return, in degrees. |
Provided if |
|
|
Range of each return, in world units (by default, meters). |
Provided if |
|
|
Intensity of each return, normalized as described here. |
Provided if |
|
|
Timestamp of each return, in nanoseconds since the start of the simulation. |
Provided if |
|
|
ID of the emitter that emitted the return. |
Provided if |
|
|
ID of the material of the object that generated the return. |
Provided if |
|
|
ID of the object that generated the return. |
Provided if |
|
|
Normal to the surface of the object that generated the return. |
Provided if |
|
|
Velocity of the object that generated the return. |
Provided if |
Enable nonzero normal output by setting --/app/sensors/nv/lidar/publishNormals=true will increase VRAM usage and might negatively impact performance.
IsaacComputeRTXLidarFlatScan#
The IsaacComputeRTXLidarFlatScan Annotator extracts depth and azimuth data from an accumulated 2D RTX Lidar scan.
It is associated with the IsaacComputeRTXLidarFlatScan node.
The IsaacComputeRTXLidarFlatScan Annotator does not support RTX Radar.
If the annotator is attached to a 3D Lidar (defined as having emitters at nonzero elevation angles), the annotator will not return any data.
Even if --/app/sensors/nv/lidar/outputBufferOnGPU=true is set, IsaacComputeRTXLidarFlatScanSimulationTime output data will be on host memory.
IsaacExtractRTXSensorPointCloudNoAccumulator#
The IsaacExtractRTXSensorPointCloud Annotator extracts the GenericModelOutput buffer’s point cloud data
into a Cartesian vector data buffer every frame. It is associated with the IsaacCreateRTXLidarScanBuffer node, with enablePerFrameOutput set to true.
Note
Isaac Sim 5.0 previously used the IsaacExtractRTXSensorPointCloud node for this annotator, but that node was removed
after performance improvements were made to the IsaacCreateRTXLidarScanBuffer node.
Reading Data from the GenericModelOutput Buffer#
Note
Isaac Sim 4.5 included the OgnIsaacReadRTXLidarData node, which provided an
example of reading data from the GenericModelOutput buffer in Python. This node has been removed
as of Isaac Sim 5.0 and replaced by the utility module and functions described below.
The isaacsim.sensors.rtx.generic_model_output Python module provides APIs for inspecting the
GenericModelOutput buffer, generated by the GenericModelOutput annotator.
For more information on the GenericModelOutput buffer, see the API documentation..
For an example of reading data from the GenericModelOutput buffer from Isaac Sim, checkout the
standalone example located at standalone_examples/api/isaacsim.sensors.rtx/inspect_lidar_metadata.py.
Resolving Object IDs as Prim Paths#
The GenericModelOutput struct includes a objId field and the IsaacCreateRTXLidarScanBuffer node outputs an optional objectId output.
In both cases, the data is provided as a numpy array of dtype np.uint8, and is only populated if --/rtx-transient/stableIds/enabled=true is set.
This data is meant to be interpreted as a sequence of 128-bit unsigned integers (effectively stride 16), which are stable, unique IDs corresponding to
unique prim paths in the scene. In other words, the i-th 128-bit unsigned integer in the array corresponds to prim generating the i-th return from the sensor.
The isaacsim.sensors.rtx.LidarRtx class provides two utility functions for resolving object IDs as prim paths.
First, LidarRtx.decode_stable_id_mapping resolves the output of the StableIdMap AOV (which can be generated from an OmniLidar, OmniRadar, or Camera prim)
as a Python dict mapping 128-bit unsigned integers to prim paths.
Second, LidarRtx.get_object_ids resolves the object ID array output from GenericModelOutput or IsaacCreateRTXLidarScanBuffer as 128-bit unsigned integers.
Refer to standalone_examples/api/isaacsim.sensors.rtx/resolve_object_ids_from_gmo.py for an example of using these functions to resolve object IDs as prim paths.
Deprecated Annotators#
Several annotators have been removed and or replaced by the annotators described above, as of Isaac Sim 5.0.
New annotator outputs are not guaranteed to be the same as the outputs of the deprecated annotators, gmoBufferPointer the table below describes affected annotators and how to replace them.
Deprecated Isaac Sim 4.5 Annotator |
Replacement |
Details |
|---|---|---|
|
|
The new annotator outputs the same data as the old annotator. |
|
|
The new annotator outputs the same data as the old annotator. |
|
|
The new annotator outputs the same data as the old annotator, excluding |
|
|
See above. |
|
|
See above. |
|
|
See above. |
|
|
See Reading Data from the GenericModelOutput Buffer for details. |