Is this page helpful?

[isaacsim.streaming.rtsp] RTSP Streaming OmniGraph Nodes#

Version: 0.1.3

Overview#

The isaacsim.streaming.rtsp extension provides RTSP streaming for camera render products in Isaac Sim. It captures rendered frames through a Replicator writer and publishes them over RTSP via omni.kit.livestream.rtsp. OmniGraph nodes are included for graph-based streaming setup. External clients can connect with any standard RTSP player using rtsp://<host>:<port><mountPath>.

Usage#

The extension is not enabled by default in the Isaac Sim full application. Enable it explicitly from the Extension Manager or start Isaac Sim with --enable isaacsim.streaming.rtsp when RTSP streaming is needed.

Functionality#

The extension registers RTSPStreamWriter with the Replicator WriterRegistry on startup. The writer supports two encoding modes: pre-encoded H.264 with per-frame SEI metadata injection, and raw CUDA buffer passthrough where encoding is handled by the livestream backend. The RTSP server is started lazily on the first rendered frame and stopped when the writer detaches. Multiple simultaneous streams are supported when each stream uses a unique port and mount path.

Integration#

The extension depends on omni.replicator.core for frame capture and annotator management, omni.kit.livestream.core and omni.kit.livestream.rtsp for the RTSP server backend, and isaacsim.core.nodes for the OmniGraph writer node base class. Camera render products are created elsewhere in the application or graph and passed to the writer at attach time.

Enable Extension#

The extension can be enabled (if not already) in one of the following ways:

Define the next entry as an application argument from a terminal.

APP_SCRIPT.(sh|bat) --enable isaacsim.streaming.rtsp

Define the next entry under [dependencies] in an experience (.kit) file or an extension configuration (extension.toml) file.

[dependencies]
"isaacsim.streaming.rtsp" = {}

Open the Window > Extensions menu in a running application instance and search for isaacsim.streaming.rtsp. Then, toggle the enable control button if it is not already active.

Python API#

RTSP Stream Writer#

class RTSPStreamWriter(*args: Any, **kwargs: Any)#

Bases: Writer

Replicator Writer that streams LdrColor frames over RTSP.

Supports two modes controlled by the encoding parameter:

  • "h264" (default): The LdrColor annotator is requested with init_params={"compression": "h264"} so the render pipeline produces H.264-encoded bytes. These are passed to stream_video_pre_encoded_with_metadata, and also have the ability to enable per-frame SEI metadata injection.

  • "raw": The LdrColor annotator delivers a CUDA RGBA buffer which is passed to stream_video_cuda_buffer. The kit-livestream RTSP backend handles encoding internally. Metadata injection is not supported in this mode.

The RTSP server is started lazily on the first rendered frame. Calling detach() (which happens automatically when BaseWriterNode resets on timeline stop) shuts the server down cleanly.

Parameters:

  • port – RTSP server port (1 to 65535). Each simultaneous stream needs a unique port.

  • mountPath – RTSP mount path (e.g. /stream); must start with /.

  • encoding – "h264" (default) for pre-encoded H.264 with metadata support, "raw" for uncompressed CUDA path.

  • width – Frame width in pixels. Used to configure the RTSP server when encoding is "h264" (the encoded byte-stream does not carry resolution). Ignored when encoding is "raw" since the resolution is read from the CUDA buffer shape.

  • height – Frame height in pixels. See width.

  • sensorSetName – Optional SRTX sensor-set name passed to the LdrColor annotator through init_params.

Raises:

ValueError – If port, mountPath, or encoding is invalid.

detach() → None#

Stop the RTSP server and detach from the render product.

write(data: dict) → None#

Called by Replicator each frame with annotator data.

The data dict is keyed by annotator name. Replicator appends a render-product suffix to the key (e.g. "LdrColor-<rp_name>"), so we match with startswith rather than an exact lookup.

Render Variable Utilities#

USD render-product / render-var helpers for the RTSP streaming extension.

These helpers exist to satisfy omni.replicator.srtx’s AnnotatorSRTX.attach validation, which requires that the AOV’s RenderVar prim already exists as a child of the render product (matched by sourceName) before a Replicator Writer is attached.

ensure_render_var_on_product(
stage: pxr.Usd.Stage,
render_product_path: str,
aov_name: str,
compression_type: str,
) → tuple[bool, str | None]#

Ensure a RenderVar for the given AOV exists as a child of the render product and is in orderedVars.

The helper is authoritative for the srtx:compression:type attribute on the rendervar prim: any pre-existing value is overwritten with compression_type (which may be the empty string, the canonical SRTX “no compression / raw” signal). The attribute is created if missing.

Parameters:

  • stage – The USD stage.

  • render_product_path – Path to the render product prim.

  • aov_name – The AOV source name to match or create.

  • compression_type – SRTX compression type to author on the rendervar’s srtx:compression:type attribute. Pass "" for raw / no compression, or one of the SRTX-recognised codec names (e.g. "h264", "h265", "hevc").

Returns:

A (success, rendervar_path) tuple. On failure rendervar_path is None and success is False.

Omnigraph Nodes#

The extension exposes the following Omnigraph nodes: