[isaacsim.benchmark.services] Benchmark Services

Version: 2.6.3

This extension provides benchmarking utilities

Enable Extension

The extension can be enabled (if not already) in one of the following ways:

Command-line interface

Define the next entry as an application argument from a terminal.

APP_SCRIPT.(sh|bat) --enable isaacsim.benchmark.services

Experience/extension configuration

Define the next entry under [dependencies] in an experience (.kit) file or an extension configuration (extension.toml) file.

[dependencies]
"isaacsim.benchmark.services" = {}

Extension Manager UI

Open the Window > Extensions menu in a running application instance and search for isaacsim.benchmark.services. Then, toggle the enable control button if it is not already active.

API

Python API

BaseIsaacBenchmarkAsync
BaseIsaacBenchmark	Front-end for benchmarking standalone scripts and other non-async snippets.

class BaseIsaacBenchmarkAsync(*args: Any, **kwargs: Any)

Bases: AsyncTestCase

async setUp(backend_type: str = 'JSONFileMetrics')

Must be awaited by derived benchmarks to properly set up the benchmark

async tearDown()

Must be awaited by derived benchmarks to properly tear down the benchmark

set_phase(

phase: str,

start_recording_frametime: bool = True,

start_recording_runtime: bool = True,

) → None

Sets benchmarking phase. Turns on frametime and runtime collection.

Parameters:

• phase (str) – Name of phase, used in output.

• start_recording_frametime (bool) – False to not start recording frametime at start of phase. Default True.

• start_recording_runtime (bool) – False to not start recording runtime at start of phase. Default True.

async store_measurements(

stop_recording_time: bool = True,

) → None

Stores measurements, metadata, and artifacts collected by all recorders during the previous phase. Optionally, ends frametime and runtime collection.

Parameters:

stop_recording_time (bool) – False to not stop recording runtime and frametime at end of phase. Default True.

async fully_load_stage(usd_path)

Await this function to open a stage and then wait for it to fully load

Parameters:

usd_path (str) – Path to USD stage.

async store_custom_measurement(

phase_name: str,

custom_measurement: <module 'isaacsim.benchmark.services.metrics.measurements' from '/builds/omniverse/isaac/omni_isaac_sim/_build/linux-x86_64/release/exts/isaacsim.benchmark.services/isaacsim/benchmark/services/metrics/measurements.py'>,

) → None

Stores the custom measurement specificed in the benchmark.

Parameters:

• phase (str) – The phase name to which the measurement belongs.

• measurement (Measurement) – The measurement object to store.

class BaseIsaacBenchmark(

benchmark_name: str = 'BaseIsaacBenchmark',

backend_type: str = 'OmniPerfKPIFile',

report_generation: bool = True,

workflow_metadata: dict = {},

)

Bases: object

Front-end for benchmarking standalone scripts and other non-async snippets.

By default this class will collect hardware performance data (see recorders), broken up by “phase”. Typical structure looks like:

benchmark = BaseIsaacBenchmark(benchmark_name=..., workflow_metadata=...)
benchmark.set_phase("loading")
# load stage, configure sim, etc.
benchmark.store_measurements()
benchmark.set_phase("benchmark")
# Actual code being benchmarked (running the sim for N frames, cloning an object, etc.)
benchmark.store_measurements()
benchmark.stop() # Shuts down benchmark, writes metrics to file

You can set any number of phases.

stop()

Stop benchmarking and write accumulated metrics to file.

set_phase(

phase: str,

start_recording_frametime: bool = True,

start_recording_runtime: bool = True,

) → None

Sets benchmarking phase. Turns on frametime and runtime collection.

Parameters:

• phase (str) – Name of phase, used in output.

• start_recording_frametime (bool) – False to not start recording frametime at start of phase. Default True.

• start_recording_runtime (bool) – False to not start recording runtime at start of phase. Default True.

store_measurements(

stop_recording_time: bool = True,

) → None

Stores measurements, metadata, and artifacts collected by all recorders during the previous phase. Optionally, ends frametime and runtime collection.

Parameters:

stop_recording_time (bool) – False to not stop recording runtime and frametime at end of phase. Default True.

fully_load_stage(usd_path: str) → None

Loads provided USD stage, blocking execution until it is fully loaded.

Parameters:

usd_path (str) – Path to USD stage.

store_custom_measurement(

phase_name: str,

custom_measurement: <module 'isaacsim.benchmark.services.metrics.measurements' from '/builds/omniverse/isaac/omni_isaac_sim/_build/linux-x86_64/release/exts/isaacsim.benchmark.services/isaacsim/benchmark/services/metrics/measurements.py'>,

) → None

Stores the custom measurement specificed in the benchmark.

Parameters:

• phase (str) – The phase name to which the measurement belongs.

• measurement (Measurement) – The measurement object to store.

Settings

Extension Settings

The table list the extension-specific settings.

Setting name	Description	Type	Default value
metrics.nvdataflow_default_test_suite_name	Default test suite name.	str	'Isaac Sim Benchmarks'
metrics.randomize_filename_prefix	Whether to add a randomly generated string as a prefix to the output filename to distinguish runs.	bool	False

The extension-specific settings can be either specified (set) or retrieved (get) in one of the following ways:

Set setting

Command-line interface

Define the key and value of the setting as an application argument from a terminal.

APP_SCRIPT.(sh|bat) --/exts/isaacsim.benchmark.services/SETTING_NAME=SETTING_VALUE

Experience/extension configuration

Define the key and value of the setting under [settings] in an experience (.kit) file or an extension configuration (extension.toml) file.

[settings]
exts."isaacsim.benchmark.services".SETTING_NAME = SETTING_VALUE

By programming

Define the key and value of the setting using the carb framework (in Python).

import carb

settings = carb.settings.get_settings()
settings.set("/exts/isaacsim.benchmark.services/SETTING_NAME", SETTING_VALUE)

Get setting

By programming

Define the key to query the value of the setting using the carb framework (in Python).

import carb

settings = carb.settings.get_settings()
value = settings.get("/exts/isaacsim.benchmark.services/SETTING_NAME")