Object Detection Synthetic Data Generation

The omni.replicator.object is a no-code-change-required extension that generates synthetic data for model training that can be used on a range of tasks from retail object detection to robotics. The extension can be run from the UI or isaac-sim container.

It takes a YAML description file that describes a mutable scene, or a hierarchy of such stacked description files as input, and outputs a description file along with graphics content including RGB, 2D/3D bounding boxes, and segmentation masks.

../_images/overview.png

Run from the UI

You can install Kit 105.1/106 or Isaac Sim 4.0.0 from Omniverse Launcher.

  1. In Omniverse Kit 105.1/106 or Isaac Sim 4.0.0 , enable omni.replicator.object from Windows -> Extension manager.

    ../_images/run-1.png

  2. Click on the folder icon or the Visual Studio Code icon on the right side of the opened extension panel:

    ../_images/run-2.png

    The root folder of the extension opens.

  3. Under PATH_TO_EXTENSION/omni/replicator/object/configs there are many description files in YAML format. It’s recommended to start with demo_kaleidoscope.yaml. Go to global.yaml and update output_path to any local folder where you can store the simulation output.

Placeholders in description files

Note

Some example description files have placeholders. For example, in global, replace OUTPUT_PATH with a valid path. In demo_bottle, replace PATH_TO_BACKGROUND_IMAGES with a folder that contains JPEG images. In tutorial_harmonizer, replace PATH_TO_ORO with data/oro_tutorial_models/oro.usd in the extension’s root folder.

  1. Put demo_kaleidoscope in the Description File text box.

  2. Click Simulate to start the simulation.

Alternatively, to save to the output_path, for the Description File, you can put the absolute path PATH_TO_EXTENSION/omni/replicator/object/configs/demo_kaleidoscope.yaml.

Note

In the above and following context, PATH_TO_EXTENSION in Kit is something like USERNAME/.local/share/ov/data/exts/v2/omni.replicator.object-0.2.16, depending on version and local path; while in Isaac in Windows it is like USERNAME\AppData\Local\ov\pkg\isaac-sim-4.0.0-rc.10\extscache\omni.replicator.object-0.2.16, in Linux it is like USERNAME/.local/share/ov/pkg/isaac-sim-4.0.0-rc.10/extscache/omni.replicator.object-0.2.16/omni/replicator/object/configs/demo_kaleidoscope.yaml.

Run from Docker

To install the Isaac Sim Docker container, visit Container Deployment.

To run the Isaac Sim Docker container:

docker run --gpus device=0 --entrypoint /bin/bash -v LOCAL_PATH:/tmp --network host -it ISAAC_SIM_DOCKER_CONTAINER_URL

Accordingly, update global.yaml to have output_path to be any folder under /tmp.

For example, to launch the simulation with demo_kaleidoscope:

bash kit/omni.app.dev.sh --no-window --enable omni.replicator.object --allow-root --/log/file=/tmp/omni.replicator.object.log --/log/level=warn --/windowless=True --/config/file=PATH_TO_EXTENSION/omni/replicator/object/configs/demo_kaleidoscope.yaml

/tmp/omni.replicator.object.log contains the messages from execution as well as from the extension. You can search the messages from the extension by filtering the file with METROPERF.

To refer to resources residing in the Nucleus server, before launching simulation, run

export OMNI_USER='$omni-api-token' (as is)

export OMNI_PASS='<API_TOKEN>' (substitute the API token from the nucleus server)

Expected Output

After the simulation, the output is stored in output_path. The output content is determined by the output switches setting.

For example, the image output of demo_bottle is:

../_images/bottle_image.jpg

While the segmentation output is:

../_images/bottle_segmentation.png

The 2D bounding box is:

bottle_0 0 -1.0 0 1028 333 1362 2159 0 0 0 0 0 0 0
bottle_1 0 -1.0 0 1895 112 2277 1694 0 0 0 0 0 0 0
bottle_2 0 -1.0 0 1281 462 1854 2159 0 0 0 0 0 0 0

in which the 4 positive numbers indicate x_min, x_max, y_min, y_max. The number -1 is where the occlusion rate should be, but since a bottle is transparent, it is -1 here.

As another example, the image output of demo_kaleidoscope is:

../_images/donuts_image.jpg

While the segmentation output is:

../_images/donuts_segmentation.png

Concepts

Description File

The description file is a YAML file that has a main key named omni.replicator.object.

The description file consists of key-value pairs. Each key-value pair is a Mutable, a Harmonizer, or a Setting.

The description file generates frames as specified. Each frame scene is randomized,:ref:graphics content<Output switches> is captured, and output to disk. Settings describe how the scene is configured and how data is output. For example, you can set the number of frames to output, whether or not to output 2D bounding boxes, or set the gravity and friction of physics simulation.

The description file populates the scene with objects that are called mutables.

Mutables randomize every frame. Sometimes you might want to constrain how they randomize. For example, to know how other mutables are randomizing and randomize correspondingly. To do so, define harmonizers.

Example Minimal Description File Definition
omni.replicator.object:
   version: 0.1.2
   num_frames: 3
   output_path: OUTPUT_PATH
   screen_height: 1080
   screen_width: 1920
   seed: 0
Simulation Workflow

Every time a simulation is launched, the description file is parsed, and then the scene along with the mutables and harmonizers are initialized. Every frame it goes through the following stages.

Free randomize

Mutable attributes that don’t depend on harmonizer can randomize freely.

Harmonize

The harmonizers randomize and collect information from free randomized mutable attributes. The collected information is called pitches. Just like real harmonizers, the pitches are collected and propagated back to all relevant mutables.

Harmonized randomize

Using the return information from the harmonizer, the harmonized mutable attributes randomizes.

Physics resolution

If gravity is turned on, physics is resolved so that objects move away from each other when they overlap or drop onto a surface. For more details, see Physics simulation explained.

Capture

Graphics content is captured.

Logging

The state of the scene in this frame is recorded and saved, such that later on, it can be restored or inspected.

Scene Restoration

To support multiple-sampling for pretrained models:

In the output content, you can use the output saved from logging of a specific frame to generate the exact same graphics content as when this frame was generated. Or you can slightly modify it to have something different but everything else is the same.

Catalog

Conventions in the linked catalog files:

Type in the tables indicates the expected data types. Where a type is expected, a macro string can be used for later evaluation of that specific type. For example, if you expect int in a value, you can either give an int or something like $[index]. See Macro for details.

Within a mutable, aside from these options, you can also specify a Mutable Attribute to evaluate to this type.

And numeric means literal or evaluated float or int.