10.10. Object Detection Synthetic Data Generation
omni.replicator.object is a no-code-change-required extension that generates synthetic data for model training, that can be used on a range of tasks from retail object detection to robotics.
It takes a description file in YAML that describes a mutable scene, or a hierarchy of such stacked description files as input, and outputs a description file along with graphics content including RGB, 2D/3D bounding boxes, segmentation masks, etc.
10.10.1. Run Instructions
The extension can be run from the UI, or isaac-sim container.
10.10.1.1. Run from the UI
In Omniverse Kit or Isaac Sim, enable omni.replicator.object from Windows -> Extension manager. A window will pop up.
Clicking on the folder icon or the Visual Studio Code icon below,
The root folder of the extension will be open. Under source/extensions/omni.replicator.object/omni/replicator/object/configs there are many description files in YAML format. It’s recommended to start with demo_kaleidoscope.yaml. To do so, go to global.yaml and update output_path to any local folder to store the simulation output.
Next, put demo_kaleidoscope in the Description File textbox, and hit Simulate to start simulation; alternatively, you can also put the absolute path .../source/extensions/omni.replicator.object/omni/replicator/object/configs/demo_kaleidoscope.yaml, with ... being the local path. Results will be stored in output_path.
10.10.1.2. Run from docker
To run the docker container,
docker run --gpus device=0 --entrypoint /bin/bash -v LOCAL_PATH:/tmp --network host -it ISAAC_SIM_DOCKER_CONTAINER_URL
Accordingly, update global.yaml to have output_path to be any folder under /tmp.
To launch the simulation, with demo_kaleidoscope for example,
bash kit/omni.app.dev.sh --no-window --enable omni.replicator.object --allow-root --/log/file=/tmp/omni.replicator.object.log --/log/level=warn --/windowless=True --/config/file=/tmp/omni.replicator.retail/source/extensions/omni.replicator.object/omni/replicator/object/configs/demo_kaleidoscope.yaml
/tmp/omni.replicator.object.log contains the messages from execution as well as from the extension. You can search the messages from the extension by filtering the file with METROPERF.
10.10.2. Concepts
10.10.2.1. The description file
The description file is a YAML file that has a key omni.replicator.object, and the value corresponding to that key is the description.
The description consists of key-value pairs. Each key-value pair is a Mutable, a Harmonizer, or a Setting.
The description generates frames as the user demands. Each frame, the scene is randomized, and graphics content is captured and output to disk. Settings describe how the scene is configured and how data is output. For example, you can set the number of frames to output, whether or not to output 2D bounding boxes, or set the gravity and friction of physics simulation.
The description composes the scene populated with objects that are called mutables.
Mutables randomize every frame. Sometimes we want to constrain how they randomize - for example, to know how other mutables are randomizing and randomize correspondingly. To do so, we can define harmonizers.
10.10.2.2. Simulation Workflow
Every time a simulation is launched, the description is parsed, and then the scene along with the mutables and harmonizers are initialized. Every frame it goes through the following stages.
Free randomize
Mutable attributes that doesn’t depend on harmonizer randomizes freely.
Harmonize
The harmonizers randomize, and collect information from free randomized mutable attributes. The collected information are called pitches. Just like real harmonizers, the pitches are collected and propagated back to all relevant mutables.
Harmonized randomize
Using the return information from the harmonizer, the harmonized mutable attributes randomizes.
Physics resolution
Physics is resolved so that objects move away from each other if they overlap, or drop onto a surface if gravity is turned on. For more details, refer to Physics simulation explained.
Capture
Graphics content is captured.
Logging
The state of the scene in this frame is recorded into a description file, such that later on it can be restored or inspected.
10.10.2.3. Scene restoration
In the output content, you can use the output description file of a specific frame to generate the exact same graphics content as when this frame was generated, once again. Or you can slightly modify it to have something different but everything else the same. In this way we support multiple-sampling for pretrained models.
10.10.3. Convention in this documentation
Type in the tables indicate the expected data types. Where a type is expected, a macro string can be used in place to be later on evaluated for that specific type. For example, if we expect int in a value, we can either give an int or something like $[index]. See Macro for details.
Within a mutable, aside from these 2 options, you can also specify a Mutable attribute to evaluate to this type.
And numeric means literal or evaluated float or int.