Customization#

Overview#

Isaacsim.Replicator.Agent (IRA) supports custom environment assets, custom character assets, and custom character animations.

Custom Environment Assets#

You are welcome to use your custom scene assets. To use custom environment assets, they must be in USD format and have the correct root layer attributes. This guide helps you find environment assets, convert scene assets to .usd format, configure and light the scene, and create the Navigation Mesh for the scene.

Find Environment Assets#

We recommend browsing the Isaac Sim Asset Browser for high quality assets that are ready-to-use for Omniverse.

You can also browse 3D asset stores like Turbosquid and Sketchfab. You can filter asset search on these sites by “USD” to find Omniverse-ready assets.

Any non-USD environment asset you want to use must be converted to USD.

Converting Non-USD Assets to USD#

The best way to convert a non-USD asset file is to determine what tools it is created with and use the corresponding Omniverse connector to convert it.
Isaac Sim has an extension that you can use to convert .fbx, .obj, .gltf formats to .usd. Refer to Omniverse Asset Converter Service to convert these formats.
You can also try to import assets from other formats using Omniverse Connectors.

The following steps are used to find the tools used created the asset and the corresponding connector. These steps use a .max file as an example.

Find the Tool of Creation#

When you find an asset on a store, it usually lists the software used to create it. For example Turbosquid contains the FORMATS section. In the example below, the asset was made with 3ds Max 2013 and V-Ray 3.2.

Turbosquid marketplace asset in 3ds Max 2013 and V-Ray 3.2 format

Some assets are created using multiple software packages, with one being used for modeling and the other being used for texture and materials. This asset used 3ds Max for modeling and V-Ray for adding materials.

Note

When multiple tools were used to create a sample, you must use the tool that makes the texture/material to perform the conversion. For example, if you tried to convert this asset with 3ds Max, the texture/material won’t be exported correctly and the asset will look grey when imported to Omniverse.

Using Omniverse Connector#

After confirming what software is used, use that software to convert the asset.

Research the software to determine if it already has a built-in USD converter. If so, follow the official documentation for the software to convert it to USD.

Typically, Omniverse has a Connector that can be used to convert your asset, even if your software doesn’t have one. Following these steps to use a Connector:

  1. Go to Omniverse Launcher app, select the EXCHANGE tab, and search for the app. For example, search for “3ds Max”.

  2. Install the Connector

  3. Re-open the app, in this example, 3ds Max.

  4. An Omniverse tab appears in your app. Click Export All or USD and follow the prompts to finish the conversion.

  5. All of the Connector installation pages have a Tutorials page. If you are feeling stuck, follow the tutorials.

If the software doesn’t have a built-in USD converter and Omniverse doesn’t have a Connector for this app yet, you can always convert the environment asset to an OBJ or FBX file and then use the Save As button in Isaac Sim to convert and save it as a USD.

Note

Some exported USD files may not open in Isaac Sim. You can try opening them in Omniverse USD Composer instead. Right-click on the asset USD file and click Collect Assets button. After the process is complete, open the collected asset USD in Isaac Sim.

Configure the Environment Asset#

With your environment asset in USD format, you might want to adjust scale or lighting before importing it into Isaac Sim.

To use a custom environment asset with IRA, you must create a NavMesh for it. This section aims to help you set up your new environment asset so it’s ready for the extension.

Unit and Coordinates System#

Isaac Sim is using meter as the unit and Z-up as the world coordinate system.

With the environment asset open in Isaac Sim, go to the right panel and select the Layer tab. Click the “Root Layer”, and the properties of the “World” are shown in the bottom right Property panel.

Typically, using Isaac Sim to convert the unit and system is the easiest way to get consistent coordinates:

  1. Open the Isaac Sim default empty stage.

  2. Drag your asset into the Stage panel on the left side.

Isaac Sim automatically creates a new Xform for it and converts it to the right config.

Alternatively, you can change the World Axis to Z and Meters Per Unit to 1 from the Layer Metadata attribute under the Root Layer Property, which can be found as the below default image shows. After changing the numbers, the scene is rotated in the wrong direction and in the wrong scale. You need to manually rotate and scale it to the right direction and size. To verify that the scene is in the right configuration, drag a character into the stage and see if it matches the size and direction of the environment.

the default stage's layer and property tabs

Note

Some scenes might be created without having real-world-scale in mind. For these assets, it is recommended to drag a character into the stage and scale the root xform of the environment asset according to how big it is compared to the scene.

Lighting the Environment#

Typically, the original lighting of an asset isn’t compatible with Isaac Sim after the conversion, or there might be no lighting at all. This section guides you through adding basic lighting to your scene.

Follow these steps to add a light source:

  1. Open the viewport.

  2. Place your mouse on the location where you want the light source.

  3. Right click.

  4. Select Create > Light.

  5. At the prompt, select the type of light source.

Add light sources to an environment if necessary.

The most important attribute of a light prim is its intensity. It determines how bright the light source is, and it can be adjusted by clicking the light prim on “Stage” and going to its Property tab on the right panel. The effects of these attributes are reflected in realtime. So, it is best to keep tuning the numbers until you like how it looks.

  • Dome Light

    A Dome Light, typically, lights the whole scene up. It is one of the most essential light types, because it is also the skysphere (background). A Dome Light can be applied with a texture file to give it a more natural background look. To do this, go to the Property tab for the light prim, find a texture file, and select it. The texture file must be in HDRI format. With an HDRI texture applied, it looks like a real sky.

To learn more about lighting, see the Omniverse lighting guide.

Building the NavMesh#

NavMesh defines the area that the agents can stand on and navigate to. It is essential for the random spawning function and command generation to work. The Omni.Anim.Navigation extension has greatly simplified this process and is continuing to evolve.

This section guides you through building a basic navigation mesh and teaches you some tricks for making modifications to the NavMesh. The steps to build a NavMesh are as follows:

  1. Add NavMesh Volume.

  2. Adjust settings.

  • NavMesh Volume

    Any polygons that intersect with the NavMesh Volume are considered walkable area. If you have a giant polygon as the ground, and the navmesh volume only encloses a small part of the ground, it still generates navmesh on the whole ground because it’s determined by intersecting polygons instead of enclosing ones.

    Follow these steps to create a Navmesh Volume:

    1. Right-click the viewport.

    2. Select Create > Navigation > NavMeshVolume. The NavMeshVolume appears in the viewport as a box wireframe.

    3. Change its scale in the regular way.

    Typically it is best to scale the NavMeshVolume to include the whole floor area, including props on the floor (tables and chairs) and exclude the ceiling. You want to include the props so that they can be treated as obstacles, and you want to exclude the ceiling so it won’t be counted as a walkable area.

Custom Character Assets#

IRA supports using your own USD character assets. Custom characters must be retargeted to the NVIDIA biped character before animations can be correctly applied. See Animation Retargeting for how to retarget a character. In addition, make sure the characters are in the same measurement (in meter) and have the same orientation (Z as up axis, -Y as forward axis).

IRA also expects the character folder to contain a list of sub-folders, where each sub-folder has only 1 USD inside (the character USD). Organize your custom character assets in this manner. Now you can set the character asset folder in the config file to your custom character assets folder and spawn custom characters. See below image for the folder structure.

Character asset folder structure.

Note

You can also refer to [Isaac Sim Assets Path]/Isaac/People/Characters as an example.

Import Unreal Characters#

The Unreal Engine Marketplace offers high-quality content that can potentially be used in IRA. In this section, we demonstrate how to enable Unreal characters for use in Isaac Sim.

  1. Export characters from Unreal as USD. Refer to Unreal Engine Connector for details.

  2. Import USD Unreal characters into Isaac Sim. Because Unreal characters are in centimeters and have a different skeleton hierarchy than the NVIDIA biped character, the following steps are required:

    1. Run the following script in the script editor to scale characters from centimeters to meters.

      Scale a character prim
       1from pxr import Gf, Usd
 2import omni.usd
 3
 4def convert_mesh(mesh_prim, scaleMatrix):
 5        # convert points
 6        points_attr = mesh_prim.GetAttribute("points")
 7        if points_attr != None:
 8                points = points_attr.Get()
 9        for i in range(len(points)):
10                p = points[i]
11                p = scaleMatrix.Transform(p)  # change unit
12                points[i] = p
13        points_attr.Set(points)
14        # convert extent
15        extent_attr = mesh_prim.GetAttribute("extent")
16        if extent_attr != None:
17                extents = extent_attr.Get()
18        else:
19                extents = None
20        if extents != None:
21                for i in range(len(extents)):
22                        extent = extents[i]
23                        extent = scaleMatrix.Transform(extent)  # change unit
24                        extents[i] = extent
25                extent_attr.Set(extents)
26
27def convert_transform(transform, scaleMatrix):
28        trans = transform.ExtractTranslation()
29        rot = transform.ExtractRotation()
30        trans = scaleMatrix.Transform(trans)  # unit conversion
31        transform.SetTranslateOnly(trans)
32        transform.SetRotateOnly(rot)
33        return transform
34
35def convert_skeleton(skeleton_prim, scaleMatrix):
36        # Convert bind pose
37        bind_transforms_attr = skeleton_prim.GetAttribute("bindTransforms")
38        if bind_transforms_attr != None:
39                bind_transforms = bind_transforms_attr.Get()
40                if bind_transforms != None:
41                        for i in range(0, len(bind_transforms)):
42                                bind_transforms[i] = convert_transform(bind_transforms[i], scaleMatrix)
43                        bind_transforms_attr.Set(bind_transforms)
44        # Convert rest pose
45        rest_transforms_attr = skeleton_prim.GetAttribute("restTransforms")
46        if rest_transforms_attr != None:
47                rest_transforms = rest_transforms_attr.Get()
48                if rest_transforms != None:
49                        for i in range(0, len(rest_transforms)):
50                                rest_transforms[i] = convert_transform(rest_transforms[i], scaleMatrix)
51                        rest_transforms_attr.Set(rest_transforms)
52        # Convert retarget pose
53        retarget_transforms_attr = skeleton_prim.GetAttribute("controlRig:retargetTransforms")
54        if retarget_transforms_attr != None:
55                retarget_transforms = retarget_transforms_attr.Get()
56                if retarget_transforms != None:
57                        for i in range(len(retarget_transforms)):
58                                if i == 0:
59                                        retarget_transforms[i] = convert_transform(retarget_transforms[i], scaleMatrix)
60                                else:
61                                        retarget_transforms[i] = convert_transform(retarget_transforms[i], scaleMatrix)
62                        retarget_transforms_attr.Set(retarget_transforms)
63        # Convert default animation source
64        animation_source_attr = skeleton_prim.GetRelationship("skel:animationSource")
65        if animation_source_attr != None:
66                skeleton_prim.CreateRelationship("skel:animationSource")
67
68# From centimeter to meter
69scale_factor = 0.01
70character_root_prim_path = "[your character prim root]"
71
72scaleMatrix = Gf.Matrix4d(1.0)
73scaleMatrix.SetScale(scale_factor)
74
75character_root_prim = omni.usd.get_context().get_stage().GetPrimAtPath(character_root_prim_path)
76for prim in Usd.PrimRange(character_root_prim):
77        if prim.GetTypeName() == "Mesh":
78                convert_mesh(prim, scaleMatrix)
79        elif prim.GetTypeName() == "Skeleton":
80                convert_skeleton(prim, scaleMatrix)

    2. Retarget the characters to NVIDIA biped character by following Animation Retargeting.

If the characters are correctly setup, they can replicate the movement of an NVIDIA biped character by running its Animation Graph.

Upon the successful conversion of Unreal character, the imported character replicates the movement of the standard AnimGraph biped character. Upon the successful conversion of Unreal character, the imported character replicates the movement of the standard AnimGraph biped character.

Custom Character Animations#

IRA supporst custom character animations in the form of custom command. You can map a custom animation to a command defined by you. Please see the IRA Custom Character Commands guide for more information.
However, before an animation is ready to be imported, you need to make sure that the animation is compatible with the NVIDIA biped skeleton. You can run the following script to retarget an animation from one skeleton to another.
omni.kit.commands.execute( 'CreateRetargetAnimationsCommand', source_skeleton_path='',
target_skeleton_path='',
source_animation_paths=[''],
target_animation_parent_path='',
set_root_identity=False )
On this page