Agent Control#

Overview#

Isaacsim.Replicator.Agent (IRA) supports controlling two different agents:

  • human characters

  • autonomous mobile robots, including the Nova Carter and iw_hub

This section discusses supported commands for each agent, how to add custom commands, how to customize the randomization of the commands, and how to inject commands during simulation.
The IRA UI provides command editors under the agent setting panel, so you can edit the commands for each agent and control the behaviors for the agents in the simulation. IRA also has the ability to randomize the commands.
Additionally, the IRA UI allows you to inject commands into a running simulation to control the agents during runtime.

Agent Behaviors#

All the agents are controlled by Behavior Script , which subscribes to USD stage events and reacts to them. You’re welcome to use your own Behavior Scripts to control the agents. You can run the following script from the Script Editor to change the path to behavior scripts of the agents spawned by IRA.

import carb
my_setting = carb.settings.get_settings()
my_character_script_path = "ENTER YOUR CHARACTER SCRIPT PATH HERE"
my_carter_script_path = "ENTER YOUR CARTER SCRIPT PATH HERE"
my_transporter_script_path = "ENTER YOUR TRANSPORTER SCRIPT PATH HERE"

character_behavior_path = "/exts/isaacsim/replicator/agent/behavior_script_settings/behavior_script_path"
nova_carter_behavior_path = "/exts/isaacsim/replicator/agent/behavior_script_settings/nova_carter_behavior_script_path"
transporter_behavior_path = "/exts/isaacsim/replicator/agent/behavior_script_settings/transporter_behavior_script_path"
my_setting.set(character_behavior_path, script_path)
my_setting.set(nova_carter_behavior_path, my_carter_script_path)
my_setting.set(transporter_behavior_path, my_transporter_script_path)

Character#

The Character Behavior control is based on the Omni.Anim.People extension. See below for more details.
The GoTo, Idle, and LookAround commands can be randomized. Characters can also Sit and Queue, but these commands require a manual input.

GoTo#

GoTo moves the character to a location. It can be followed by a single point or a sequence of points, where the last point specifies the ending rotation that the character must have upon reaching its destination. If you would prefer to not set a rotation, you can use “_” for the ending rotation.

When a sequence of points is given, the character crosses every point. If Navmesh Navigation and Dynamic Obstacle Avoidance is on, then the character tries to avoid static and dynamic obstacles.

Example: Character_02 GoTo 10 10 0 90

Note

  • Character rotation in a fixed spot is done by rotating the whole character body.

  • Depending on the navigation mesh of the stage, characters might collide with static obstacles while walking. If this is observed, rebuild the navigation mesh with a higher agent radius.

  • Character dynamic avoidance is best effort and not guaranteed. In some scenarios where static and dynamic obstacles are present together, dynamic avoidance might cause erratic movement leading characters to walk into another character or into a static obstacle.

Idle#

Idle makes the character stand still. It takes a duration value in seconds and performs the action for that duration.

Example: Character_02 Idle 10

LookAround#

LookAround makes the character stand in the same spot, while moving its head from left to right. It takes a duration value in seconds and performs the action for that duration.

Example: Character_03 LookAround 10

Sit#

Sit makes the character go to a prim and attempt to sit on it. After the specified duration ends, the character stands back up. It takes a prim_path for the seat and a duration value in seconds and performs the action on that prim for that duration.

Example: Character_03 Sit /World/Chair 5

Note

The Sit action plays a fixed animation, so the seat_prim must be scaled to fit the animation. Additionally, the seat_prim must contain two child prims: walk_to_offset and interact_offset. The walk_to_offset translation specifies the location where the character will walk to and its rotation defines the character’s final orientation before interacting with the chair. The interact_offset specifies the character’s location and rotation when the sitting animation is played. For example, refer to Multi_Event_Simulation in the samples folder.

Queue#

For the Queue command, you must define a queue. A queue can be defined in the command file as follows:

command structure:

    # defines the queue
    Queue Example_Queue

    # Defines each spot in the queue. Structure Queue_Spot 'Queue_name' 'Spot_Index' x y z rotation
    Queue_Spot Example_Queue -1 11 0 0 90

    Queue_Spot Example_Queue 0 9 0 0 0

    Queue_Spot Example_Queue 1 7 0 0 0

    Queue_Spot Example_Queue 2 5 0 0 0

After defining the queue, characters can be moved to the queue using the following command structure:

# make the character enter the queue
Character_01 Queue Example_Queue

# action to perform when the character becomes the first in queue
Character_01 LookAround 10

# destination to go to when leaving the queue. In this example, Character_01 will go to 10, 10, 0.
Character_01 Dequeue Example_Queue 10 10 0 _
expected result for the Queue command

Note

  • All three commands listed above are mandatory for the queue command to work correctly.

  • When constructing a queue, each queue spot must be at least 0 meter apart from each other.

Robot#

Robot behavior control is supported by the Isaacsim.Wheeled_Robots extensions. The GoTo and Idle commands can be randomized by IRA. The iw.hub robot supports the additional LiftUp and LiftDown commands, which both require manual input.

GoTo#

The GoTo command moves the robot from its current location to a target location. The command expects the X Y Z world coordinate of the target location.

Example: Nova_Carter_01 GoTo 10.02 5.9 0

Idle#

The Idle command makes the robot stay where it is for a given duration in seconds. It can be combined with the GoTo command to make the robot go to a position, stay there for a while, and then go to another position.

Example: Nova_Carter_01 Idle 5

LiftUp/LiftDown#

The iw.hub robot also supports the LiftUp and LiftDown commands, which move the loader on the iw.hub up or down by 4 cm.

Example: Transporter_02 LiftUp

Avoid Moving Objects#

Characters can be made to avoid small and slow moving objects (like robots) by attaching the dynamic_obstacle.py script to the moving object. Avoidance is best effort and not guaranteed. Follow these steps:

  1. Find the ‘moving component’ of the object you want to avoid. This is generally the base prim of a physics articulation.

  2. On the prim, Right Click > Add > Python Scripting.

  3. In the property window, find the Python Scripting property, click on Select Asset and select the dynamic_obstacle.py script in the omni.anim.people extension files.

Note

Verify that the dynamic_obstacle.py on obstacle and the character_behavior.py on your characters are in the same folder.

Custom Commands and Randomization#

Custom Character Commands#

IRA supports custom commands for the characters by allowing you to import new animations as commands. Custom commands can be then randomized by setting up the IRA transition matrix.

Prerequisite#

You can create custom commands by providing animation USDs with the required attributes to our extension. When animation USDs are setup with the following attributes, they are identified as new command types and an animation graph node is created accordingly.

List of attributes to define an animation USD as custom command:

  • CustomCommandName: The name of the custom command. Data type is string.

  • CustomCommandTemplate: The command template used to register this animation. Data type is string.

  • CustomCommandAnimStartTime: The animation start time. Data type is float.

  • CustomCommandAnimEndTime: The animation end time. Data type is float.

There are three types of custom command templates:

  • Timing: Plays the animation for a given time. The command structure is the same as Idle command.

  • TimingToObject: Walks the character to an object and plays animation. The command structure is the same as Sit command.

  • GoToBlend: Blends the animation into current walking animations. The command structure is the same as GoTo command.

Refer to the this guide for how to add and set USD attributes in Omniverse.

Add a Custom Command#

  1. Open an animation USD, add all the attributes listed above to the root prim. In this example, “CustomCommandName” is set to “CallingStand” and “CustomCommandTemplate” is set to “Timing”.

  1. Open the Custom Command UI window by going to Tools > Replicator > Custom Command.

  2. In the Custom Command window, click Add and select the animation USD in step 1. The extension will now recognize the new command.

  1. After the Custom Command is added, click Set Up Simulation from the IRA UI and you can use the Command textbox from the Character Settings panel to test the new command. If you want the newly added command to be randomized by IRA, see Custom Character Command Randomization

Note

The following animation assets have been configured as custom commands. You can directly load them for testing.

  • [Isaac Sim Asset Path]/Isaac/People/Animations/push_button.skelanim.usd

expected result for the GoTo command

  • [Isaac Sim Asset Path]/Isaac/People/Animations/type_keyboard.skelanim.usd

expected result for the GoTo command

Custom Character Command Randomization#

IRA allows you to add custom commands with custom animations, as well as customize the randomization of these commands. To add new commands, refer to Custom Command. For every new command added, IRA automatically adds an entry to the Command Randomization transition matrix.

You can control the possibilities of command randomization by defining the command transition matrix. The command transition matrix determines the possibility of transitioning to a state from a state. Each row of the transition matrix represents, under the current command (row head), what the possibility of going to the next command is.

For example, in the following transition matrix, a GoTo command has a 50% chance of being followed by an Idle command and 50% chance of being followed by a LookAround command. A LookAround command has a 70% chance of being followed by a GoTo command and a 30% chance of being followed by Idle.

                  GoTo        Idle  LookAround
      GoTo           0         0.5         0.5
      Idle         0.6           0         0.4
LookAround         0.7         0.3           0

Note

The sum of probabilities for each row must be equal to 1.

To change the transition matrix, follow the steps below.

  1. click Tools on the menu bar, and then click Replicator > Custom Command to open the UI. In the UI window under “Custom Command Randomization” panel, you will see the current transition matrix.

  2. Edit each possibility entry as you need.

  3. Click Save Transition Matrix.

  4. Now IRA will use the new transition matrix for agent command randomization. Click Set Up Simulation from the “Agent SDG” window, and it will take effects.

Command Injection#

IRA allows you to inject commands into a running simulation to control the agents in real time. This enables you to adjust the simulation during runtime to create emergent events or simulate a particular scenario.

To enable the Command Injection UI window, click Tools on the menu bar, and then click Replicator and Command Injection.

Command injection works similar to regular commands. The format is agent_name command params and there is one panel for all the agents. This feature only works while the simulation is running. To inject commands, type the commands in the Command Injection UI textbox and then click the Inject button.

When a command is injected, the agent stops whatever command it is currently executing and starts the injected command immediately.

Note

  • If a character is performing the Sit command and a new command is injected, the character stands up first and then executes the injected command.

  • Command Injection only works with controlling the agents individually. Global command like Queue will not work.

Below is an image of the command injection UI. When clicking the Inject button, Character_01 will stop whatever it’s doing and execute the GoTo the world coordinate (1, 1, 0). Similarly, Nova_Carter_02 will stop whatever it’s doing and Idle for 10 seconds.

On this page