Is this page helpful?

Humanoid workflow: Train locomotion in AGILE and deploy in Isaac Sim#

Use this five-stage tutorial series to bring a custom humanoid from a simulation-ready USD asset to a trained reinforcement learning (RL) locomotion policy and a deployment application. The series uses the Unitree H2 as the reference robot, AGILE as the reinforcement learning training framework, and Isaac Sim or Isaac ROS as the deployment framework.

The stages separate the asset, task wiring, Markov decision process (MDP), training, and deployment layers. Complete them in order. At the end of each stage, use the troubleshooting table and completion checklist to identify problems before they become harder to diagnose in the next layer.

Unitree H2 walking through a warehouse under reinforcement learning policy control

Learning objectives#

By the end of this series, you can:

  • Prepare and validate a humanoid USD for locomotion training.

  • Add a custom humanoid and velocity-tracking task to AGILE.

  • Explain how AGILE composes an MDP from commands, actions, observations, rewards, terminations, events, and curricula.

  • Adapt every H2 environment configuration to another humanoid.

  • Train a policy, interpret its learning curves, evaluate it, and export its artifacts.

  • Deploy a policy through standalone Python in Isaac Sim or Isaac ROS Deploy.

  • Identify whether a failure comes from the asset, task, MDP, training, or deployment layer.

Prerequisites#

  • Use a Linux workstation with a compatible NVIDIA GPU, as required by AGILE.

  • Obtain a humanoid URDF or USD with complete visual meshes, collision meshes, inertial properties, and joint limits.

  • Learn the basic USD rigid-body, joint, drive, and articulation concepts in Articulate a Basic Robot.

  • Review the Isaac Lab overview and its introduction to manager-based RL environments. AGILE builds on these Isaac Lab concepts.

  • Understand the basic goal of RL training: improve a policy by collecting experience and maximizing expected cumulative reward.

Important

The commands and paths in this series use the Unitree H2 and the Velocity-H2-History-v0 task. For another humanoid, replace the robot asset, joint and link names, actuator parameters, task registration, controlled joints, MDP terms, runner configuration, and deployment integration. Replacing only the H2 USD does not adapt the workflow to another robot.

Common terms to know#

Term

Meaning

USD

Universal Scene Description, the format that stores the robot’s geometry, joints, physical properties, and other scene data.

Articulation

A set of rigid bodies connected by joints and simulated as one mechanism.

AGILE

The framework used in this tutorial to define humanoid tasks, train policies, and evaluate them.

Markov decision process (MDP)

The learning problem defined by observations, actions, rewards, commands, terminations, events, and curricula.

Observation and action

An observation is information provided to the policy; an action is the command the policy returns to the robot.

Actor and critic

The actor produces policy actions, while the critic estimates their long-term value during training.

Gymnasium task ID

The registered name that selects an environment and its training configuration, such as Velocity-H2-History-v0.

RSL-RL and PPO

RSL-RL is a reinforcement learning library used in AGILE, and proximal policy optimization (PPO) is the algorithm used to train the locomotion policy.

Tutorial stages#

Stage

Goal

Why it comes here

Continue when

Stage 1

Prepare the humanoid USD.

The training configuration can only be correct when it describes the same physical robot as the USD.

The floating-base articulation and its recorded configuration values are trustworthy.

Stage 2

Create a custom humanoid task in AGILE.

Registration connects the robot, environment, and learning runner before you design task behavior.

AGILE resolves the custom task ID to the intended robot, environment, and runner configurations.

Stage 3

Configure and validate the locomotion MDP.

The MDP defines what the policy can sense and do, what behavior learning values, and when episodes end.

The environment passes deterministic structural and play-script checks.

Stage 4

Train, evaluate, and export the policy.

Training is useful only after the robot and MDP are internally consistent.

Evaluation demonstrates command tracking and the run contains deployable artifacts.

Stage 5

Deploy the policy.

Deployment must reproduce the observations, actions, actuators, and timing used during training.

The selected runtime reproduces the expected simulated locomotion behavior.

Tutorial pages#

  1. Stage 1: Prepare the humanoid USD

  2. Stage 2: Create a custom humanoid task in AGILE

  3. Stage 3: Configure and validate the locomotion MDP

  4. Stage 4: Train, evaluate, and export

  5. Stage 5: Deploy the policy