Stage 3: Configure and validate the locomotion MDP

The Stage 2 task registration tells AGILE which classes to load. This stage makes the loaded environment a coherent velocity-tracking learning problem for your robot. Work through every H2 *Cfg class even when you expect to keep its values: a term that resolves the wrong body, joint, sensor, or timing can train without an obvious configuration error.

This tutorial explains how the H2 configuration applies AGILE’s manager-based design. Use How Task Configs Compose MDP Components for the framework-wide pattern and MDP Components for the complete catalog. The sections below focus only on terms present in h2/velocity_history_env_cfg.py.

Understand the velocity-tracking problem

A Markov decision process (MDP) describes a sequential decision problem. At each control step, the policy receives an observation and produces an action. The simulator advances the robot, the environment measures the result, and the reward indicates how useful that transition was for the task.

                      command: desired vx, vy, yaw rate
                                   |
                                   v
observation history -------> [ policy ] -------> joint-position action
       ^                                             |
       |                                             v
       |                                      [ robot + terrain ]
       |                                             |
       +---- next state, reward, termination <-------+

For H2, the task samples desired planar velocity, the actor observes deployable robot state and five frames of history, and the action controls 14 leg and waist joints. Rewards shape velocity tracking, balance, safety, and motion quality. Terminations bound failed episodes, events vary the simulated experience, and curricula change difficulty as learning progresses.

The simulator’s complete state is larger than the actor observation. H2 uses asymmetric actor-critic training: the actor receives signals that deployment can reproduce, while the critic receives privileged simulator state to estimate value during training. Only the actor becomes the deployed policy.

Use these recommendation labels throughout this stage:

• Must change: The value names or models H2-specific anatomy or dynamics.

• Inspect: The H2 value is a useful starting point, but you must confirm its scale, range, and intent.

• Probably leave the same: Keep the value for the first validation run unless your platform or task gives a concrete reason to change it.

Note

The code excerpts in this stage are abbreviated. A ... stands for arguments that are present in the real configuration but omitted here to keep the relevant fields visible, so treat every excerpt as a reading guide rather than a file you can copy. Work from the AGILE sources named in each section.

MySceneCfg: Scene and sensors

Recommended action: Must change

Replace the robot configuration and every robot-relative prim path. Inspect the terrain and contact model.

The scene is the physical world in which the MDP runs. It is not an observation by itself; it supplies the robot, terrain, and sensors that observation, reward, termination, and event terms query. See MDP Components: Terrains for AGILE’s terrain choices.

@configclass
class MySceneCfg(InteractiveSceneCfg):
    terrain = TerrainImporterCfg(
        ...,
        terrain_type="generator",
        terrain_generator=MEDIUM_ROUGH_TERRAIN_CFG,
        max_init_terrain_level=1,
    )
    robot = unitree_h2.H2_DELAYED_DC_MOTOR.replace(prim_path="{ENV_REGEX_NS}/Robot")
    contact_forces = ContactSensorCfg(prim_path="{ENV_REGEX_NS}/Robot/.*", history_length=3, track_air_time=True)
    sky_light = AssetBaseCfg(...)
    height_measurement_sensor = RayCasterCfg(...)
    height_measurement_sensor_left_foot = RayCasterCfg(...)
    height_measurement_sensor_right_foot = RayCasterCfg(...)

H2 member	Recommendation	What to verify
terrain	Inspect	Start on a plane to isolate robot errors, then enable the H2 medium-rough terrain. Open the real configuration to check max_init_terrain_level and the physics_material friction and restitution values, which the excerpt above elides.
robot	Must change	Spawn the Stage 2 robot configuration at {ENV_REGEX_NS}/Robot.
contact_forces	Inspect	Cover every rigid body used by contact rewards or terminations and retain enough history for air-time or contact filtering.
sky_light	Probably leave the same	Lighting affects visualization, not this state-based policy.
height_measurement_sensor	Must change	Attach the ray caster to the custom base or pelvis path. The excerpt elides the ray caster’s arguments; open the real configuration to set offset and the pattern_cfg extent for your robot’s height.
height_measurement_sensor_left_foot	Must change	Replace the full H2 left-foot prim path. A path that resolves no prim silently invalidates dependent logic.
height_measurement_sensor_right_foot	Must change	Replace the full H2 right-foot prim path and keep its frame convention symmetric with the left sensor.

CommandsCfg: Desired behavior

Recommended action: Inspect

Begin with conservative velocities that your robot can track, then expand the range after a stable baseline.

Commands parameterize the task without changing its structure. The H2 policy is asked to track sampled forward, lateral, and yaw velocities. See MDP Components: Commands and its Velocity Commands section.

@configclass
class CommandsCfg:
    base_velocity = mdp.UniformNullVelocityCommandCfg(
        asset_name="robot",
        resampling_time_range=(8.0, 12.0),
        rel_standing_envs=0.25,
        rel_heading_envs=1.0,
        heading_command=False,
        debug_vis=True,
        ranges=mdp.UniformNullVelocityCommandCfg.Ranges(
            lin_vel_x=(-0.5, 0.5),
            lin_vel_y=(-0.5, 0.5),
            ang_vel_z=(-1.0, 1.0),
        ),
    )

resampling_time_range keeps each target active for 8 to 12 s, while rel_standing_envs=0.25 reserves a quarter of environments for learning zero-command balance. Keep debug_vis=True during validation so the target arrow remains visible.

Important

The ranges block defines the task’s training envelope. H2 samples forward and lateral velocity from -0.5 to 0.5 m/s and yaw velocity from -1.0 to 1.0 rad/s. Positive and negative bounds make the policy practice both directions. These are desired velocities, not actuator limits, and the trained policy is not expected to generalize reliably beyond them. Start a custom robot with a conservative envelope, then expand these three ranges only after it tracks the initial commands without falling.

ActionsCfg: Policy control interface

Recommended action: Must change

Replace the controlled-joint set, verify its resolved order, and choose a safe position-action scale.

Actions translate policy output into commands for the simulated robot. See MDP Components: Actions, Joint Position Actions, and Random Actions.

CONTROLLED_JOINT_NAMES = unitree_h2.LEG_JOINT_NAMES + ["waist_roll_joint", "waist_pitch_joint"]


@configclass
class ActionsCfg:
    joint_pos = mdp.JointPositionActionCfg(
        ...,
        joint_names=CONTROLLED_JOINT_NAMES,
        scale=0.5,
        use_default_offset=True,
    )
    random_pos = mdp.RandomActionCfg(
        ...,
        joint_names_exclude=unitree_h2.LEG_JOINT_NAMES + unitree_h2.WAIST_JOINT_NAMES,
    )

H2 member	Recommendation	What it does
CONTROLLED_JOINT_NAMES	Must change	Defines the action width and joint order. H2 uses 12 leg joints plus waist roll and pitch.
joint_pos	Must change	Interprets policy output as offsets from the default pose. Verify joint resolution, scale, offset, and clipping before training.
random_pos	Inspect	Moves joints outside the locomotion policy with a trapezoidal profile. For H2, this varies the upper body without consuming policy output; remove or retarget it if those joints must remain fixed or policy-controlled.

The trained model’s output width equals the resolved joint_pos action width. Preserve this order through observation construction, symmetry augmentation, export, and deployment.

ObservationsCfg: Actor and critic inputs

Recommended action: Inspect

Preserve deployable actor signals, replace controlled joints, and deliberately choose the history length.

Observations are the information available for decision making. See MDP Components: Observations, Observation Groups, and History Stacking.

@configclass
class ObservationsCfg:
    @configclass
    class HistoryPolicyCfg(ObsGroup):
        base_ang_vel = ObsTerm(...)
        projected_gravity = ObsTerm(...)
        velocity_commands = ObsTerm(func=mdp.generated_commands, params={"command_name": "base_velocity"})
        controlled_joint_pos = ObsTerm(
            func=mdp.joint_pos_rel,
            params={"asset_cfg": SceneEntityCfg("robot", joint_names=CONTROLLED_JOINT_NAMES)},
        )
        controlled_joint_vel = ObsTerm(...)
        actions = ObsTerm(...)

        def __post_init__(self):
            self.history_length = 5
            self.enable_corruption = True
            ...

    @configclass
    class PrivilegedVelocityCriticCfg(ObsGroup): ...

    policy: HistoryPolicyCfg = HistoryPolicyCfg()
    critic: PrivilegedVelocityCriticCfg = PrivilegedVelocityCriticCfg()

H2 actor term	Recommendation	Purpose
base_ang_vel	Inspect	Gives the actor IMU-like angular velocity. Confirm the scale, noise, frame, and deployability.
projected_gravity	Probably leave the same	Encodes body orientation relative to gravity without requiring a global heading.
velocity_commands	Probably leave the same	Tells the actor which planar velocity to track. Its command_name must match CommandsCfg.
controlled_joint_pos	Must change	Provides positions relative to the default pose in the exact controlled-joint order.
controlled_joint_vel	Must change	Provides controlled-joint velocity. Confirm scale and noise against the custom sensing pipeline.
actions	Probably leave the same	Supplies the previous policy action, which helps the policy model control dynamics and smoothness.
history_length=5	Inspect	Gives H2 temporal context. More history increases input width and deployment state; less history removes information the H2 design expects.
enable_corruption=True	Inspect	Applies actor observation noise during training. Align noise with plausible sensor uncertainty.

The critic group includes velocity_commands, base_lin_vel, base_ang_vel, projected_gravity, all joint_pos and joint_vel, and previous actions. These privileged terms improve value estimation but must not leak into the actor group. Keep enable_corruption=False for the critic baseline. Verify that every actor term can be recreated by the target deployment runtime and that actor ordering is stable.

RewardsCfg: Learning objective and shaping

Recommended action: Must change

Replace robot-specific bodies, joints, heights, and stance values. Keep weights close to H2 for the first training run, then tune from term-level metrics rather than total reward alone.

Rewards turn desired behavior and constraints into the scalar learning signal optimized by PPO. See MDP Components: Rewards, especially Tracking Rewards, Aesthetic Rewards, and Regularization Rewards. The complete H2 reward structure follows, with every active term listed.

@configclass
class RewardsCfg:
    termination_penalty = RewTerm(...)
    track_lin_vel_xy_exp = RewTerm(...)
    track_ang_vel = RewTerm(...)
    base_height = RewTerm(...)
    orientation = RewTerm(...)
    torques = RewTerm(...)
    ankle_torques = RewTerm(...)
    ankle_roll_torques = RewTerm(...)
    lin_vel_z = RewTerm(...)
    ang_vel_xy = RewTerm(...)
    dof_vel = RewTerm(...)
    action_rate = RewTerm(...)
    action_rate_rate = RewTerm(...)
    dof_pos_limits = RewTerm(...)
    dof_vel_limits = RewTerm(...)
    torque_limits = RewTerm(...)
    feet_slip = RewTerm(...)
    feet_roll = RewTerm(...)
    feet_yaw_diff = RewTerm(...)
    feet_yaw_mean = RewTerm(...)
    root_acc = RewTerm(...)
    feet_distance = RewTerm(...)
    jumping = RewTerm(...)

Start with the basic task rewards, which cover command tracking, survival, balance, joint and actuator limits, and foot contact. Confirm that the policy can learn stable command tracking with this smaller objective before adding specialized shaping terms.

Basic task rewards

H2 reward	Recommendation	Why it exists
termination_penalty	Inspect	Makes falls and other failure terminations costly. Confirm that it is not applied to ordinary timeouts.
track_lin_vel_xy_exp	Probably leave the same	Primary task signal for forward and lateral command tracking. Inspect the error width std when the custom robot cannot initially earn a gradient.
track_ang_vel	Must change	Primary yaw-rate tracking signal. Replace the H2 pelvis body and verify its frame.
base_height	Must change	Encourages the robot’s intended standing height. Replace the H2 pelvis height and sensor.
orientation	Must change	Encourages the pelvis and torso to remain upright. Replace both body names and inspect the tolerance.
torques	Must change	Penalizes effort across controlled joints. Persistent large values point to gains, scale, pose, or task demands that are too aggressive.
lin_vel_z	Probably leave the same	Penalizes vertical base motion that does not serve planar tracking.
ang_vel_xy	Probably leave the same	Penalizes base roll and pitch rate, exposing oscillation and loss of balance.
dof_vel	Must change	Penalizes controlled-joint speed. Confirm its joint set and avoid suppressing necessary gait motion.
action_rate	Must change	Penalizes changes between consecutive actions. It is the first smoothness term to inspect for abrupt motion.
dof_pos_limits	Must change	Penalizes approaches to custom joint-position limits.
dof_vel_limits	Must change	Penalizes speeds beyond a soft fraction of the custom velocity limits.
torque_limits	Must change	Penalizes applied torque beyond actuator limits and exposes frequent saturation.
feet_slip	Must change	Penalizes tangential foot motion during contact. Replace both foot-body selections and inspect the contact threshold.

After the basic configuration trains successfully, add specialized rewards gradually to refine actuator use, stance, and motion quality. Introduce one group at a time and verify that it improves the intended behavior without reducing command tracking or stability. These H2 terms are starting points, not requirements for every humanoid.

Specialized rewards

H2 reward	Recommendation	Specialized purpose
ankle_torques	Must change	Adds stronger regularization to all H2 ankle joints.
ankle_roll_torques	Must change	Adds an even stronger H2 ankle-roll penalty for its low-torque actuator design.
action_rate_rate	Must change	Penalizes changes in action rate, shaping higher-order smoothness after basic control is stable.
feet_roll	Must change	Discourages feet from rolling away from a flat contact orientation.
feet_yaw_diff	Must change	Discourages excessive relative yaw between the feet.
feet_yaw_mean	Must change	Aligns mean foot yaw with the pelvis; useful for H2 stance aesthetics and direction consistency.
root_acc	Inspect	Penalizes whole-body acceleration as a broad motion-quality regularizer.
feet_distance	Must change	Encourages H2’s 0.2 m reference stance width. Replace it with a kinematically appropriate value.
jumping	Must change	Penalizes both feet losing contact when commanded locomotion should remain grounded. Inspect the force threshold for the custom mass and contact model.

Reward scales interact. A large penalty can make standing still or terminating early more profitable than tracking. During diagnosis, log every term, verify its sign and magnitude, and temporarily disable specialized terms when they obscure the basic task objective. Training Tips: Reward Recipe provides the companion tuning guidance.

TerminationsCfg: Episode boundaries

Recommended action: Must change

Replace H2 body names, height thresholds, and contact assumptions before running the environment.

Terminations end transitions that no longer provide useful locomotion experience. They are part of the MDP: changing them changes which future rewards the agent can collect. See MDP Components: Terminations.

@configclass
class TerminationsCfg:
    time_out = DoneTerm(func=mdp.time_out, time_out=True)
    base_orientation = DoneTerm(...)
    illegal_contacts = DoneTerm(...)
    illegal_base_height = DoneTerm(...)

H2 term	Recommendation	What to verify
time_out	Probably leave the same	Marks the configured episode horizon as a timeout rather than a failure.
base_orientation	Must change	Replace torso_link and choose an angle that detects an unrecoverable fall without ending recoverable balance errors too early.
illegal_contacts	Must change	Replace pelvis and torso contact bodies, force threshold, and minimum height.
illegal_base_height	Must change	Replace the pelvis body, height sensor, and threshold derived from the custom standing height.

Log which term ends each episode. If episodes end immediately, diagnose term resolution and thresholds before changing rewards.

ViewerCfg: Diagnostic camera

Recommended action: Probably leave the same

Change the camera only when the custom robot’s scale or orientation makes validation difficult.

The viewer configuration does not change the MDP or learned policy. It selects the camera pose, resolution, tracked asset, and environment index used for human inspection. The class appears in the same environment composition described by How Task Configs Compose MDP Components.

@configclass
class ViewerCfg:
    eye: tuple[float, float, float] = (0.0, -5.0, 2.0)
    lookat: tuple[float, float, float] = (0.0, 0.0, 0.5)
    cam_prim_path: str = "/OmniverseKit_Persp"
    resolution: tuple[int, int] = (1280, 720)
    origin_type = "asset_root"
    asset_name: str = "robot"
    env_index: int = 0

Keep asset_name="robot" consistent with the scene. Use env_index to follow a reproducible environment while investigating contacts, resets, or command arrows.

LocomotionEventCfg: Resets, disturbances, and randomization

Recommended action: Inspect

Replace H2 body patterns and scale every randomization or disturbance range to credible custom-robot values.

Events modify the environment at startup, reset, or scheduled intervals. Randomization reduces dependence on one exact simulation, while resets and disturbances broaden the states from which the policy must recover. See MDP Components: Events, Reset Events, and Randomization Events.

@configclass
class LocomotionEventCfg:
    # Startup randomization
    randomize_physics_material = EventTerm(...)
    randomize_actuator_gains = EventTerm(...)
    randomize_joint_friction = EventTerm(...)
    randomize_joint_armature = EventTerm(...)
    randomize_bodies_mass = EventTerm(...)
    randomize_base_mass = EventTerm(...)
    randomize_bodies_com = EventTerm(...)
    randomize_base_com = EventTerm(...)

    # Scheduled disturbances
    apply_external_force_torque = EventTerm(...)
    apply_external_force_torque_extremities = EventTerm(...)
    push_robot = EventTerm(...)

    # Episode resets
    reset_base = EventTerm(...)
    reset_robot_joints = EventTerm(...)

H2 event	Recommendation	What to inspect
randomize_physics_material	Inspect	Use plausible static and dynamic friction and restitution ranges.
randomize_actuator_gains	Inspect	Keep gain scales near the measured actuator model for the first run.
randomize_joint_friction	Must change	Match the custom actuator model and units.
randomize_joint_armature	Must change	Scale around credible custom armature values; H2’s range is not morphology-independent.
randomize_bodies_mass	Inspect	Use uncertainty supported by the asset or hardware measurements.
randomize_base_mass	Must change	Replace pelvis and scale additive mass to the robot or payload use case.
randomize_bodies_com	Inspect	Keep per-body center-of-mass offsets physically plausible.
randomize_base_com	Must change	Replace pelvis and reduce H2’s asymmetric range unless the payload distribution justifies it.
apply_external_force_torque	Must change	Replace the pelvis body and scale forces and torques to robot mass and recovery goals.
apply_external_force_torque_extremities	Must change	Replace H2 wrist and ankle patterns and remove bodies not present on the custom robot.
reset_base	Must change	Check pose and velocity ranges, terrain bounds, and the H2 foot-body pattern used by the reset function.
reset_robot_joints	Inspect	Confirm that scaling the default pose by 0.8 to 1.2 creates valid, collision-free poses.
push_robot	Inspect	Begin with smaller velocity impulses if the untrained robot terminates immediately.

Validate without aggressive randomization first when debugging structure. Re-enable one category at a time so you can identify whether failures come from the nominal model or the robustness envelope. See Training Tips: Domain Randomization for broader guidance.

CurriculumCfg: Changing difficulty during learning

Recommended action: Probably leave the same

Keep the progression structure initially, but verify its distances, success counts, schedules, and reward names.

A curriculum changes the distribution or objective as the policy improves. See MDP Components: Curriculum and Terrain Curriculum.

@configclass
class CurriculumCfg:
    terrain_levels = CurrTerm(...)
    increase_action_rate_regularization = CurrTerm(...)
    increase_action_rate_rate_regularization = CurrTerm(...)

terrain_levels uses 4.0 m and 2.0 m movement thresholds, requiring four successes to advance and ten failures to regress. increase_action_rate_regularization starts at step 50,000 and ramps its target weight to -1.0 over 100,000 steps. The second smoothness curriculum starts at step 60,000 and ramps action_rate_rate to -0.5 over the same duration. Confirm that both reward names exactly match RewardsCfg. Disable terrain curriculum for a flat-terrain structural baseline.

H2LowerVelocityHistoryEnvCfg: Compose and time the environment

Recommended action: Must change

Rename the class and verify control frequency, physics frequency, episode duration, sensor periods, and evaluation behavior.

The top-level class assembles every manager configuration and fixes the timing relationship between the policy and simulator. This is the concrete H2 application of How Task Configs Compose MDP Components.

@configclass
class H2LowerVelocityHistoryEnvCfg(ManagerBasedRLEnvCfg):
    scene: MySceneCfg = MySceneCfg(num_envs=4096, env_spacing=2.5)
    observations: ObservationsCfg = ObservationsCfg()
    actions: ActionsCfg = ActionsCfg()
    commands: CommandsCfg = CommandsCfg()
    viewer: ViewerCfg = ViewerCfg()
    rewards: RewardsCfg = RewardsCfg()
    terminations: TerminationsCfg = TerminationsCfg()
    events: LocomotionEventCfg = LocomotionEventCfg()
    curriculum: CurriculumCfg = CurriculumCfg()

    def __post_init__(self):
        self.controller_freq = 50.0
        self.physics_freq = 200.0
        self.episode_length_s = 30.0
        self.max_episode_length_offset_s = 0.0
        self.decimation = int(self.physics_freq / self.controller_freq)
        self.sim.dt = 1.0 / self.physics_freq
        self.sim.render_interval = self.decimation
        self.sim.physics_material = self.scene.terrain.physics_material
        ...
        self.scene.contact_forces.update_period = self.sim.dt
        self.scene.height_measurement_sensor.update_period = self.sim.dt
        self.scene.height_measurement_sensor_left_foot.update_period = self.sim.dt
        self.scene.height_measurement_sensor_right_foot.update_period = self.sim.dt
        self.only_positive_rewards = False
        ...

    def eval(self):
        self.observations.eval = mdp.EvaluationObservationsCfg()
        self.rewards = None
        self.curriculum = None

At 50 Hz control and 200 Hz physics, one policy action spans four physics steps. Actuator delay is specified in physics steps, so changing either frequency changes the delay represented in seconds. Sensor update periods must remain synchronized with the quantities their MDP terms consume.

When AGILE switches to evaluation mode, eval() disables training-only rewards and curricula and adds a separate observation group for evaluation metrics. It does not replace the observations consumed by the policy. Keep the actor observation fields, history length, action count, and ordering identical to training so that the trained and exported policies still match their inputs and outputs.

Validate the environment

Run the environment with generated test actions before training. This does not require a trained policy:

uv run scripts/play.py --task Velocity-H2-History-v0 --num_envs 2 --viz kit

For a custom humanoid, replace Velocity-H2-History-v0 with the task ID registered in Stage 2. Check resets, joint directions, action scaling, contacts, observation shapes, and terminations. Fix environment errors before you tune rewards.

The play script loads the environment without an RL policy, removes training-only components, and sends smooth sinusoidal actions to controlled joints. The joints use staggered phase offsets and a range of frequencies. This is a smoke test for joint mapping, action scaling, simulation stability, and resets; it is not a test of learned locomotion.

The green arrow shows the sampled target linear velocity, and the blue arrow shows the robot’s measured linear velocity. Arrow direction indicates travel direction and arrow length indicates speed. Because no trained policy is tracking the command, the arrows are not expected to align during this test.

Troubleshooting by layer

Symptom	Likely layer	Check
The task fails during manager initialization.	Entity resolution.	Check every robot, joint, body, sensor, action, command, and reward-name reference for stale H2 names.
The wrong joints move or action width is unexpected.	ActionsCfg.	Print the resolved CONTROLLED_JOINT_NAMES order and check overlapping actuator or random-action groups.
Actor observation width changes or deployment cannot reproduce a term.	ObservationsCfg.	Check controlled-joint resolution, history shape, concatenation settings, and actor-versus-critic placement.
Environments terminate immediately.	TerminationsCfg or reset events.	Log the terminating term, inspect body paths and thresholds, and verify the reset pose before changing rewards.
Reward is large but target arrows do not align after training begins.	RewardsCfg.	Compare individual reward metrics and look for a posture, termination, or regularizer that dominates tracking.
The nominal environment works but randomized runs fail instantly.	LocomotionEventCfg.	Re-enable randomizations one group at a time and scale ranges to measured uncertainty.
Motion is stable but too slow or oscillatory.	Timing, actions, or actuators.	Compare control rate, physics rate, decimation, delay, action scale, stiffness, damping, and saturation.

Completion checklist

• Every H2 *Cfg class has been reviewed for the custom robot.

• Every joint, body, sensor, command, action, and reward-name reference resolves as intended.

• Actor observations are deployable, ordered, and fixed-width; privileged signals remain critic-only.

• All 23 reward terms have an explicit reason to keep, adapt, or disable them.

• Terminations produce credible reasons and do not end valid initial states.

• The play script starts reliably, resets cleanly, and moves only the intended joints before training begins.

• Control, physics, sensor, actuator-delay, and episode timing are internally consistent.

• The environment runs repeatedly without errors and actions affect only their intended joints.

Continue with Stage 4: Train, evaluate, and export.