Stage 4: Train, evaluate, and export

Training now optimizes a policy against the robot and MDP validated in Stages 1–3. Keep the first run a baseline: changing the environment, rewards, network, and PPO settings at the same time makes failures difficult to attribute.

Train the policy

Run the following command from the AGILE repository. For another humanoid, replace the task ID and run name. See Training Guide: Training for the complete command interface and Hydra Overrides for targeted configuration experiments.

uv run scripts/train.py \
    --task Velocity-H2-History-v0 \
    --num_envs 4096 \
    --max_iterations 50000 \
    --logger tensorboard \
    --run_name h2_full_50000 \
    --headless \
    --viz none

AGILE creates a timestamped run under:

logs/rsl_rl/velocity_h2_history/<TIMESTAMP>_h2_full_50000/

The timestamp identifies one immutable experiment. Note the full run directory printed in the terminal; use the same directory for evaluation, export, and deployment. Do not combine a checkpoint with configuration artifacts from another run.

Monitor training with TensorBoard

TensorBoard is included in AGILE’s locked environment. Open another terminal in the AGILE repository and run:

uv run tensorboard --logdir logs/rsl_rl/velocity_h2_history --port 6006

Open http://localhost:6006 in a browser. Use Training Tips: Key Plots to Watch as the interpretation guide. For H2, check that:

• track_lin_vel_xy_exp and track_ang_vel improve while episodes approach the 30 s limit.

• termination_penalty decreases instead of dominating the reward.

• feet_slip, action_rate, and torque_limits remain small as the terrain level increases.

A rising total reward alone does not prove that the robot learned the intended behavior. Pair metrics with periodic visual evaluation, and change one hypothesis-driven variable per comparison run.

Evaluate the trained policy

Run the AGILE evaluation script with one environment and the Kit visualizer. Replace RUN_DIRECTORY_NAME with the timestamped directory name created during training:

uv run scripts/eval.py \
    --task Velocity-H2-History-v0 \
    --load_run RUN_DIRECTORY_NAME \
    --num_envs 1 \
    --num_steps 10000 \
    --real-time \
    --viz kit

The evaluation script loads the newest matching checkpoint when you do not specify one. Supply an explicit checkpoint during comparisons so that run selection cannot change the result. See Evaluation: Isaac Lab Evaluation for deterministic scenarios, trajectory capture, metrics, and reports.

The green arrow shows sampled target linear velocity and the blue arrow shows measured linear velocity. The H2 task samples forward and lateral velocity from -0.5 to 0.5 m/s and yaw angular velocity from -1.0 to 1.0 rad/s, resampling every 8–12 seconds; 25% of commands request a standstill. The learned policy controls 14 joints. Its separate randomized action moved the upper body during training, so that motion is not part of the policy output.

Test positive and negative forward velocity, lateral velocity, yaw velocity, zero command, resets, ordinary disturbances, and expected terrain. Use deterministic schedules when comparing checkpoints so each policy receives the same test.

Inspect and export the run

When evaluation loads a raw RSL-RL checkpoint, it exports TorchScript and ONNX policies under the run’s exported/ directory. AGILE also provides explicit export commands in Training Guide: Policy Export.

Keep these artifacts from the same run:

<RUN_DIRECTORY>/
|-- params/
|   |-- env.yaml
|   `-- agent.yaml
`-- exported/
    `-- policy.pt

env.yaml records the environment and deployment-relevant robot settings. agent.yaml records the learning setup. policy.pt contains the trained actor. Stage 5 adds the IO descriptor or LEAPP bundle needed by the selected deployment path.

Troubleshooting by layer

Symptom	Likely layer	Check
Episode length remains near zero.	Environment or termination.	Return to Stage 3 and inspect reset validity and per-term termination counts before tuning PPO.
Total reward rises but tracking error does not improve.	Reward design.	Compare individual rewards for exploitation or a posture and regularization objective that dominates tracking.
Policy noise grows rapidly and motion becomes erratic.	Exploration or scaling.	Check entropy, action scale, observation scale, actuator gains, and the AGILE policy-noise guidance.
Value loss is unstable.	Critic or reward scale.	Check privileged critic observations, reward magnitudes, learning rate, and invalid numerical values.
One run behaves differently after resume.	Run selection or configuration drift.	Confirm the checkpoint, task, environment YAML, agent YAML, and overrides all come from the intended run.
Evaluation loads the wrong checkpoint.	Evaluation arguments.	Set the checkpoint explicitly instead of relying on newest-match selection.
Training looks successful but visual motion is poor.	Objective mismatch.	Test deterministic commands and compare tracking, contacts, limits, smoothness, and termination metrics.

Completion checklist

• The baseline run uses the Stage 3 environment without unrelated simultaneous changes.

• Episode length and task-tracking metrics improve, not only total reward.

• Policy noise, entropy, value loss, and curricula remain interpretable and stable.

• Visual evaluation covers all command directions, standstill, resets, disturbances, and expected terrain.

• The selected checkpoint is identified explicitly.

• env.yaml, agent.yaml, and policy.pt come from the same timestamped run.

• The evaluated policy tracks commands and recovers from ordinary disturbances well enough to deploy.

Continue with Stage 5: Deploy the policy.