Reinforcement Learning

Reinforcement Learning#

This section introduces the overall architecture and submodules of the embodichain RL (Reinforcement Learning) module. The RL framework supports mainstream algorithms (such as PPO) and provides flexible components for policy, buffer, trainer, etc., making it easy to extend and customize.

Overview#

The embodichain RL module is used to train agents to accomplish tasks in simulation environments. It mainly includes algorithm implementations, policy networks, data buffers, training processes, and utility tools.

Training-time evaluation measures learning progress when enabled in the trainer. After training, embodichain eval-policy reloads a saved checkpoint and reports completed-episode reward, length, success signals and task metrics. Qualification compares those measurements with criteria for the task and deployment. See Policy Evaluation for a headless locomotion example and qualification criteria. The bundled locomotion PPO configurations disable training-time evaluation, so use saved-checkpoint evaluation to assess a trained policy.

Architecture Diagram Example#

  +-------------------+
  |   train.py        |
  +-------------------+
           |
           v
  +-------------------+
  | Trainer           |
  +-------------------+
    |    |      |   |
    v    v      v   v
Algo  Policy Buffer Env

  • train.py is responsible for entry, config parsing, and module initialization.

  • Trainer coordinates algorithm, policy, buffer, and environment.

  • Algo/Policy/Buffer/Env are independent, making extension easy.

Module Categories#

  • Algorithm (algo/): RL algorithm implementations, including BaseAlgorithm, PPO, etc.

  • Buffer (buffer/): Trajectory data buffer, such as RolloutBuffer.

  • Models (learning/rl/policies/): RL policy interfaces, actor-critic models and policy builders. Generic MLPs and normalization live in learning/models/.

  • Trainer (utils/trainer.py): Main training loop and logging management.

  • Config (utils/config.py): Algorithm config class definitions.

  • Train Script (train.py): RL training entry script.

Extension and Customization#

  • Users can customize algorithms (by inheriting BaseAlgorithm), policies (by inheriting Policy), buffers, etc.

  • Supports multi-environment parallelism, event-driven extension, and flexible config management.

  • It is recommended to manage all parameters via config files for reproducibility and batch experiments.

Common Issues and Best Practices#

  • Config files may use JSON or YAML for easy management and reproducibility.

  • Parallel environment sampling can significantly improve training efficiency.

  • The event-driven mechanism allows flexible insertion of custom logic (such as evaluation, saving, callbacks).

  • It is recommended to use WandB/TensorBoard for training process visualization.

Example#

embodichain train-rl --config embodichain_tasks/configs/tasks/classic_control/cart_pole/agents/ppo.yaml

For more details, please refer to the source code and API documentation of each submodule.

See also#

See Also#

Contents