nico-bohlinger/RL-X: A framework for Reinforcement Learning research.

A framework for Reinforcement Learning research.

│ Overview │ Getting Started │ Documentation │ Citation │

Overview

Highlights

• 💡 Perfect to understand and prototype algorithms:

• ⚒️ Known DL libraries: Implementations in PyTorch and JAX
• ⚡ Maximum speed: Just-In-Time (JIT) compilation and parallel environments
• 🧪 Mix and match and extend: Generic interfaces between algorithms and environments
• ⛰️​ Custom environments: Examples for MuJoCo, Isaac Lab, ManiSkill or custom socket communication
• 🚀​ GPU environments: MJX, Warp, Isaac Lab and ManiSkill can run thousands of parallel environments
• 🤖​ Robot learning: Training and deployment for locomotion and motion tracking with the G1 and Go2
• ⚽ RoboCup: Training for the RoboCup soccer competition in MuJoCo and MJX
• 🕰️ Memory architectures: PPO with GRU, LSTM, Transformer, History Window, Mamba-2, Memory Actions
• 📈 Experiments: Checkpoints, Evaluation, Console log, Tensorboard, Weights & Biases, SLURM, Docker

Implemented Algorithms

• Proximal Policy Optimization (PPO)
• Proximal Policy Optimization + Differentiable Trust Region Layers (PPO+DTRL)
• Proximal Policy Optimization + Gated Recurrent Unit (PPO+GRU)
• Proximal Policy Optimization + Long Short-Term Memory (PPO+LSTM)
• Proximal Policy Optimization + Transformer (PPO+Transformer)
• Proximal Policy Optimization + History Window (PPO+HistoryWindow)
• Proximal Policy Optimization + Mamba-2 (PPO+Mamba-2)
• Proximal Policy Optimization + Memory Actions (PPO+MemoryActions)
• Early Stopping Policy Optimization (ESPO)
• Trust Region Policy Optimization (TRPO)
• Simple Policy Optimization (SPO)
• Flow Policy Optimization (FPO)
• Diffusion Policy Policy Optimization (DPPO)
• Diffusion Models for Maximum Entropy Reinforcement Learning (DIME)
• Relative Entropy Pathwise Policy Optimization (REPPO)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed Deep Deterministic Gradient (TD3)
• Fast Twin Delayed Deep Deterministic Gradient (FastTD3)
• Soft Actor Critic (SAC)
• Fast Soft Actor Critic (FastSAC)
• Flash Soft Actor Critic (FlashSAC)
• Randomized Ensembled Double Q-Learning (REDQ)
• Dropout Q-Functions (DroQ)
• Bigger, Regularized, Optimistic (BRO)
• CrossQ
• XQC
• Simplicity Bias (SimBa)
• Simplicity Bias V2 (SimBaV2)
• Truncated Quantile Critics (TQC)
• Aggressive Q-Learning with Ensembles (AQE)
• Maximum a Posteriori Policy Optimization (MPO)
• Fast Maximum a Posteriori Policy Optimization (FastMPO)
• Deep Q-Network (DQN)
• Deep Q-Network with Histogram Loss using Gaussians (DQN HL-Gauss)
• Double Deep Q-Network (DDQN)
• Categorical Deep Q-Network (C51)
• Parallelized Q-Network (PQN)
• Generative Adversarial Imitation Learning (GAIL)
• Adversarial Inverse Reinforcement Learning (AIRL)
• Adversarial Motion Priors (AMP)
• Noise Conditioned Energy Based Annealed Rewards (NEAR)
• Trust Region Inverse Reinforcement Learning + Trust Region Loss (TRIRL+PPO)
• Trust Region Inverse Reinforcement Learning + Differentiable Trust Region Layers (TRIRL+DTRL)

Usable Environments

• Gymnasium

• EnvPool

• MuJoCo Playground

• Custom MuJoCo

• Custom Isaac Lab

• Custom ManiSkill

• Custom Interface

All listed environments are directly embedded in RL-X and can be used out-of-the-box.

For further information on the environments (README) and algorithms (README) and how to add your own, read the respective README files.

Getting Started

Install

conda create -n rlx python=3.11.4
conda activate rlx
git clone [email protected]:nico-bohlinger/RL-X.git
cd RL-X
pip install -e .[all] --config-settings editable_mode=compat
pip uninstall $(pip freeze | grep -E '\-cu12|\-cu13' | cut -d '=' -f 1) -y
pip install "torch>=2.7.0" --index-url https://download.pytorch.org/whl/cu118 --upgrade
pip install "jax[cuda12]"

Example

cd experiments
python experiment.py

Example for Google Colab:

@incollection{bohlinger2023rlx,
      title={RL-X: A Deep Reinforcement Learning Library (not only) for RoboCup},
      author={Nico Bohlinger and Klaus Dorer},
      booktitle={Robot World Cup},
      pages={228--239},
      year={2023},
      publisher={Springer}
}