Puffer PHC

This is a simplified version of the Perpetual Humanoid Control repository. The class inheritance and dependencies have been greatly reduced, and PufferLib is used to train the policy. Thank you to Zhengyi Luo for the awesome research and open sourcing the code.

This project was sponsored by Puffer AI. If you like what you see, please contact Puffer AI for priority service.

This repo implements only the single primitive model, which can perform 99.0% of the AMASS 11313 motions. See the wandb log for training details. The AMASS checkpoints are shared in the onedrive folder.

Getting Started

The default path trains on the Newton/MuJoCo backends (Python 3.12, no Isaac Gym required). Isaac Gym is still supported as a legacy backend; see Legacy: Isaac Gym below.

1. Clone this repository.

git clone https://github.com/kywch/puffer-phc.git

cd puffer-phc

1. Using pixi, setup the virtual environment and install the dependencies. Install pixi, if you haven't already. See pixi documentation for more details. The following command is for linux.

curl -fsSL https://pixi.sh/install.sh | bash

The following command installs the dependencies and activates the virtual environment:

pixi shell

It runs two backends: mujoco_gpu (mujoco-warp, batched on GPU, much faster) for training, and mujoco_cpu (MuJoCo C, one solver per env — not batched, so keep --env.num-envs small) for eval and debugging rather than as a training alternative.

1. Download the SMPL parameters from SMPL, and unzip them into smpl folder. Rename the files basicmodel_neutral_lbs_10_207_0_v1.1.0, basicmodel_m_lbs_10_207_0_v1.1.0.pkl, basicmodel_f_lbs_10_207_0_v1.1.0.pkl to SMPL_NEUTRAL.pkl, SMPL_MALE.pkl and SMPL_FEMALE.pkl.

1. Train a policy. config.ini defaults to backend = mujoco_gpu and solver_preset = newton_120:

python scripts/train.py --config config.ini -m <MOTION FILE PATH>

--env.solver-preset (mujoco backends only) selects the physics preset: newton_120 (the default) or newton_180 (higher fidelity, slower) for Newton-native training; isaacgym_parity_240 (or the higher-fidelity, ~2x slower isaacgym_parity_480) for evaluating checkpoints trained on Isaac Gym — these two are eval-only, not meant for training. See docs/newton_mujoco_physics.md and docs/physics_solver.md for details.

MuJoCo training presets are new; Isaac Gym is the validated training path so far (see docs/newton_mujoco_physics.md).

Add --track to log locally to Aim under experiments/.aim; view with pixi run aim up --repo experiments --host 127.0.0.1 --port 43800. Merge runs from another machine: pixi run aim runs --repo /experiments cp --destination experiments .

To prepare your own motion data, please see the convert_amass_data.py script in the scripts folder. After conversion, you can visually inspect the data with the vis_motion_mj.py script.

1. Play the trained policy. This opens a live viewer and needs a display. On the mujoco backends (Newton ViewerGL) red markers on grey bodies show the reference motion; keys: R reset, T resample motions, K toggle markers, left/right arrows switch the followed env (mujoco_cpu shows only env 0):

python scripts/train.py --mode play -c <CHECKPOINT PATH>

Use the same solver preset the policy was trained with. For checkpoints trained on Isaac Gym, such as the shared AMASS checkpoints linked above, add --env.solver-preset isaacgym_parity_240.

For batch evaluation (e.g., using 4096 envs to evaluate AMASS 11313 motions), run:

python scripts/train.py --mode eval -c <CHECKPOINT PATH> -m <MOTION FILE PATH>

1. Sweep the hyperparameters using CARBS.

python scripts/train.py --config config.ini --mode sweep --tag <name>

To adjust the sweep parameters and range, edit the config.ini file. You might need to comment/uncomment some parts in the train.py script, sweep_carbs() function to make the sweep work.

--mode sweep requires --tag ; every sweep trial is tracked and tagged in Aim. Filter in the Aim UI/search with "" in run.tags.

1. Run the tests.

pixi run -e default test

Legacy: Isaac Gym

Isaac Gym (Python 3.8, pytorch 2.3.1/CUDA 12.1, Isaac Gym 1.0rc4) is kept as a legacy backend. It requires its own pixi environment and a manual Isaac Gym install.

1. Activate the isaacgym environment:

pixi shell -e isaacgym

1. Install Isaac Gym and gymtorch. Download and unzip Isaac Gym from here. Then, install isaacgym by running the following command inside the virtual environment:

cd <isaac gym directory>/python
    pip install -e .

Also, install gymtorch by running the following command inside the repository directory:

pixi run -e isaacgym build_gymtorch

You can test if the installation is successful by running:

pixi run -e isaacgym test_deps

1. Download the SMPL parameters as in step 3 above (the original, chumpy-based .pkl files work as-is in the isaacgym environment; no re-save needed).

1. Train a policy. config.ini defaults to mujoco_gpu, so pass --env.backend isaacgym here and in the commands below.

python scripts/train.py --config config.ini -m <MOTION FILE PATH> --env.backend isaacgym

1. Play the trained policy. This opens the basic Isaac Gym viewer (no reference markers) and needs a display.

python scripts/train.py --mode play -c <CHECKPOINT PATH> --env.backend isaacgym

For batch evaluation (e.g., using 4096 envs to evaluate AMASS 11313 motions), run:

python scripts/train.py --mode eval -c <CHECKPOINT PATH> -m <MOTION FILE PATH> --env.backend isaacgym

1. Run the tests.

pixi run -e isaacgym test

Notes

• python scripts/train.py --help shows the full list of options for the environment and training, which allows you to override the defaults in the config file. Per-backend overrides go in [
  .] (e.g. [env.isaacgym], [train.isaacgym]); CLI flags still win.
• I tested the style discriminator in the original PHC repo and here, saw that it did not improve the imitation performance, so it is optional and off by default. To enable it, set use_amp_obs = True in the config, or add --env.use-amp-obs True to the command.
• I saw several times that "fine-tuning" pretrained weights with -c resulted in much faster learning. The L2 init reg loss is being logged in the training dashboard, and you can see greater L2 distance when learning from scratch compared to fine-tuning.
• As mentioned in the PHC paper, I saw that the 6-layer MLP perform better than shallow MLPs, and SiLU activations perform better than ReLUs.
• Adding LayerNorm before the last layer tamed the gradient norm, and I swept the hyperparameters with the max grad norm of 10 (the original PHC repo uses 50).
• Observations are RMS normalized, but the rewards/values are not. The gamma was set to 0.98 to manage the value loss. The hyperparameter sweeps consistently converged to small lambda values, so I chose 0.2.
• Also, the sweeps consistently converged to very aggressive clip coefs (0.01) and higher learning rates. I speculate that since the trainer uses the same learning rate for both actor and critic, it's using actor clipping to slow down the actor learning relative to the critic.
• I tried using LSTM for both the actor and critic, respectively, and it didn't work better. These LSTM policies are included in the code, so feel free to try them.

References

• Main PHC code, motion lib, poselib, and data scripts are from: PHC and Isaac Gym
• The PPO and CARBS sweep code is from: PufferLib
• Sample motion data is from: CMU Motion Capture Dataset, Subject 5, Motion 6