Profile
Back to NewsBack
GitHub Trending 5 min
Reader Mode
flowhmr/flowhmr: Official implementation of FlowHMR, a generative model for physically plausible motion capture from monocular video.

flowhmr/flowhmr: Official implementation of FlowHMR, a generative model for physically plausible motion capture from monocular video.

10 hours ago

FlowHMR: Physically Plausible Motion Capture from Video

Hugging Face Demo Space arXiv Project Page Hugging Face Model

News

  • 🎥🎥🎥 [2026-11-22] Video inference now uses YOLO26m for person detection and Pi4 for camera pose and intrinsics estimation by default.
If GPU memory is limited, choose SimpleVO to estimate camera rotation on the CPU. Reconstruct and preview 3D motion in your browser, with NPZ/FBX export.
  • 🚀🚀🚀 [2026-10-30] Updated inference acceleration: approximately 10× faster SAM feature extraction
(with compile) and over 5× faster end-to-end inference (static camera, with compile) on our 300-frame A100 benchmark. See the resource and performance report.

How to Use FlowHMR

Training and evaluation — Synthetic data generation · Pretraining and GRPO post-training · Wild-4K evaluation clip list

Model Zoo

| Model | Description | Params | Tracking Success ↑ | Download | |---|---|:---:|:---:|:---:| | flowhmr_base | Base model | 0.46B | 78.79% | 🤗 HF | | flowhmr_latest | flowhmr_base + GRPO with PHC+ physics reward (recommended) | 0.46B | 82.47% | 🤗 HF |

Tracking Success: the rate at which the PHC+ controller successfully tracks the predicted motion in simulation, measured on Wild-4K.

Getting Started

Follow the guide for each step, or paste the prompt into your coding agent.

Installation

Installation guide

FlowHMR supports two setups:

  • V2M: inference, web demo, and base model training in a single environment
(tested with Python 3.10 + PyTorch 2.5.1, cu118).
  • Full pipeline: V2M plus the PHC+ physics simulator for GRPO post-training and
physics tracking. The simulator runs in a separate Isaac Gym environment (tested with Isaac Gym Preview 4, Python 3.8 + PyTorch 1.13, cu117). The two environments share no imports and communicate through files on /dev/shm.

The default video setup uses YOLO26m and Pi4 and requires access to SAM 3D Body and the SMPL-H body model. VGGT-Omega requires separate access when selected.

V2M:

Install FlowHMR from https://github.com/flowhmr/flowhmr following docs/install.md with the V2M setup.

Full pipeline:

Install FlowHMR from https://github.com/flowhmr/flowhmr following docs/install.md with the full pipeline setup.

Web Demo

Web demo guide

Upload a video in the browser and reconstruct its 3D motion; when several people are detected, pick the one to reconstruct. Requires the V2M setup.

Launch the FlowHMR web demo following docs/web_demo.md.

Inference

Inference guide

Reconstruct motion from a video on the command line. The pipeline runs person detection (YOLO26m or YOLOX-L), camera estimation (Pi4, VGGT-Omega, SimpleVO, or a fixed pose), body feature extraction (SAM 3D Body), and FlowHMR, and saves the motion as .npz (plus .fbx if FBX export is installed). New runs use YOLO26m for person detection and Pi4 to estimate camera poses and intrinsics. Use --detector yolox for YOLOX-L or --camera vggt for VGGT-Omega. If camera estimation runs out of GPU memory, select SimpleVO (--camera simplevo) to estimate camera rotation on the CPU.

Run FlowHMR on my video following docs/inference.md.

Data

Data guide

  • Training data is not bundled. Bring your own in the
documented format, or render synthetic videos from SMPL-H motions with the Blender toolkit in data/rendering_data.
  • Wild-4K is the 4182-clip in-the-wild evaluation set behind Tracking Success.
We release the clip list (clip IDs from Koala-36M-v1), not the videos.

Training

Training guide

Training has two stages: pretraining the base model, then GRPO post-training with a physics reward. Prepare training data first (see Data and Prepare data).

Pretraining (V2M setup) trains the flow-matching base model, flowhmr_base.

Train the FlowHMR base model following docs/training.md.

GRPO post-training (full pipeline setup) fine-tunes flowhmr_base with the PHC+ simulator as the reward: each generated motion is scored by how well the PHC+ controller tracks it. This produces flowhmr_latest.

Run FlowHMR GRPO post-training following docs/training.md.

Acknowledgements

We thank the authors of these projects for releasing their code and models:

Citation

If you find FlowHMR useful in your research, please cite:

@article{wang2026flowhmr,
  title={FlowHMR: Physically Plausible Motion Capture from Video},
  author={Wang, Zhanke and Zhao, Chengfeng and Shuai, Qing and Lin, Jingzhong and Li, Heng and Ling, Zeyu and Wen, Yuxin and Li, Jing and Kang, Di and Guo, Chunchao and Bao, Linchao},
  journal={arXiv preprint arXiv:2610.03691},
  year={2026}
}

License

The FlowHMR source code is licensed under the Apache License 2.0. Third-party libraries and components are subject to their respective licenses. For model weights, please refer to the license specified in the model card.

Chat with me