Visit the website!

Please visit the website! (after leaving a ⭐star⭐! 😇)

https://perfanalytics.github.io/pose2sim

\ \ \

Pose2Sim

https://github.com/user-attachments/assets/51a9c5a1-a168-4747-9f99-b0670927df95

Pose2Sim provides a workflow for 3D markerless kinematics (human or animal), as an alternative to traditional marker-based MoCap methods.

Pose2Sim is free and open-source, requiring low-cost hardware but with research-grade accuracy and production-grade robustness. It gives maximum control over clearly explained parameters. It also respects data privacy and GDPR rules since it runs locally. Any combination of phones, webcams, or GoPros can be used with fully clothed subjects, so it is particularly adapted to the sports field, the doctor's office, or for outdoor 3D animation capture.

[!TIP]

For real-time analysis with a single camera, please consider Sports2D (note that the motion must lie in the sagittal or frontal plane).

[!NOTE]

Features:

- 🎯 Research-Grade Accuracy (2-6° joint angle error, validated by peer-reviewed studies)

- 📹 Any Cameras (phones, webcams, GoPros...)

- 👥 Multi-Person Support (track multiple people simultaneously)

- 🤸 Full 3D kinematics (complete OpenSim skeletal analysis with joint angles)

Use cases:

- 🏀 Sports Analysis (field-based 3D motion capture)

- 🏥 Clinical Assessment (gait analysis in doctor's office)

- 🎭 Animation (outdoor 3D capture with fully clothed subjects)

- 🔬 Research (biomechanics studies with multiple participants)

[!NOTE]

Fun fact: Pose2Sim stands for "OpenPose to OpenSim", as it originally used OpenPose inputs (2D keypoints coordinates) and led to an OpenSim result (full-body 3D joint angles). Pose estimation is now performed with more recent models from RTMPose, and custom models (from DeepLabCut, for example) can also be used.

[!IMPORTANT]

This project is completely free and open-source: it you like it and find it useful, please consider supporting it!\

You can help make it more visible and sustainable by ⭐ Giving us a star ⭐ or ❤️ Becoming a sponsor. ❤️

[!IMPORTANT]

If you want to contribute to Sports2D or Pose2Sim, please see How to contribute or join the Discord community!

1. Installation and Demonstration

1. Use on your own data

1. All the parameters
2. Utilities
3. How to cite and how to contribute

Installation and Demonstration

Installation

Let's get started!

[!NOTE]

If you'd rather use conda, you can still use the old installation procedure. Still works fine but not recommended, since uv is faster, lighter, better at handling dependencies, and generally more modern.

1. Set up a uv environment:

Open a terminal (conda, powershell, bash, or zsh).

On Windows:

# Install uv
  powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
  # Create uv environment
  uv venv "$env:USERPROFILE\.venv\pose2sim" --python 3.13 # or 3.11, or 3.12 
  # Activate the uv environment
  & "$env:USERPROFILE\.venv\pose2sim\Scripts\Activate.ps1"

On Linux or MacOS:

# Install uv
  curl -LsSf https://astral.sh/uv/install.sh | sh
  # Create uv environment
  uv venv ~/.venv/pose2sim --python 3.13 # or 3.11, or 3.12 
  # Activate the uv environment
  source ~/.venv/pose2sim/bin/activate

[!TIP]

#### Activate your environment

Remembering the command for activating the uv environment can be a pain. Just type Ctrl+R in your terminal and start typing activate to find it.

2. Install Pose2Sim:

Open a terminal (conda, powershell, bash, or zsh).\ Activate your environment.

• OPTION 1: Latest stable version:

uv pip install pose2sim --upgrade

• OPTION 2: For developers who want to test and edit the bleeding edge version:

git clone --depth 1 https://github.com/perfanalytics/pose2sim.git
    cd pose2sim
    uv pip install -e .

3. Optional:

Open a terminal, activate your environment, and run nvidia-smi. If this results in an error, your GPU is probably not compatible with CUDA. If not, note the "CUDA version": it is the latest version your driver is compatible with (more information on this post).

Then go to the ONNXruntime requirement page, note the latest compatible CUDA and cuDNN requirements. Next, go to the pyTorch website and install the latest version that satisfies these requirements (beware that torch 2.4 ships with cuDNN 9, while torch 2.3 installs cuDNN 8). For example:

uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

Finally, install ONNX Runtime with GPU support:

uv pip uninstall onnxruntime
  uv pip install onnxruntime-gpu

Check that everything went well within Python with these commands:

import torch; import onnxruntime as ort
  print(torch.cuda.is_available(), ort.get_available_providers())
  # Should print "True ['CUDAExecutionProvider', ...]"

[!TIP]

#### Note on storage use:

A full installation takes up to 14 GB of storage space. HoweveGPU and GUI supports are not mandatory and take about 5 GB. Tcache can be cleared to save space if you don't care for futuinstallations to be instantaneous.

A minimal installation with carefully chosen pose models and without GPU support would take less than 2 GB. \

Demonstration Part-1: End to end video to 3D joint angle computation

Test the full pipeline!

• Open a terminal (conda, powershell, bash, or zsh) and activate your environment.
• Find the Demo folder under \Pose2Sim\Demo_SinglePerson:

uv pip show pose2sim # to find <pose2sim_path>

• Copy-paste the Demo folder wherever you like, and rename it as you wish (manually or using the cp command).\

cd <Demo_SinglePerson_path>
  ipython

• Try the following code:

from Pose2Sim import Pose2Sim
    Pose2Sim.calibration()
    Pose2Sim.poseEstimation()
    Pose2Sim.synchronization()
    Pose2Sim.personAssociation()
    Pose2Sim.triangulation()
    Pose2Sim.filtering()
    Pose2Sim.markerAugmentation()
    Pose2Sim.kinematics()

[!NOTE]

3D marker trajectories are stored as .trc files in each trial folder in the pose-3d directory.\

3D joint angles are stored as .mot files in the kinematics directory. Scaled .osim models are also stored in the same directory.\

Processing details and statistics are stored in logs.txt.

Demonstration Part-2: Visualize your results with OpenSim or Blender

_Visualize your results and look in detail for potential areas of improvement._

Basic visualization with the OpenSim GUI

• Install OpenSim GUI:\

• Visualize results:

Further investigation with the Pose2Sim Blender add-on

• Install the add-on:\

• Visualize results:\

https://github.com/user-attachments/assets/53bfd9e2-f542-410f-8e9f-7b70229d598b

[!NOTE]

The OpenSim skeleton is not rigged yet. Feel free to contribute!

Open a terminal (conda, powershell, bash, or zsh) and activate your environment.

Multi person analysis

Discover another person, hidden all along!

Similarly to Part-1, find the Multi-Person Demo folder under \Pose2Sim\Demo_MultiPerson, and move it and rename it if you like. Make sure you set multi_person = true in your Config.toml file. Go to the folder and start python:

cd <Demo_MultiPerson_path>
  ipython
  from Pose2Sim import Pose2Sim
  Pose2Sim.runAll()

[!NOTE]

Pose2Sim uses sophisticated algorithms to Match people across views, Track IDs over time, Handle occlusions.

Batch processing

Scale from research studies to production pipelines!\

Run numerous analyses with different parameters and minimal friction.

Find the Batch Demo folder under \Pose2Sim\Demo_Batch, and move it and rename it if you like. Go to the folder and start python:

cd <Demo_Batch_path>
ipython
from Pose2Sim import Pose2Sim
Pose2Sim.runAll()

[!NOTE]

This lets you run entire sessions with consistent calibration and global settings across trials.\

Trial-specific customization is possible when needed

[!TIP]

The batch processing structure requires a root Config.toml file, as well as one in each of the trial directories. Global parameters are given in the root one. Any trial can override specific parameters by uncommenting keys and their values in the trial-specific Config.toml files.

For example, try uncommenting [project] and set frame_range = [10,99] in the root Config.toml, and uncomment [pose] and set mode = 'lightweight' in the Trial_2 one.

Run Pose2Sim from the folder if you want to batch process the whole session, or from a subfolder if you only want to process a specific trial.

| SingleTrial | BatchSession |

|-----------------|--------------------|

|

SingleTrial                    
├── calibration
├── videos
└── Config.toml

|

BatchSession                     
├── calibration 
├── Trial_1   
│   ├── videos 
│   └── Config.toml # Trial specific parameters
├── Trial_2 
│   ├── videos 
│   └── Config.toml # Different trial parameters
└── Config.toml # Global parameters

|

Demonstration Part-4: Go further

Playing with parameters and try different features!

Default parameters are provided in Config.toml but can be edited.\

All of them are clearly documented: feel free to play with them!

• Try the calibration tool:\

Pose2Sim.calibration()

• Pass an updated config dictionary instead of editing Config.toml, and test whole_body pose estimation:

import toml
  config_dict = toml.load("<Demo_SinglePerson_path>/Config.toml")
  config_dict.get("project").update({"project_dir":"<Demo_SinglePerson_path>"})
  config_dict.get("pose").update({"pose_model": "whole_body","overwrite_pose": True})
  Pose2Sim.poseEstimation(config_dict)
# Or even simpler, just pass the updated parameters
  config_dict = {"project": {"project_dir": "<Demo_SinglePerson_path>"}, 
                 "pose": {"pose_model": "whole_body", "overwrite_pose": True}}
  Pose2Sim.poseEstimation(config_dict)
# Or else, you can simply pass the path to a .toml file
  Pose2Sim.poseEstimation("<Custom_path>/your_config.toml")

• Run all stages at once:

from Pose2Sim import Pose2Sim
  Pose2Sim.runAll()
  # or: 
  # Pose2Sim.runAll(do_calibration=True, do_poseEstimation=True, do_synchronization=True, do_personAssociation=True, do_triangulation=True, do_filtering=True, do_markerAugmentation=True, do_kinematics=True)

Too slow for you?

Pose2Sim can probably run 10 - 20 times faster with a few tweaks!

Project set up:

• Set multi_person = True in your Config.toml file, and remove all the detections other than the ones of interest. It works as well as the single person mode, and is much faster. In the future, I plan to add a person selection feature (like on Sports2D) and to remove the old single person mode.
• Limit frame_range to only process a subset of frames.

Pose2Sim.poseEstimation():

• Use your GPU: This makes pose estimation significantly faster, without any impact on accuracy. See Installation section for more information.\

• Set display_detection = false. Do not display results in real time.\

• Set parallel_workers_pose = 'auto' or an integer (number of threads). 'auto': one thread per video (ie 4 on the Demo data). This requires display_detection = false. No impact on accuracy either.\

• Set det_frequency = 100 in Config.toml. Run the bounding box detector and the pose estimator on the first frame; for all subsequent frames, only run pose estimation. No impact on accuracy but may miss detection or swap some person IDs if several persons are in the scene.\

• Use mode = 'lightweight': Will use a lighter version of RTMPose, which is faster but less accurate.\

• Set save_video = 'none'. Do not save images and videos\

• Set tracking_mode = 'sports2d' or tracking_mode = 'none'. If several persons are in the scene, use the sports2d tracker or no tracker at all, but not 'deepsort' (sports2d tracking is almost instantaneous though).

Pose2Sim.personAssociation():\ Do not run if there is only one person in the scene.

Pose2Sim.triangulation():\ Not much to do here.

Pose2Sim.filtering():\ You can skip this step, but it is quite fast already.

Pose2Sim.markerAugmentation():\ Very fast, too. Note that marker augmentation won't necessarily improve results so you can consider skipping it.

Pose2Sim.kinematics():

• Set use_simple_model = true. Use a simpler OpenSim model, without muscles and constraints. Note that the spine will be stiff and shoulders will be a simple ball joint, but this is accurate enough for most gait-related tasks\

• Set parallel_workers_kinematics = 'auto' or an integer (number of processes). Only works in multi-person mode.

Use on your own data

Start filming

Start right away!

Film your participant(s) from at least 2 points of view.

[!TIP]

#### Camera placement:

- With 2 cameras, one person: According to Samani et al, 2026, best results are achieved when one camera is in front of the person, the other at 45° to the side, both at hip level. Acceptable results are achieved with one camera in front and one on the side at 90°; or two cameras in front, one at hip level and the other at eye level. Zhao et al, 2026 recommend front or back configurations, especially for frontal- and tranverse-plane angle measurements.\

With 2 cameras, multiple people: Try to place cameras in a way that minimizes occlusions.

- With multiple cameras: Make sure the cameras see the subjects from as many different angles as possible and with minimal occlusions. HowNote that top views right above the subjects do not yield good results.

- In difficult settings with limited space and obstacles: consider using the Lab Camera Optimizer.

[!NOTE]

- Synchronization: If your cameras are not natively synchronized, you will need one of the participants to do a sharp vertical movement.\

More on it in the Synchronization section.

- Calibration: If your cameras are not natively calibrated, you will need to film a chessboard or a charucoboard (once in the camera's lifetime) and to retrieve the coordinates of some object in the scene (everytime you move the camera).\

More on it in the Calibration section.

[!NOTE]

Uncluttered background and good lighting conditions will help you get better results. Note that tripods may sometimes be mistaken for people.

[!TIP]

#### Minimize storage use:

The main drawback of working with videos is the size of the files. Here are some example settings (inspired by Hardy et al, 2026) to significantly reduce storage use with little impact on accuracy:

- Compress videos: Increasing crf from 20 to 30 has virtually no impact on accuracy, and decreases storage use by up to 10 times.

- Decrease resolution: Decreasing from HD (1080 x 1920) to SD (720 x 1280) or even further has little impact, and decreases storage use by about 8 times.

- Audio does not matter: Audio tracks take between 1 and (rarely) 10% of the file size.

- Camera number: 6 is generally a good camera number. Of course, complex motions and occlusions require more cameras, and gait requires fewer.

- Face blurring: Preserves privacy with little impact on accuracy. Use this Utility script.

- Keep high framerate: Framerates below 60 Hz lead to accuracy drops, especially for fast motions.

In practice, here is how you can compress your video and decrease its resolution with ffmpeg:

> ffmpeg -i original_vid.MP4 

1. Open a terminal, enter uv pip show pose2sim, report package location. \

\pose2sim.

1. Copy the appropriate Demo folder as your project template (Demo_SinglePerson, Demo_MultiPerson, or Demo_Batch). \

Paste it to your preferred location and rename as desired.

1. Edit the parameters in the [project] section of Config.toml, e.g. project_dir, participant_height, participant_mass, etc.
2. Add your videos to the videos folder. We recommend renaming them for clarity.
3. The rest of the tutorial will explain to you how to populate the Calibration and videos folders, run each Pose2Sim step, and edit the Config.toml file.

Initial project structure:

| SingleTrial | BatchSession |

|-----------------|--------------------|

|

SingleTrial                    
├── calibration
├── videos
└── Config.toml

|

BatchSession                     
├── calibration 
├── Trial_1   
│   ├── videos 
│   └── Config.toml # Trial specific parameters
├── Trial_2 
│   ├── videos 
│   └── Config.toml # Different trial parameters
└── Config.toml # Global parameters

|

Estimate 2D pose

_Estimate 2D pose from images with RTMPose or another pose estimation solution._

With RTMPose (default):

RTMPose is a state-of-the-art pose estimation solution that is accurate, fast, flexible, and natively integrated in Pose2Sim.

Open a terminal in your project folder, activate your environment, and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.poseEstimation()

This will run pose estimation on your videos and save the results in the pose folder. They are provided in the OpenPose format, with one .json file per frame per camera, containing keypoint coordinates and confidence scores.

[!TIP]

See the Too slow for you? section to make it faster by deactivating real-time display, running parallel pose estimation, using your GPU, increasing the detection frequency, and using a lighter model.

[!TIP]

#### Other available models

- Whole body: Use 'Whole_body' pose_model in Config.toml to track 133 keypoints on the body, face, and hands. Note that it is slower and slightly less accurate than the default 'Body_with_feet' model on body keypoints.

- Wrist motion: Use 'Whole_body_wrist' pose_model. Effectively runs the 'whole_body' model but ignores all face and finger keypoints, except 2 per hand.

- Lower body: Use 'Lower_body' pose_model. Effectively runs the 'body_with_feet' model but ignores all upper body keypoints.

- Others: You can also use the Hand, Face, Animal, or any other RTMlib model (see next tip).

[!TIP]

#### Use custom detection and pose models

- Other than 'lightweight', 'balanced', or 'performance' modes, you can use any other pose estimation models through RTMLib (hand, face, animal, or any custom trained models).

- The (optional) detection model, pose model, and input sizes can be written in a dictionary (within triple quotes) as shown below. Models can be local paths or URLs, with .onnx or .zip extensions. Make sure the input_sizes are within square brackets.

> # Equivalent to mode='balanced', with body_with_feet pose model

> mode = """{'det_class':'YOLOX', > 'det_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/yolox_m_8xb8-300e_humanart-c2c7a14a.onnx', > 'det_input_size':[640, 640], > 'pose_class':'RTMPose', > 'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.onnx', > 'pose_input_size':[192,256]}""" > > # With one-stage RTMO model > # Requires pose_model = 'Body'. Marker augmentation won't work, Kimatic analysis will > mode = """{'pose_class':'RTMO', > 'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmo/onnx_sdk/rtmo-m_16xb16-600e_body7-640x640-39e78cc4_20231211.onnx', > 'pose_input_size':[640, 640]}""" > > # With animal pose estimation: > # Marker augmentation won't work, and you will need to create your owOpenSim skeleton for kinematic analysis. > mode = """{'pose_class':'RTMPose', > 'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.onnx', > 'pose_input_size':[256,256]}""" > > # Same approach for hand or face pose estimation, check the RTMLib domentation for more information. >

[!WARNING]

Note that it does not currently work well for acrobatic movements where the person is upside down. We are working on a solution but in the meantime, you can try MediaPipe BlazePose (see here).

With MMPose (coming soon):

Coming soon

With DeepLabCut:

If you need to detect specific points on a human being, an animal, or an object, you can also train your own model with DeepLabCut. In this case, Pose2Sim is used as an alternative to AniPose.

1. Train your DeepLabCut model and run it on your images or videos (more instruction on their repository)
2. Translate the h5 2D coordinates to json files (with DLC_to_OpenPose.py script, see Utilities). Note that the names of your camera folders must follow the same order as in the calibration file, and end with '_json':

` cmd DLC_to_OpenPose -i input_h5_file

3. Edit pose.CUSTOM in Config.toml, and edit the node IDs so that they correspond to the column numbers of the 2D pose file, starting from zero. Make sure you also changed the pose_model and the tracked_keypoint.\
  You can visualize your skeleton's hierarchy by changing pose_model to CUSTOM and writing these lines:

python config_path = r'path_to_Config.toml' import toml, anytree config = toml.load(config_path) pose_model = config.get('pose').get('pose_model') model = anytree.importer.DictImporter().import_(config.get('pose').get(pose_model)) for pre, _, node in anytree.RenderTree(model): print(f'{pre}{node.name} id={node.id}')

4. Create an OpenSim model if you need inverse kinematics.
</br>
With OpenPose (legacy):
> [!WARNING]
> RTMlib is faster, more accurate, and easier to install than OpenPose. This is a legacy option.
> [!WARNING]
> OpenPose model files are apparently not available on their website anymore. Send me an email at [email protected] if you want me to forward them to you!
The accuracy and robustness of Pose2Sim have been thoroughly assessed only with OpenPose, BODY_25B model. Consequently, we recommend using this 2D pose estimation solution. See OpenPose repository for installation and running. Windows portable demo is enough.
Open a command prompt in your OpenPose directory. \
  Launch OpenPose for each videos folder:

cmd bin\OpenPoseDemo.exe --model_pose BODY_25B --video \videos\cam01.mp4 --write_json \pose\pose_cam01_json

* The BODY_25B model has more accurate results than the standard BODY_25 one and has been extensively tested for Pose2Sim. \
You can also use the BODY_135 model, which allows for the evaluation of pronation/supination, wrist flexion, and wrist deviation.\
All other OpenPose models (BODY_25, COCO, MPII) are also supported.\
Make sure you modify the Config.toml file accordingly.
Use one of the json_display_with_img.py or json_display_with_img.py scripts (see Utilities) if you want to display 2D pose detections.

</br>
With MediaPipe BlazePose:
> [!WARNING]
> RTMlib is more accurate than Mediapipe, except for acrobatic movements.
Mediapipe BlazePose is very fast, fully runs under Python, handles upside-down postures and wrist movements (but no subtalar ankle angles). \
However, it is less robust and accurate than OpenPose, and can only detect a single person.
Use the script Mediapipe_to_pose_or_trc.py (see Utilities) to run Mediapipe under Python, and store the detected coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format:

cmd Mediapipe_to_pose_or_trc -i input_file -dJs

Type in Mediapipe_to_pose_or_trc -h for explanation on parameters.
Make sure you changed the pose_model and the tracked_keypoint in the Config.toml file.

</br>
With AlphaPose (legacy):
> [!WARNING]
> RTMlib is faster, more accurate, and easier to install than AlphaPose. This is also a legacy option.
AlphaPose is one of the main competitors of OpenPose, and its accuracy is comparable. As a top-down approach (unlike OpenPose which is bottom-up), it is faster on single-person detection, but slower on multi-person detection.\
All AlphaPose models are supported (HALPE_26, HALPE_68, HALPE_136, COCO_133, COCO, MPII). For COCO and MPII, AlphaPose must be run with the flag "--format cmu".
Install and run AlphaPose on your videos (more instruction on their repository)
Translate the AlphaPose single json file to OpenPose frame-by-frame files (with AlphaPose_to_OpenPose.py script, see Utilities):

cmd AlphaPose_to_OpenPose -i input_alphapose_json_file

* Make sure you changed the pose_model and the tracked_keypoint in the Config.toml file.
</br>
Synchronize your cameras
> _2D points can be triangulated only if they represent the same body position across all cameras: therefore, views need to be synchronized. This module helps you do it._\
For each camera, the algorithm computes mean vertical speed for the chosen keypoints, and synchronizes by finding the time offset for which the correlation is highest.
> [!TIP]
> Skip this step if your cameras are natively synchronized.
Open a terminal in your project folder, activate your environment, and run ipython:

python from Pose2Sim import Pose2Sim Pose2Sim.synchronization()

<img src="Content/P2S_synchronization.png" width="760">

<br>
You can choose the keypoints to synchronize on, the reference person, and the time window for analysis. You can either tune these parameters in the GUI (set synchronization_gui = true) or set them in your Config.toml file.

<img src="Content/synchro_multi.jpg" width="760">
<img src="Content/synchro.jpg" width="760">
> [!TIP]
> Works best when:
> 
> - the participant does not move towards or away from the cameras
> - they perform a clear vertical movement
> - the capture lasts at least 5 seconds, so that there is enough data to synchronize on
> - the capture lasts a few minutes maximum, so that cameras are less likely to drift with time
> [!NOTE]
> Alternatively, synchronize cameras using a flashlight, a clap, or a clear visual or audio event. The speed of sound being lesser than the speed of light, visual events should be preferred.
> [!TIP]
> GoPro cameras can also be synchronized with a timecode or by GPS (outdoors).
</br>
Calibrate your cameras
> _**Calculate camera intrinsic properties (lens characteristics) and extrinsic parameters (positions and orientations).\
> Convert a preexisting calibration file, or calculate intrinsic and extrinsic parameters from scratch.**_
Open a terminal in your project folder, activate your environment, and run ipython:

python from Pose2Sim import Pose2Sim Pose2Sim.calibration() `

Output file:

---

Table of Contents

1. Convert from Caliscope, AniPose, FreeMocap, Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV
2. Calculate from scratch

1. Calculate intrinsic parameters with a board 2. Calculate extrinsic parameters with scene landmarks, a static or a moving board, or pose estimation keypoints

---

Convert from Caliscope, AniPose, FreeMocap, Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV

If you already have a calibration file, set calibration_type type to convert in your Config.toml file, follow the instructions below for your specific calibration type, and run Pose2Sim.calibration().

[!NOTE]

If the original calibration file does not provide any residual errors, they will be logged as NaN. This is not an error and can be ignored.

• From Caliscope (recommended if your cameras are synchronized), Dynamic Extrinsic Camera Calibrator (based on keypoints, not a board! But requires synchronized videos, does not estimate intrinsics, and not commercially available), AniPose, or FreeMocap:

- Copy your .toml calibration file to the Pose2Sim Calibration folder. - Calibration can be skipped since these formats are natively supported by Pose2Sim. - Note: It seems like the FreeMoCap calibration is in millimeters rather than in meters. Just open your calibration.toml file and multiply all the translation values by 1000.

• From Qualisys:

- Export calibration to .qca.txt within QTM (see there). - Copy it in the Calibration Pose2Sim folder. - set convert_from to 'qualisys' in your Config.toml file. Change binning_factor to 2 if you film in 540p. - If you set your cameras vertically and the videos are rendered sideways, you need to rotate them and the calibration file before running pose estimation. Use this script.

• From Optitrack: Exporting calibration will be available in Motive 3.2. In the meantime:

- Calculate intrinsics with a board (see next section). - Use their C++ API to retrieve extrinsic properties. Translation can be copied as is in your Calib.toml file, but TT_CameraOrientationMatrix first needs to be converted to a Rodrigues vector with OpenCV. See instructions here. - Use the Calib.toml file as is and do not run Pose2Sim.calibration()

• From Vicon:

- Copy your .xcp Vicon calibration file to the Pose2Sim Calibration folder. - set convert_from to 'vicon' in your Config.toml file. No other setting is needed.

• From OpenCap:

- Copy your .pickle OpenCap calibration files to the Pose2Sim Calibration folder. - set convert_from to 'opencap' in your Config.toml file. No other setting is needed.

• From EasyMocap:

- Copy your intri.yml and extri.yml files to the Pose2Sim Calibration folder. - set convert_from to 'easymocap' in your Config.toml file. No other setting is needed.

• From bioCV:

- Copy your bioCV calibration files (no extension) to the Pose2Sim Calibration folder. - set convert_from to 'biocv' in your Config.toml file. No other setting is needed.

---

Calculate from scratch

_Calculate calibration parameters with a chessboard or a charucoboard, with measurements on the scene, or automatically with pose estimation keypoints._\

Take heart, it is not that complicated once you get the hang of it!

[!TIP]

- Try the calibration tool on the Demo data by changing calibration_type to calculate in Config.toml.

- Then try again after changing [calibration.calculate.extrinsics.extrinsics_method] to charuco, then keypoints.

For the sake of practicality, there are voluntarily few board images for intrinsic calibration in our Demo, and few points to click for extrinsic calibration. In spite of this, your reprojection error should be under 1-2 cm, which does not hinder the quality of kinematic results in practice.

---

Calculate intrinsic parameters with a board

[!NOTE]

#### Intrinsic parameters:

Camera properties (focal length, optical center, distortion), stored in the matrix and distortion arrays of a Calib.toml file.

They usually need to be calculated only once in their lifetime. In theory, cameras with the same model and settings will have identical intrinsic parameters, so they can be copied from one Calib.toml file to another. In practice, small variations can occur.\

Warning: This does not hold true if you zoom in or out. In this case, you will need to recalculate intrinsic parameters.

• For each camera, film a chessboard or a charucoboard. You can either move the board in front of the camera, or move the camera around the board.\

N.B.: Make sure you film from multiple angles and cover most of frame.

• Create a folder for each camera in your Calibration\intrinsics directory and copy your images or videos in them.
• Adjust corners_nb, square_size, and marker_resolution in the [calibration.calculate.intrinsics] section of Config.toml.
• Run Pose2Sim.calibration().

- For each image, if you are satisfied with the detection, press Y. - If you want to dismiss a specific image, press N. - If you want to improve the detection by clicking the points, press C. Right click to undo a click, and H to label a point as hidden. Use the mouse whell to zoom and pan.

[!TIP]

If you are confident in your detections, you can set show_detection_intrinsics to false. If you already calculated intrinsic parameters earlier, you can skip this step altogether by setting overwrite_intrinsics to false.

[!TIP]

#### Chess or ChArUco board requirements:

Generate a chess or charuco board at calib.io.

A ChArUco board is more robust to occlusions or orientation ambiguities than a chessboard.

> - Intrinsics: Many corners, e.g. 5x8 corners on A4/US-letter format\

Extrinsics (static): Large squares with several corners, e.g. 3x6 on A0 format\

Extrinsics (dynamic): Large squares, e.g. 1x2 corners on A3/US-ledger format,

- Asymmetric: Rows ≠ Columns (or rows should be odd if columns is even)

- Border: Wide white border around the pattern

- Flat: The board must be completely flat

- Focus: Sharp, in-focus images

- No glare: Avoid reflections

>

[!IMPORTANT]

Common errors:

- Specifying the external, instead of the internal number of corners. Specifically, corners_nb = rows-1, cols-1 (compare with calib.io).

- Taking photos from the scene instead of extracting frames from a video. The photo image format is often different from the video one, which skews intrinsic calibration.

[!IMPORTANT]

Intrinsic calibration error should be close to or below 0.5 px.

---

Calculate extrinsic parameters with scene landmarks, a static or a moving board, or pose estimation keypoints

[!NOTE]

#### Extrinsic parameters:

Camera positions and orientations in space, stored in the rotation and translation arrays of a Calib.toml file.

> They need to be recalculated every time a camera is moved. Several methods are available, using known scene coordinates, a static board or a moving board, or pose estimation keypoints.

[!IMPORTANT]

Extrinsic calibration error should be below 1 cm, but depending on your application, results will still be potentially acceptable up to 2.5 cm.

##### With scene coordinates

• Manually measure the 3D coordinates of 10 or more points in the scene (tiles, wall lines, boxes, treadmill dimensions...). These points should be as spread out as possible.
• Set extrinsics_method = 'scene' in Config.toml.
• Replace object_coords_3d by your coordinates in the [calibration.calculate.extrinsics.scene] section.
• Once your cameras are in place, make a quick recording of the raw scene with all cameras. Only one frame is needed, but do not just take a photo unless you are sure it does not change the image format.
• Copy your videos into the calibration/extrinsics folder.
• Run Pose2Sim.calibration(). You will be prompted to click on the corresponding image points for each view.

[!TIP]

- Works even on unsynchronized cameras

- Very accurate if the landmark coordinates are sufficiently spread out

- If there is no measurable item in the scene, you can temporarily bring something in (a table, for example), perform calibration, and then remove it before starting to capture motion

- Providing that scene coordinates can be retrieved later on, it can be used on old, forgotten video files

##### With a static board

• Put a large chess or charuco board in the field of view. See board requirements here.

- If laid horizontally on the ground: The cameras must be high enough to correctly detect the board. - If laid vertically (board_position = 'vertical'): The cameras can only be on one side of the board.

• Set extrinsics_method = 'charuco_static' or 'chess_static'` in Config.toml