list all available datasets

load dataset with default options and official split

split_ratio: train/test/val

or use dataset options

Please refer to Dataset Cards for more details about the dataset_opts.
:1st_place_medal: Benchmark Performances and Model Zoo
Results Calibration
Please refer to the results calibration to verify the correctness of the python implementations compared with official scripts in matlab or python.
⏬ Download Benchmark Datasets
For convenience, we upload all related datasets to huggingface IQA-Toolbox-Dataset, and corresponding meta information files to huggingface IQA-Toolbox-Dataset-metainfo. 
Here are example codes to download them from huggingface:
>[!CAUTION]
> we only collect the datasets for academic, research, and educational purposes. It is important for the users to adhere to the usage guidelines, licensing terms, and conditions set forth by the original creators or owners of each dataset.

save_dir = './datasets' os.makedirs(save_dir, exist_ok=True)

filename = "koniq10k.tgz" snapshot_download("chaofengc/IQA-Toolbox-Datasets", repo_type="dataset", local_dir=save_dir, allow_patterns=filename, local_dir_use_symlinks=False)

os.system(f"tar -xzvf {save_dir}/{filename} -C {save_dir}")

Download meta information from Huggingface with git clone or update with git pull:

cd ./datasets/meta_info git pull

Examples to specific dataset options can be found in ./pyiqa/default_dataset_configs.yml. Details of the dataloader interface and meta information files can be found in Dataset Preparation

Performance Evaluation Protocol
We use official models for evaluation if available. Otherwise, we use the following settings to train and evaluate different models for simplicity and consistency:
| Metric Type   | Train     | Test                                       | Results                                                  | 
| ------------- | --------- | ------------------------------------------ | -------------------------------------------------------- |
| FR            | KADID-10k | CSIQ, LIVE, TID2008, TID2013               | FR benchmark   |
| NR            | KonIQ-10k | LIVEC, KonIQ-10k (official split), TID2013, SPAQ | NR benchmark   |
| Aesthetic IQA | AVA       | AVA (official split)                       | IAA benchmark |
| Face IQA | CGFIQA | CGFIQA (official split)                    | Face IQA benchmark |
| Efficiency | CPU/GPU Time, GPU Memory | Average on $1080\times800$ image inputs | Efficiency benchmark |
Results are calculated with:
PLCC without any correction. Although test time value correction is common in IQA papers, we want to use the original value in our benchmark.
Full image single input. We do not use multi-patch testing unless necessary.
Basically, we use the largest existing datasets for training, and cross dataset evaluation performance for fair comparison. The following models do not provide official weights, and are retrained by our scripts:

| Metric Type   | Reproduced Models |
| ------------- | ----------------------------- |
| FR            | wadiqam_fr  |
| NR            | cnniqa, dbcnn, hyperiqa, metaiqa, wadiqam_nr |
| Aesthetic IQA | nima, nima-vgg16-ava      |

>[!NOTE]
>- Due to optimized training process, performance of some retrained approaches may be different with original paper.
>- Results of all retrained models by ours are normalized to [0, 1] and change to higher better for convenience.
>- Results of KonIQ-10k, AVA are both tested with official split.
>- NIMA is only applicable to AVA dataset now. We use inception_resnet_v2 for default nima.
>- MUSIQ is not included in the IAA benchmark because we do not have train/split information of the official model.

Benchmark Performance with Provided Script
Here is an example script to get performance benchmark on different datasets:

NOTE: this script will test ALL specified metrics on ALL specified datasets

Test default metrics on default datasets

Test with your own options

python benchmark_results.py --metric_opt options/example_benchmark_metric_opts.yml tid2013 tid2008

python benchmark_results.py --metric_opt options/example_benchmark_metric_opts.yml --data_opt options/example_benchmark_data_opts.yml

## :hammer_and_wrench: Train

Example Train Script
Example to train DBCNN on LIVEChallenge dataset

train for single experiment

train N splits for small datasets

Example for distributed training

## :electric_plug: Plugin Authoring (Experimental)
You can add custom datasets/architectures/models/losses/metrics from an external package via Python entry points.
1) Add entry points in your plugin pyproject.toml

[project.entry-points."pyiqa.models"] my_models = "my_pyiqa_plugin.register:register_models"

[project.entry-points."pyiqa.archs"] my_archs = "my_pyiqa_plugin.register:register_archs"

[project.entry-points."pyiqa.datasets"] my_datasets = "my_pyiqa_plugin.register:register_datasets"

[project.entry-points."pyiqa.losses"] my_losses = "my_pyiqa_plugin.register:register_losses"

[project.entry-points."pyiqa.metrics"] my_metrics = "my_pyiqa_plugin.register:register_metrics"

### 2) Minimal registration function

my_pyiqa_plugin/register.py

def register_models(): from .my_model import MyModel

# Registers with key "MyModel" by default. MODEL_REGISTRY.register(MyModel)

After installing your plugin package, PyIQA will discover and load these entry points lazily when a registry is queried.

:beers: Contribution
Any contributions to this repository are greatly appreciated. Please follow the contribution instructions for contribution guidance.
:scroll: License
This work is licensed under the PolyForm Noncommercial License 1.0.0 (see LICENSE) and the NTU S-Lab License for applicable components.
:bookmark_tabs: Citation
If you find our codes helpful to your research, please consider to use the following citation:

Please also consider to cite our works on image quality assessment if it is useful to you:

@article{wu2024qalign,
  title={Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels},
  author={Wu, Haoning and Zhang, Zicheng and Zhang, Weixia and Chen, Chaofeng and Li, Chunyi and Liao, Liang and Wang, Annan and Zhang, Erli and Sun, Wenxiu and Yan, Qiong and Min, Xiongkuo and Zhai, Guangtai and Lin, Weisi},
  journal={International Conference on Machine Learning (ICML)},
  year={2024},
  institution={Nanyang Technological University and Shanghai Jiao Tong University and Sensetime Research},
  note={Equal Contribution by Wu, Haoning and Zhang, Zicheng. Project Lead by Wu, Haoning. Corresponding Authors: Zhai, Guangtai and Lin, Weisi.}
}

:heart: Acknowledgement

The code architecture is borrowed from BasicSR. Several implementations are taken from: IQA-optimization, Image-Quality-Assessment-Toolbox, piq, piqa, clean-fid

We also thanks the following public repositories: MUSIQ, DBCNN, NIMA, HyperIQA, CNNIQA, WaDIQaM, PieAPP, paq2piq, MANIQA , MetaIQA

:e-mail: Contact