VoiceBench: Benchmarking LLM-Based Voice Assistants

🏆 Leaderboard | 📄 Paper | 🤗 Data

We encourage new result submissions through the issue tracker. The leaderboard will be updated accordingly.

News

• 2026.04.20 Check out HalluAudio, a comprehensive benchmark for hallucination detection in LALMs.
• 2025.04.20 Released wildvoice, a crowd-sourced dataset comprising human-recorded speech with diverse accents.
• 2025.04.12 Released bbh, a crowd-sourced dataset comprising human-recorded speech, for evaluating the reasoning ability of voice assistants.
• 2024.12.11 Updated the VoiceBench Leaderboard to include mmsu.
• 2024.12.10 Added a curated list of awesome voice assistants.
• 2024.11.24 Expanded the test samples in VoiceBench to include mmsu, covering 12 diverse domains from mmlu-pro.
• 2024.11.12 Updated the VoiceBench Leaderboard to include: 1) Mini-Omni2, GPT-4o-Audio, and Whisper-v3+GPT-4o, and 2) multiple-choice QA from OpenBookQA.
• 2024.10.30 Expanded the test samples in VoiceBench to include: 1) the complete set of open-ended QA from alpacaeval, and 2) multiple-choice QA from openbookqa.

Table of Contents

• Setup
• Dataset
• Evaluation
• Awesome Voice Assistants
• Citation

Setup

conda create -n voicebench python=3.10
conda activate voicebench
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121
pip install xformers==0.0.23 --no-deps
pip install -r requirements.txt

Dataset

The data used in this project is available at VoiceBench Dataset hosted on Hugging Face.

You can access it directly via the link and integrate it into your project by using the Hugging Face datasets library.

How to Use the Dataset

To load the dataset in your Python environment:

from datasets import load_dataset
Load the VoiceBench dataset
Available subset: alpacaeval, commoneval, sd-qa, ifeval, advbench, ...
dataset = load_dataset("hlt-lab/voicebench", 'alpacaeval')

Available Data

| Subset | # Samples | Audio Source | Task Type | |-----------------|:---------:|:------------:|:---------------------:| | alpacaeval | 199 | Google TTS | Open-Ended QA | | alpacaeval_full | 636 | Google TTS | Open-Ended QA | | commoneval | 200 | Human | Open-Ended QA | | wildvoice | 1,000 | Human | Open-Ended QA | | openbookqa | 455 | Google TTS | Multiple-Choice QA | | mmsu | 3,074 | Google TTS | Multiple-Choice QA | | sd-qa | 553 | Human | Reference-Based QA | | mtbench | 46 | Google TTS | Multi-Turn QA | | ifeval | 345 | Google TTS | Instruction Following | | bbh | 1,000 | Human | Reasoning | | advbench | 520 | Google TTS | Safety |

PS: alpacaeval contains helpful_base and vicuna data, while alpacaeval_full is constructed with the complete data. alpacaeval_full is used in the leaderboard.

Evaluation

Step 1: Get the Voice Assistant's Response

python main.py --model naive --data alpacaeval --split test --modality audio

Supported Arguments:

• --model: Specifies the model to use for generating responses. Replace naive with the model you want to test (e.g., qwen2, diva).
• --data: Selects the subset of the dataset. Replace alpacaeval with other subsets like commoneval, sd-qa, etc., depending on your evaluation needs.
• --split: Chooses the data split to evaluate.

• --modality: Use audio for spoken instructions, text for text-based instructions.

Step2: Automatic GPT-4 Evaluation

python api_judge.py --src_file naive-alpacaeval-test-audio.jsonl

Note: This step should be skipped for other datasets, as they are not evaluated using GPT-4.

Step3: Get the Final Results

python evaluate.py --src_file result-naive-alpacaeval-test-audio.jsonl --evaluator open

• --evaluator: Specifies the evaluator type:

Awesome Voice Assistants

Citation

@article{chen2024voicebench,
  title={VoiceBench: Benchmarking LLM-Based Voice Assistants},
  author={Chen, Yiming and Yue, Xianghu and Zhang, Chen and Gao, Xiaoxue and Tan, Robby T. and Li, Haizhou},
  journal={arXiv preprint arXiv:2410.17196},
  year={2024}
}