Lhotse is a Python library aiming to make multimodal (speech, audio, video, image, text) data preparation flexible and accessible to a wider community. Alongside k2, it is a part of the next generation Kaldi speech processing library.

Tutorial presentations and materials

• (_Interspeech 2023_) Tutorial notebook
• (_Interspeech 2023_) Tutorial slides
• (_Interspeech 2021_) Recorded lecture (3h)

About

Main goals (updated for 2025)

• Scale to multimodal data pipelines including audio, text, image, and video modalities.
• Provide state-of-the-art dataloading algorithms such as dataset blending and efficient on-the-fly bucketing.
• Handle data randomization (or de-duplication) for distributed multi-node training.
• Attract a wider community to multimodal processing tasks with a Python-centric design.
• Provide standard data preparation recipes for commonly used corpora.
• Flexible data preparation for model training with the notion of audio/video cuts.
• Support for efficient sequential I/O data formats such as Lhotse Shar (similar to webdataset).

Tutorials

We offer the following tutorials available in examples directory:

• Basic complete Lhotse workflow
• Transforming data with Cuts
• WebDataset integration
• How to combine multiple datasets
• Lhotse Shar: storage format optimized for sequential I/O and modularity
• Image and Video Support in Lhotse

• Icefall recipes: where k2 and Lhotse meet.
• Minimal ESPnet+Lhotse example:

!image

Lhotse introduces the notion of audio cuts, designed to ease the training data construction with operations such as mixing, truncation and padding that are performed on-the-fly to minimize the amount of storage required. Data augmentation and feature extraction are supported both in pre-computed mode, with feature matrices stored on disk (optionally using lilcom-compressed backends for better storage efficiency), and on-the-fly mode that computes the transformations upon request. Additionally, Lhotse introduces feature-space cut mixing to make the best of both worlds.

!image

Installation

Lhotse supports Python version 3.10 and later.

Pip

Lhotse is available on PyPI:

pip install lhotse

To install the latest, unreleased version, do:

pip install git+https://github.com/lhotse-speech/lhotse

Development installation

For development installation, you can fork/clone the GitHub repo and install with pip:

git clone https://github.com/lhotse-speech/lhotse cd lhotse pip install -e '.[dev]' pre-commit install # installs pre-commit hooks with style checks

# Running unit tests pytest test

# Running linter checks pre-commit run

This is an editable installation (-e option), meaning that your changes to the source code are automatically reflected when importing lhotse (no re-install needed). The [dev] part means you're installing extra dependencies that are used to run tests, build documentation or launch jupyter notebooks.

Environment variables

Lhotse uses several environment variables to customize it's behavior. They are as follows:

• LHOTSE_REQUIRE_TORCHAUDIO - when it's set and not any of 1|True|true|yes, we'll not check for torchaudio being installed and remove it from the requirements. It will disable many functionalities of Lhotse but the basic capabilities will remain (including reading audio with soundfile).
• LHOTSE_AUDIO_DURATION_MISMATCH_TOLERANCE - used when we load audio from a file and receive a different number of samples than declared in Recording.num_samples. This is sometimes necessary because different codecs (or even different versions of the same codec) may use different padding when decoding compressed audio. Typically values up to 0.1, or even 0.3 (second) are still reasonable, and anything beyond that indicates a serious issue.
• LHOTSE_AUDIO_BACKEND - may be set to any of the values returned from CLI lhotse list-audio-backends to override the default behavior of trial-and-error and always use a specific audio backend.
• LHOTSE_IO_BACKEND - may be set to any of the values returned from CLI lhotse list-io-backends to override how Lhotse opens paths, URLs, and URIs via open_best() (for example when reading manifests or URL-backed AudioSources). The same list is also available in Python via lhotse.available_io_backends().
• LHOTSE_S3_LOCAL_MIRROR_ROOTS - optional os.pathsep-separated local roots (: on Unix, ; on Windows) used to resolve URL-backed audio without rewriting s3://bucket/key paths in manifests. Roots are searched in order using the layout //; archive-member URLs require only the local archive to exist. Missing objects retain their original URI and use the configured remote backend. AISBatchLoader excludes successfully mirrored audio from AIStore requests.
• LHOTSE_RESAMPLING_BACKEND - may be set to any of the value returned from CLI lhotse list-resampling-backends to override the default behaviour.
• LHOTSE_FEATURES_STORAGE_BACKEND - may be set to any valid feature storage backend name (e.g. numpy_files, lilcom_chunky) to override the default feature storage backend (which is numpy_files). Use lhotse.available_storage_backends() to inspect the currently usable choices, or lhotse.storage_backend_statuses() / CLI lhotse list-storage-backends for a full list that also marks unavailable backends with install hints. If you have lilcom installed and want smaller feature archives, lilcom_chunky is the preferred choice.
• LHOTSE_AUDIO_LOADING_EXCEPTION_VERBOSE - when set to 1 we'll emit full exception stack traces when every available audio backend fails to load a given file (they might be very large).
• LHOTSE_DILL_ENABLED - when it's set to 1|True|true|yes, we will enable dill-based serialization of CutSet and Sampler across processes (it's disabled by default even when dill is installed).
• LHOTSE_LEGACY_OPUS_LOADING - (=1) reverts to a legacy OPUS loading mechanism that triggered a new ffmpeg subprocess for each OPUS file.
• LHOTSE_PREPARING_RELEASE - used internally by developers when releasing a new version of Lhotse.
• TORCHAUDIO_USE_BACKEND_DISPATCHER - when set to 1 and torchaudio version is below 2.1, we'll enable the experimental ffmpeg backend of torchaudio.
• AIS_ENDPOINT is read by AIStore client to determine AIStore endpoint URL. Required for AIStore dataloading.
• AIS_CONNECT_TIMEOUT - used by AIStore SDK to set the connection timeout (in seconds) for AIStore client requests. Set to 0 to disable (no timeout). If not set, the SDK default is used (3s).
• AIS_READ_TIMEOUT - used by AIStore SDK to set the read timeout (in seconds) for AIStore client requests. Set to 0 to disable (no timeout). If not set, the SDK default is used (20s).
• RANK, WORLD_SIZE, WORKER, and NUM_WORKERS are internally used to inform Lhotse Shar dataloading subprocesses.
• READTHEDOCS is internally used for documentation builds.
• LHOTSE_MSC_OVERRIDE_PROTOCOLS - when set, it will override your input protocols before feeding to MSCIOBackend. Useful when you don't want to change your existing url format but want to use MSCIOBackend. For example, if you have s3://s3-bucket/path/to/my/object and gs://gs-bucket/path/to/my/object, you can set LHOTSE_MSC_OVERRIDE_PROTOCOLS=s3,gs to override the urls to msc://s3-bucket/path/to/my/object and msc://gs-bucket/path/to/my/object.
• LHOTSE_MSC_PROFILE - when set, it will override the your bucket name before feeding to MSCIOBackend. Useful when your msc profile is not the same as your bucket name. For example, if you have s3://s3-bucket/path/to/my/object, you can set LHOTSE_MSC_OVERRIDE_PROTOCOLS=s3 and LHOTSE_MSC_PROFILE=msc-s3-profile to override the url to msc://msc-s3-profile/path/to/my/object.
• LHOTSE_MSC_BACKEND_FORCED - when set to True, forces Lhotse to use MSCIOBackend for all URLs. Use with caution as functionality may break if MSC does not support the provided URL format.

Optional dependencies

Other pip packages. You can leverage optional features of Lhotse by installing the relevant supporting package:

• pip install lhotse[lilcom] to enable lilcom-compressed feature and array storage backends. If storage efficiency is important, lilcom_chunky is the preferred feature-storage backend once this dependency is installed.
• torchcodec (>= 0.9, requires torch >= 2.9) is supported as an audio backend when detected. It is a PyTorch-native audio decoder built on FFmpeg. Install it via pip install torchcodec. When installed, it takes precedence over torchaudio in the default backend chain.
• torchaudio used to be a core dependency in Lhotse, but is now optional. Refer to official PyTorch documentation for installation.
• pip install lhotse[kaldi] for a maximal feature set related to Kaldi compatibility. It includes libraries such as kaldi_native_io (a more efficient variant of kaldi_io) and kaldifeat that port some of Kaldi functionality into Python.
• pip install lhotse[orjson] for up to 50% faster reading of JSONL manifests.
• pip install lhotse[webdataset]. We support "compiling" your data into WebDataset tarball format for more effective IO. You can still interact with the data as if it was a regular lazy CutSet. To learn more, check out the following tutorial:
• pip install h5py if you want to extract speech features and store them as HDF5 arrays.
• pip install dill. When dill is installed, we'll use it to pickle CutSet that uses a lambda function in calls such as .map or .filter. This is helpful in PyTorch DataLoader with num_jobs>0. Without dill, depending on your environment, you'll see an exception or a hanging script.
• pip install aistore to read manifests, tar fles, and other data from AIStore using AIStore-supported URLs (set AIS_ENDPOINT environment variable to activate it). See AIStore documentation for more details.
• pip install smart_open to read and write manifests and data in any location supported by smart_open (e.g. cloud, http).
• pip install opensmile for feature extraction using the OpenSmile toolkit's Python wrapper.
• pip install multi-storage-client for read and write manifests and data in different storage backends. See multi-storage-client for more details.

# CLI lhotse install-sph2pipe

# Python from lhotse.tools import install_sph2pipe install_sph2pipe()

It will download it to ~/.lhotse/tools, compile it, and auto-register in PATH. The program should be automatically detected and used by Lhotse.

Examples

We have example recipes showing how to prepare data and load it in Python as a PyTorch Dataset. They are located in the examples directory.

A short snippet to show how Lhotse can make audio data preparation quick and easy:

from torch.utils.data import DataLoader
from lhotse import CutSet, Fbank
from lhotse.dataset import VadDataset, SimpleCutSampler
from lhotse.recipes import prepare_switchboard
Prepare data manifests from a raw corpus distribution.
The RecordingSet describes the metadata about audio recordings;
the sampling rate, number of channels, duration, etc.
The SupervisionSet describes metadata about supervision segments:
the transcript, speaker, language, and so on.
swbd = prepare_switchboard('/export/corpora3/LDC/LDC97S62')
CutSet is the workhorse of Lhotse, allowing for flexible data manipulation.
We create 5-second cuts by traversing SWBD recordings in windows.
No audio data is actually loaded into memory or stored to disk at this point.
cuts = CutSet.from_manifests(
    recordings=swbd['recordings'],
    supervisions=swbd['supervisions']
).cut_into_windows(duration=5)
We compute the log-Mel filter energies and store them on disk;
Then, we pad the cuts to 5 seconds to ensure all cuts are of equal length,
as the last window in each recording might have a shorter duration.
The padding will be performed once the features are loaded into memory.
cuts = cuts.compute_and_store_features(
    extractor=Fbank(),
    storage_path='feats',
    num_jobs=8
).pad(duration=5.0)
Construct a Pytorch Dataset class for Voice Activity Detection task:
dataset = VadDataset()
sampler = SimpleCutSampler(cuts, max_duration=300)
dataloader = DataLoader(dataset, sampler=sampler, batch_size=None)
batch = next(iter(dataloader))

The VadDataset will yield a batch with pairs of feature and supervision tensors such as the following - the speech starts roughly at the first second (100 frames):

!image

Acknowledgment

Some contributions to this project were supported by National Science Foundation CCRI award 2120435.