VisionCaptioner

VisionCaptioner is a local desktop application designed to automate the creation of detailed captions for image & video datasets.

Built specifically for AI researchers and enthusiasts training custom models (LoRA, Fine-tuning, Flux, Z-Image Turbo, Qwen-Image, SDXL, Wan, HunyuanVideo etc). VisionCaptioner leverages Vision-Language Models from the Qwen-VL and Google Gemma 4 families to generate high-quality, context-aware descriptions in batch.

!Screenshot VisonCaptioner

✨ Features

• User-Friendly Interface: Simple GUI to manage your image & video captioning tasks.
• Caption generation Automatically generate captions using Qwen-VL or Google Gemma 4 models.

• Masking Support: Functionality to create mask files using Segment Anything 3 model with promptable subject.

• Video Extraction Functionality to extract frames from videos containing a prompted subject.
• Quality Assurance Analyze images for blur, resolution, face/eyes detection, and missing masks.

• Local Execution: Runs entirely on your machine for privacy and control.
• CommandLine Interface: Option to use this from the commandline and/or scripts.

🛠️ Installation

Prerequisites

• Python 3.10+
• NVIDIA GPU (with CUDA).
• AMD GPU (with ROCm on Linux)
• Apple Silicon Mac (M-series, with Metal / MPS)

Windows Setup

git clone https://github.com/Brekel/VisionCaptioner.git
cd VisionCaptioner
python -m venv venv
.\venv\Scripts\activate
python -m pip install --upgrade pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt

AMD GPU with ROCm support for Windows, more info here: https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/windows/install-pytorch.html

Linux Setup

Python Setup:

git clone https://github.com/Brekel/VisionCaptioner.git
cd VisionCaptioner
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu130
pip3 install -r requirements.txt

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1

Some Linux systems may require additional packages for the Qt-based GUI to work:

sudo apt install libxcb-cursor0 libxcb-xinerama0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-shape0 libxcb-xfixes0 libxcb-xkb1 libxkbcommon-x11-0

macOS Setup (Apple Silicon)

Runs on M-series Macs using Metal / MPS. Install the default PyTorch wheels (they ship MPS support, so there is no CUDA index URL):

git clone https://github.com/Brekel/VisionCaptioner.git
cd VisionCaptioner
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
pip install torch torchvision
pip install -r requirements.txt

• NVIDIA/CUDA-only packages (bitsandbytes, nvidia-ml-py, triton) are skipped automatically on macOS, so the Int8/NF4 quantization options are unavailable (the app loads full precision instead). Use None (BF16) or FP16, or a GGUF model.
• Transformers models run on MPS. For GGUF models, use the built-in installer on the Captions tab (📥 button): it fetches a prebuilt Metal llama-cpp-python wheel (pip install llama-cpp-python does not work — see readme_models.md).
• Launch with python main.py or ./run.sh.

Troubleshooting: inconsistent Name error while installing PyTorch

If pip install torch torchvision --index-url ... fails with:

Requested typing-extensions>=4.10.0 ... has inconsistent Name:
expected 'typing-extensions', but metadata has 'typing_extensions'
...
ERROR: Cannot install torch==... because these package versions have conflicting dependencies.

your pip is too old: it compares the raw package name instead of normalizing it, and typing_extensions / typing-extensions are the same package. A fresh venv seeds whatever pip shipped with your Python (Python 3.10 ships pip 23.0.1), which is why this hits on a clean install. Upgrading pip inside the venv is the fix, so do not skip this line:

python -m pip install --upgrade pip

If you cannot upgrade pip, pre-install the affected packages from PyPI before the torch command:

pip install typing-extensions jinja2

Do not work around it by switching --index-url to --extra-index-url. pip does not prioritize one index over the other — it merges both and picks the highest version, so you will silently get the CPU-only torch from PyPI instead of the CUDA/ROCm build.

Update

• Windows: double-click update.bat (or run it from a terminal)
• Linux / macOS: ./update.sh (you may need to chmod +x update.sh once)

cd VisionCaptioner
git pull
venv\Scripts\activate          # Windows
source venv/bin/activate     # Linux / macOS
pip install --upgrade -r requirements.txt

🚀 Launching the Application

Start manually:

python main.py

📥 Download/Install Models

• Models can be installed using the built-in downloads manager on the Captions tab (📥💾 button).
• Alternatively, you can manually download models from HuggingFace into the /models folder.
• More info in the readme_models.md file.

📖 General Usage

• First select your Image/Video folder at the top (using Browse button or drag & drop)
• Note that all settings have a tooltip description if you hover your mouse over them
• Use the Captions tab to generate captions for images and videos
• Use the Review & Edit tab to review and edit captions
• Use the Video Extraction tab to extract frames from videos
• Use the Mask Segmentation tab to create masks for images (this is optional)
• Use the Quality Assurance tab to analyze and curate image quality in your dataset

🖥️ CLI - Command Line Interface usage

• For advanced users, the tool can be used from the commandline or scripts
• It will automatically read the settings.json file generated by the user
• Parameters can be overruled on the commandline
• Please refer to commandline_interface.md for documentation

Acknowledgements

Google DeepMind - For developing and open-sourcing the Gemma 4 family of multimodal models.

Meta AI - For developing and open-sourcing the Segment Anything Model 3 (SAM3).

This project was inspired by:

AI Lab's ComfyUI-QwenVL node for ComfyUI:

• https://github.com/1038lab/ComfyUI-QwenVL

• https://github.com/Nerogar/OneTrainer

🙏 Citation & Support

• Giving a ⭐ on GitHub.
• Citing the project as: Brekel - VisionCaptioner (https://brekel.com).
• Follow on Twitter/X: https://x.com/brekelj
• Checking out other tools at brekel.com.