MLX-VLM

• Installation
• Usage

• 1-bit Affine Inference
• Activation Quantization (CUDA)
• Multi-Image Chat Support

• Model-Specific Documentation
• Vision Feature Caching
• TurboQuant KV Cache
• Distributed Inference
• Fine-tuning

Model-Specific Documentation

Some models have detailed documentation with prompt formats, examples, and best practices:

| Model | Documentation | |-------|---------------| | LFM3 / LFM3-VL | Docs | | LFM3-Audio | Docs | | LFM3-Omni | Docs | | DeepSeek-OCR | Docs | | DeepSeek-OCR-2 | Docs | | Unlimited-OCR | Docs | | DOTS-OCR | Docs | | DOTS-MOCR | Docs | | ERNIE 4.5 VL | Docs | | GLM-OCR | Docs | | Phi-4 Reasoning Vision | Docs | | MiniCPM-o | Docs | | PaddleOCR-VL | Docs | | OXR | Docs | | Phi-4 Multimodal | Docs | | MolmoPoint | Docs | | LocateAnything | Docs | | Moondream2 | Docs | | Moondream3 | Docs | | EmbeddingGemma 2 | Docs | | Gemma 4 | Docs | | Muse Spark Open | Docs | | MiniMax M3 / M3.1 Flash | Docs | | Falcon-OCR | Docs | | IndicOCR | Docs | | PP-DocLayoutV3 | Docs | | Granite Vision 3.2 | Docs | | Granite Vision (4.0, 4.1) | Docs | | MiniCPM-OCR 1.0 | Docs | | MiniCPM-OCR Jev | Docs | | MiniCPM-V 4.7 | Docs | | MiniCPM-V 4.6 | Docs | | GLiNER2.5 | Docs | | BERT | Docs | | LLaVA-OneVision | Docs | | K2-Horizon | Docs | | Z1T-0 | Docs | | Spark-X2.5 | Docs | | Ternary Bonsai 2 | Docs | | Flint | Docs | | Nemotron 3.5 Super VL / Bolt / Nano Omni | Docs | | Idol (image to 3D Gaussian splats) | Docs |

Installation

The easiest way to get started is to install the mlx-vlm package using pip:

pip install -U mlx-vlm

The Gradio chat UI needs an extra dependency that is not part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as zsh, do not treat [ui] as a glob pattern.

Agent Skills

This repo ships an agent-skills bundle under skills/ for common MLX-VLM workflows — usage, conversion, development, and support. Skills load into a coding agent (Claude Code, Codex, Gemini) so it follows the right project conventions instead of guessing.

| Skill | Description | |-------|-------------| | cli-inference | Run and debug command-line inference (mlx_vlm.generate) — text/image/audio inputs and image-generation flags. | | server-inference | Run and debug the local server across the models, chat, responses, messages, audio, image, cache, and metrics endpoints. | | convert-quantize | Convert and quantize Hugging Face models to MLX (mlx_vlm.convert) — bits/group size, quant modes, RTN/AWQ, mixed recipes. | | add-new-model | Port a new architecture into mlx_vlm/models — config, weight-name mapping, reuse a similar model, add a test class. | | benchmarking | Produce credible, reproducible perf numbers and fork-vs-main A/B tables for PRs. | | contributing | Shape a change to pass review — code/config/test placement, pre-commit hooks, and PR expectations. | | hf-cache-models | List cached, custom, and loaded models through the server's model discovery endpoint. | | reproducible-github-issues | Turn CLI or server failures into concise, reproducible GitHub issues. |

Validate the bundle at any time:

python3 skills/scripts/validate_skills.py

Install from a local checkout:

# Claude Code
/plugin marketplace add /path/to/mlx-vlm
/plugin install mlx-vlm-skills@mlx-vlm
Codex CLI
codex plugin marketplace add /path/to/mlx-vlm
codex plugin add mlx-vlm-skills@mlx-vlm
Gemini CLI
gemini extensions install /path/to/mlx-vlm/skills

Usage

Command Line Interface (CLI)

Generate output from a model using the CLI:

# Text generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"
Image generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg
Audio understanding
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav
Multi-modal understanding (Image + Audio)
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wav
Speech generation
mlx_vlm.generate --model openbmb/MiniCPM-o-4_5 --output-modality audio --prompt "Say hello." --ref-audio /path/to/voice.wav --output speech.wav --max-tokens 256
Image to 3D Gaussian splats
mlx_vlm.generate --model speridlabs/idol --output-modality 3d --image object.png --output object.ply

Thinking Budget

For thinking models (e.g., Qwen3.5), you can limit the number of tokens spent in the thinking block:

mlx_vlm.generate --model mlx-community/Qwen3.5-2B-4bit \
  --thinking-budget 50 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>" \
  --enable-thinking \
  --prompt "Solve 2+2"

| Flag | Description | |------|-------------| | --enable-thinking | Activate thinking mode in the chat template | | --thinking-budget | Max tokens allowed inside the thinking block | | --thinking-start-token | Token that opens a thinking block (default: ) | | --thinking-end-token | Token that closes a thinking block (default: ) |

When the budget is exceeded, the model is forced to emit \n and transition to the answer. If --enable-thinking is passed but the model's chat template does not support it, the budget is applied only if the model generates the start token on its own.

On the server, thinking mode is disabled by default. Start the server with --enable-thinking to make thinking mode the default for requests that do not specify it:

mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

You can also set server defaults for the thinking budget and delimiter tokens:

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"

Requests can override the server defaults with enable_thinking, thinking_budget, thinking_start_token, or thinking_end_token. chat_template_kwargs.enable_thinking is also accepted; an explicit top-level enable_thinking takes precedence.

Speculative Decoding

Speed up generation by drafting several candidate tokens with a small "drafter" model and verifying them in a single target forward pass. Three drafter families are supported.

| Flag | Description | |------|-------------| | --draft-model | HuggingFace repo or local path for the drafter | | --draft-kind | Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP) | | --draft-block-size | Override the drafter's configured block size |

See docs/usage.md for Python API examples including batch generation.

DFlash, DFlash2, and DSpark

A lightweight block-diffusion drafter that predicts multiple tokens per round, typically 2–3× faster.

# Text generation with speculative decoding
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0 --enable-thinking
Also works with images
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --image examples/images/cats.jpg \
  --prompt "Describe this image." \
  --max-tokens 256 --temperature 0 --enable-thinking
Server with speculative decoding
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash

DFlash2 adds dynamic convolutions and a candidate-path selector. The published Qwen3.8-27B checkpoint is auto-detected and uses the shared exact DFlash target verification path. For the fastest quantized setup, convert the drafter to 4-bit; the verifier adapts between three and five rows from recent acceptance:

mlx_vlm.convert --hf-path z-lab/Qwen3.8-27B-DFlash2 \
  --mlx-path Qwen3.8-27B-DFlash2-4bit \
  --quantize --q-bits 4 --q-group-size 64
mlx_vlm.generate --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0
mlx_vlm.server --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit

Liquid AI's DSpark checkpoint uses a Qwen3-style block drafter plus a learned Markov correction head. It is auto-detected and runs through the exact target verification path:

mlx_vlm.generate --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark \
  --prompt "Explain speculative decoding in three sentences." \
  --max-tokens 256 --temperature 0
mlx_vlm.server --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark

The published DSpark block_size: 9 means nine proposals, or ten target rows after adding the anchor token. On MLX, DSpark verifies seven proposals plus the anchor by default: eight rows exactly fill the verifier threadgroup, while nine or ten rows pad to sixteen and run slower. The trained width remains available with --draft-block-size 10. The checkpoint's confidence head is loaded for parity, and DSpark decoding currently requires greedy sampling (temperature=0).

Muse Glimmer's published assistant checkpoint is auto-detected as DFlash:

mlx_vlm.generate --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0
mlx_vlm.server --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant

DFlash draft-cache windowing is available from the Python API. During speculative decoding the target model still verifies every proposed token with its full KV cache; this knob only changes the DFlash drafter cache. When draft_window_size is set, the drafter keeps at most that many recent committed tokens in its own KV cache instead of attending over the full generated prefix. That reduces draft-side cache length and memory, but it can lower acceptance because the drafter has less context than the target verifier. On MLX, the full draft cache is usually faster for Qwen3.5 DFlash, so windowing defaults to None; set it only when you want to experiment with this compact recent-token cache tradeoff:

from mlx_vlm import load
from mlx_vlm.generate import generate
from mlx_vlm.speculative.drafters import load_drafter
model, processor = load("Qwen/Qwen3.5-4B")
draft_model, draft_kind = load_drafter("z-lab/Qwen3.5-4B-DFlash")
draft_model.config.draft_window_size = 256  # None disables windowing
result = generate(
    model,
    processor,
    "Write a quicksort in Python.",
    max_tokens=512,
    temperature=0,
    draft_model=draft_model,
    draft_kind=draft_kind,
)

Gemma 4 MTP

Multi-Token Prediction: Google's 4-layer "assistant" drafter that shares K/V with the target and drafts multiple tokens autoregressively from a constant position. Pass --draft-kind mtp to dispatch the MTP round-loop.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0
Server
mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4

Supported pairings (target ↔ drafter):

| Target | Drafter | |---------------------------------|------------------------------------------| | mlx-community/gemma-4-E2B-it-bf16 | mlx-community/gemma-4-E2B-it-assistant-bf16 | | mlx-community/gemma-4-E4B-it-bf16 | mlx-community/gemma-4-E4B-it-assistant-bf16 | | mlx-community/gemma-4-26B-A4B-it-bf16 | mlx-community/gemma-4-26B-A4B-it-assistant-bf16 | | mlx-community/gemma-4-31B-it-bf16 | mlx-community/gemma-4-31B-it-assistant-bf16 |

Measured speedups (greedy, byte-identical output): up to 3.94× on 26B-A4B and 2.29× on 31B at B=4. See mlx_vlm/speculative/drafters/gemma4_assistant/README.md for full sweeps and architecture notes.

Gemma 4 EAGLE-3

EAGLE-3 drafts from three target hidden-state captures with a lightweight one-layer speculator. The Red Hat Speculators checkpoint auto-detects as --draft-kind eagle3.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0
Server
mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3

MiniMax M3 EAGLE-3

MiniMax M3 supports the released Inferact/MiniMax-M3-EAGLE3 drafter. Convert the target with mlx_vlm.convert because mlx_lm.convert does not know the minimax_m3_vl model type.

mlx_vlm.convert \
  --hf-path MiniMaxAI/MiniMax-M3 \
  --mlx-path ~/MiniMax-M3-4bit \
  --quantize --q-bits 4 \
  --trust-remote-code
mlx_vlm.convert \
  --hf-path Inferact/MiniMax-M3-EAGLE3 \
  --mlx-path ~/MiniMax-M3-EAGLE3
mlx_vlm.generate \
  --model ~/MiniMax-M3-4bit \
  --draft-model ~/MiniMax-M3-EAGLE3 \
  --draft-kind eagle3 \
  --draft-block-size 3 \
  --prompt "Explain MiniMax Sparse Attention in one paragraph." \
  --max-tokens 256 --temperature 0

The public MiniMax M3 BF16 checkpoint advertises MTP metadata but does not publish mtp or nextn tensors, so use the released EAGLE-3 drafter for that checkpoint.

MiniMax M3 also supports image/video prompts, MiniMax thinking tags, MiniMax tool-call parsing, MSA index caches, and MXFP8 config loading. See mlx_vlm/models/minimax_m3_vl/README.md for model-specific conversion and runtime notes.

Reading the server's draft counters

With a drafter loaded, mlx_vlm.server adds four fields to a response's timings, named after llama.cpp's:

| Field | Meaning | |---|---| | draft_kind | The drafter family that ran | | draft_rounds | Verification rounds: target forward passes after the prefill | | draft_n | Tokens proposed, --draft-block-size minus one per round (the block includes the anchor token) | | draft_n_accepted | Proposed tokens the target's greedy choice matched |

draft_n_accepted is a measure of the drafter, not of the output. A round is recorded when the target verifies it, before the stop token or max_tokens cuts the reply, so matches past the end of the reply are counted and the field can exceed predicted_n. predicted_n - draft_n_accepted is therefore not the number of target forward passes. Use draft_rounds:

accept_length = (predicted_n - 1) / draft_rounds

The first token comes from the prefill, before any draft exists, so it is left out of the numerator. The three counters are differences of a tally kept on the drafter, so they belong to one request only when one request is in flight; with concurrent requests they are shared across the batch.

Chat UI with Gradio

The Gradio chat UI requires the optional ui extra, which the base mlx-vlm install does not include:

pip install -U 'mlx-vlm[ui]'

Then launch the chat interface:

mlx_vlm.chat_ui --model mlx-community/Qwen2-VL-2B-Instruct-4bit

Python Script

Here's an example of how to use MLX-VLM in a Python script:

import mlx.core as mx
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load the model
model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = load_config(model_path)
Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
image = [Image.open("...")] can also be used with PIL.Image.Image objects
prompt = "Describe this image."
Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=len(image)
)
Generate output
output = generate(model, processor, formatted_prompt, image, verbose=False)
print(output)

Audio Example

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load model with audio support
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
Prepare audio input
audio = ["/path/to/audio1.wav", "/path/to/audio2.mp3"]
prompt = "Describe what you hear in these audio files."
Apply chat template with audio
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_audios=len(audio)
)
Generate output with audio
output = generate(model, processor, formatted_prompt, audio=audio, verbose=False)
print(output)

Multi-Modal Example (Image + Audio)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load multi-modal model
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
Prepare inputs
image = ["/path/to/image.jpg"]
audio = ["/path/to/audio.wav"]
prompt = ""
Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt,
    num_images=len(image),
    num_audios=len(audio)
)
Generate output
output = generate(model, processor, formatted_prompt, image, audio=audio, verbose=False)
print(output)

Decision models

Run shared typed decision prediction from the CLI:

python -m mlx_vlm decide --model nativ-community/decider-2b \
  --state "Please refund my duplicate charge" \
  --questions '{"department":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":["billing","technical","sales"]}}'

Use --state-file request.txt for UTF-8 text and --questions-file questions.json for named questions. mlx_vlm.decide runs the same command.

Models that read images take a repeatable --image, and models that read speech take one --audio (each a path or URL). The state is optional when media are given:

python -m mlx_vlm decide --model LiquidAI/d1-omni-600M \
  --image http://images.cocodataset.org/val2017/000000039769.jpg \
  --questions '{"cats":{"type":"choice","instructions":"How many cats are there?","criteria":["one","two","more"]}}'
python -m mlx_vlm decide --model LiquidAI/d1-omni-600M \
  --audio https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac \
  --questions '{"speaking":{"type":"bool","instructions":"Is anyone speaking in this clip?"}}'

Preload a decision model in the existing server:

python -m mlx_vlm server --decision-model nativ-community/decider-2b
curl http://localhost:8080/v1/decisions \
  -H 'Content-Type: application/json' \
  -d '{"state":"Please refund my duplicate charge","questions":{"department":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":["billing","technical","sales"]}}}'

An explicit model in the request selects a checkpoint; otherwise the currently loaded decision model is used. Without either, the endpoint returns HTTP 400. Requests may add "images", a list of URLs, paths, or data URIs, or "audio", one URL, path, or base64 data URI; state may then be null. Decision requests are non-streaming and use the server's API-key authentication.

Both interfaces return the shared predict() result and preserve model-specific scoring. Decider, Laya, d1-3B, and d1-omni-600M support choice, bool, and score; the two LiquidAI d1 models also read images, and d1-omni reads speech. Consult each model's README for criteria and calibration details.

Server (FastAPI)

Start the server:

mlx_vlm.server --port 8080

Preload a model at startup (Hugging Face repo or local path)
mlx_vlm.server --model <hf_repo_or_local_path>
Preload separate model kinds at startup
mlx_vlm.server --model <language_model> \
  --image-model <image_generation_model> \
  --tts-model <text_to_speech_model> \
  --stt-model <speech_to_text_model>
Preload a model with adapter
mlx_vlm.server --model <hf_repo_or_local_path> --adapter-path <adapter_path>
With trust remote code enabled (required for some models)
mlx_vlm.server --trust-remote-code
Enable thinking mode by default for requests that do not override it
mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking
Configure thinking defaults at startup
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"
Require bearer authentication for API endpoints
mlx_vlm.server --api-key <secret-token>
Include local models outside the Hugging Face cache
mlx_vlm.server --model-dir /Volumes/Models --model-dir ~/my-custom-model

Server Options

• --model: Preload a language model at server startup, accepts a Hugging Face repo ID or local path (optional, loads lazily on first request if omitted)
• --image-model: Preload an image generation model at server startup
• --mesh-model: Preload an image-to-3D model for asynchronous PLY generation
• --tts-model: Preload a text-to-speech model at server startup
• --stt-model: Preload a speech-to-text model at server startup
• --embedding-model: Preload an embedding model at server startup
• --decision-model: Preload a decision model at server startup
• --reranker-model: Preload a supported reranker model at server startup
• --model-dir: Additional model folder or parent containing model folders to include in discovery; repeat for multiple paths. Overrides MLX_VLM_MODEL_PATHS (paths separated by os.pathsep, : on macOS/Linux)
• --adapter-path: Path for adapter weights to use with the preloaded model
• --draft-model: Speculative drafter path or HF id (e.g. z-lab/Qwen3.8-27B-DFlash2, z-lab/Qwen3.5-4B-DFlash, RedHatAI/gemma-4-31B-it-speculator.eagle3, google/gemma-4-31B-it-assistant, Inferact/MiniMax-M3-EAGLE3) — enables speculative decoding for ~2× or higher throughput
• --draft-kind: Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP)
• --draft-block-size: Override the drafter's configured block size
• --host: Host address (default: 0.0.0.0)
• --port: Port number (default: 8080)
• --trust-remote-code: Trust remote code when loading models from Hugging Face Hub
• --enable-thinking: Enable thinking mode by default for requests that do not set enable_thinking
• --thinking-budget: Default maximum number of tokens allowed inside a thinking block
• --thinking-start-token: Default token that opens a thinking block
• --thinking-end-token: Default token that closes a thinking block (--thinking-eos-token is also accepted)
• --kv-bits: Number of bits for KV cache quantization (e.g. 8 for uniform, 3.5 for TurboQuant)
• --kv-quant-scheme: KV cache quantization backend (uniform or turboquant)
• --kv-key-bits / --kv-value-bits: Override the bit-width for keys or values individually (see Per-tensor KV quantization)
• --kv-key-scheme / --kv-value-scheme: Override the quantization backend for keys or values individually
• --kv-group-size: Group size for uniform KV cache quantization (default: 64)
• --max-kv-size: Maximum KV cache size in tokens
• --vision-cache-size: Max number of cached vision features (default: 20)
• --min-pixels / --max-pixels: Default minimum/maximum input image pixel area for processors that support these options (also MLX_VLM_MIN_PIXELS / MLX_VLM_MAX_PIXELS). Unset values use the model's preprocessing defaults.
• --log-progress-interval: Decoded tokens between progress log messages; 0 disables periodic decode progress (default: 10)
• --api-key: Bearer token required for inference, model discovery, and management endpoints
• --log-level: Logging level — DEBUG, INFO, WARNING, ERROR, CRITICAL (default: INFO)

For higher-resolution image input, including OCR with Bonsai 2, set pixel-area defaults at startup:

mlx_vlm.server --model prism-ml/Ternary-Bonsai-2-2B-mlx-2bit \
  --min-pixels 1048576 --max-pixels 4194304

Chat Completions, Responses, and Messages requests can override these defaults with top-level min_pixels and max_pixels fields (use extra_body with an OpenAI client). For example, "min_pixels": 1048576 requests at least 1024 × 1024 pixels of area, preserving aspect ratio and rounding to the processor's patch size. On Bonsai 2 this corresponds to approximately 1024 image tokens. Higher resolution uses more context, memory, and processing time. Both limits must be positive and the effective minimum must not exceed the maximum. Omitted or null request values inherit the server defaults.

Change these defaults on a running server without restarting or reloading the model:

curl -X PATCH http://localhost:8080/v1/settings \
  -H 'Content-Type: application/json' \
  -d '{"min_pixels": 1048576, "max_pixels": 4194304}'

Use GET /v1/settings to inspect the current values. New requests inherit the updated limits; requests already queued for generation keep their resolved options, and per-request overrides still take precedence. The pair is updated atomically; invalid limits appear in rejected and leave both limits unchanged. Set either setting to null to use that limit from the model's preprocessing configuration. Updates last until the server restarts; the {"op": "replace", "values": {}} form resets all runtime settings to their startup defaults, including CLI pixel limits. Include Authorization: Bearer when the server has an API key configured.

The existing resize_shape request field also applies before image processing in the batching server. Processor pixel limits still apply afterward. Token counting and streaming context checks use the same options as generation.

At INFO, the server logs request start/completion, chunked-prefill progress, time to first token, periodic decode throughput, and the final token counts. Set --log-level DEBUG to emit decode progress for every token and add its token number, token ID, and decoded text to the same log entry. Decode progress uses rate for the instantaneous inter-token rate; decode completion uses the same field name for aggregate decode throughput measured across completed token intervals.

OpenAI-compatible streaming responses expose throughput under timings.predicted_per_second. Token-bearing SSE chunks report the instantaneous inter-token rate, while terminal and usage chunks report the aggregate rate as (tokens - 1) / (last_token_time - first_token_time). The first token reports null because it has no preceding token interval.

You can also set trust remote code via environment variable:

MLX_TRUST_REMOTE_CODE=true mlx_vlm.server

The server provides multiple endpoints for different use cases and supports dynamic model loading/unloading with caching (one model at a time).

Continuous Batching

The server supports continuous batching for higher throughput when handling multiple concurrent requests. New requests join the active batch immediately without waiting for existing requests to finish, and mixed batches of image and text-only requests are supported.

Continuous batching is enabled automatically when the server loads a model. You can pre-load a model at startup so it's ready to serve immediately:

mlx_vlm.server --port 8080 --model mlx-community/Qwen2.5-VL-3B-Instruct-4bit

Verify via the health endpoint:

curl http://localhost:8080/health
{"status":"healthy","loaded_model":"...","apc_enabled":false}

If --model is omitted, the model is loaded on the first request.

Automatic Prefix Caching (APC)

Automatic Prefix Caching reuses model cache state across requests that share the same prefix. It is useful for repeated long documents, long chat histories, or retrieval contexts where each request appends a short new suffix.

APC builds a cache plan from model.make_cache() in the same style as vLLM's hybrid cache manager: each layer gets a cache spec, compatible specs form cache groups, and one coordinator selects a common reusable prefix across the groups. Dense attention models use pageable K/V blocks. Hybrid full/sliding-attention, recurrent/SSM, MLA/composite, VLM, and Omni layouts use restorable state checkpoints for the components that cannot be concatenated safely. Generation code uses the same coordinator API for both paths.

APC has two tiers:

• Warm memory: keeps reusable APCBlock tensors in process memory. This is the fastest path, but it keeps both the reusable block pool and the runtime KVCache.
• Warm disk: persists cached prefixes as safetensors shards so they survive process restarts. Warm-disk reads build the layer-major prompt cache directly without promoting restored blocks into the APCBlock pool; writes can still populate both memory and disk tiers.

Python Script

Use APCManager directly when calling stream_generate:

from pathlib import Path
from mlx_vlm import load, stream_generate
from mlx_vlm.apc import APCManager, DiskBlockStore
from mlx_vlm.prompt_utils import apply_chat_template
model_id = "Qwen/Qwen3-VL-4B-Instruct"
model, processor = load(model_id)
disk = DiskBlockStore(
    Path("~/.cache/mlx-vlm/caching").expanduser(),
    namespace=model_id,
    max_bytes=3 * (1 << 30),  # 3 GB disk cap; use None for uncapped
)
apc = APCManager(num_blocks=4096, block_size=16, disk=disk)
Optional diagnostics: inspect the automatically inferred cache groups.
print(apc.coordinator(model).plan.describe())
document = Path("long_document.txt").read_text()
try:
    # First request computes the full prefix and stores reusable K/V blocks.
    prompt1 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nSummarize the key decisions.",
        num_images=0,
    )
    for _ in stream_generate(
        model, processor, prompt1, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        pass
# Second request shares the same document prefix and only prefills the suffix.
    prompt2 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nList the open engineering risks.",
        num_images=0,
    )
    for chunk in stream_generate(
        model, processor, prompt2, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        print(chunk.text, end="", flush=True)
print(apc.stats_snapshot())
finally:
    apc.close()

To compare cold, warm-memory, warm-disk, and disk-eviction behavior with a model, use the same direct API path:

import os
import tempfile
import time
from pathlib import Path
from mlx_vlm import load, stream_generate
from mlx_vlm.apc import APCManager, DiskBlockStore
from mlx_vlm.prompt_utils import apply_chat_template
model_id = "Qwen/Qwen3-VL-4B-Instruct"
contexts = [8000, 20000, 50000, 100000]
disk_cap_gb = 0  # 0 means uncapped
shard_max_blocks = 256
context_sweep_max_tokens = 1  # one token is enough to measure prefill reuse
test_prompt_tokens = 8000
fill_prompts = 80
eviction_disk_cap_gb = 3.0
os.environ["APC_DISK_SHARD_MAX_BLOCKS"] = str(shard_max_blocks)
model, processor = load(model_id)
tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor
def disk_cap_bytes(gb: float):
    return None if gb <= 0 else int(gb * (1 << 30))
def make_context(target_tokens: int, seed: int = 0) -> str:
    line = (
        f"Document {seed}: APC benchmark content with deterministic facts, "
        "dates, identifiers, and repeated technical notes.\n"
    )
    line_tokens = max(1, len(tokenizer.encode(line, add_special_tokens=False)))
    text = line * max(1, target_tokens // line_tokens)
    while len(tokenizer.encode(text, add_special_tokens=False)) < target_tokens:
        text += line
    return text
def make_prompt(context: str, question: str) -> str:
    return apply_chat_template(
        processor,
        model.config,
        prompt=f"{context}\n\n{question}",
        num_images=0,
    )
def run_once(apc: APCManager, context: str, question: str, max_tokens: int = 32):
    prompt = make_prompt(context, question)
    apc.reset_stats()
last = None
    output = []
    start = time.perf_counter()
    for chunk in stream_generate(
        model,
        processor,
        prompt,
        max_tokens=max_tokens,
        temperature=0.0,
        apc_manager=apc,
    ):
        output.append(chunk.text)
        last = chunk
if last is None:
        raise RuntimeError("generation returned no chunks")
return {
        "wall_s": time.perf_counter() - start,
        "prompt_tokens": last.prompt_tokens,
        "prompt_tps": last.prompt_tps,
        "generation_tps": last.generation_tps,
        "apc": apc.stats_snapshot(),
        "text": "".join(output).strip(),
    }
def print_result(label: str, result: dict) -> None:
    stats = result["apc"]
    print(
        f"{label:<12} "
        f"prompt_tokens={result['prompt_tokens']:>7} "
        f"prompt_tps={result['prompt_tps']:>8.1f} "
        f"gen_tps={result['generation_tps']:>7.1f} "
        f"matched={stats.get('matched_tokens', 0):>7} "
        f"disk_hits={stats.get('disk_hits', 0):>5} "
        f"disk_evictions={stats.get('disk_evictions', 0):>5}"
    )
def open_apc(cache_root: Path, namespace: str, disk_gb: float) -> APCManager:
    disk = DiskBlockStore(
        cache_root,
        namespace=namespace,
        max_bytes=disk_cap_bytes(disk_gb),
    )
    return APCManager(num_blocks=4096, block_size=16, disk=disk)
def run_context_sweep() -> None:
    print("cold / warm-memory / warm-disk")
    with tempfile.TemporaryDirectory() as tmp:
        cache_root = Path(tmp)
        for target_tokens in contexts:
            context = make_context(target_tokens)
            namespace = f"{model_id}-context-{target_tokens}"
            apc = open_apc(cache_root, namespace, disk_cap_gb)
            try:
                print(f"\ncontext ~= {target_tokens} text tokens")
                print_result(
                    "cold",
                    run_once(
                        apc,
                        context,
                        "Summarize the key decisions.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
                print_result(
                    "warm-memory",
                    run_once(
                        apc,
                        context,
                        "List the open engineering risks.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
            finally:
                # Closing waits for queued disk writes before reopening the disk tier.
                apc.close()
apc = open_apc(cache_root, namespace, disk_cap_gb)
            try:
                print_result(
                    "warm-disk",
                    run_once(
                        apc,
                        context,
                        "Extract the implementation timeline.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
            finally:
                apc.close()
def run_disk_eviction_workload() -> None:
    print("\ndisk eviction workload")
    with tempfile.TemporaryDirectory() as tmp:
        cache_root = Path(tmp)
        namespace = f"{model_id}-eviction"
        test_context = make_context(test_prompt_tokens, seed=0)
apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
        try:
            print_result(
                "seed",
                run_once(apc, test_context, "Summarize the retained test prefix."),
            )
        finally:
            apc.close()
apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
        try:
            for i in range(fill_prompts):
                fill_context = make_context(test_prompt_tokens, seed=i + 1)
                run_once(
                    apc,
                    fill_context,
                    f"Summarize filler document {i + 1}.",
                    max_tokens=1,
                )
                if (i + 1) % 10 == 0:
                    stats = apc.stats_snapshot()
                    print(
                        f"filled={i + 1:>3} "
                        f"disk_gb={stats.get('disk_bytes', 0) / (1 << 30):.2f} "
                        f"disk_evictions={stats.get('disk_evictions', 0)}"
                    )
        finally:
            apc.close()
apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
        try:
            print_result(
                "post-fill",
                run_once(
                    apc,
                    test_context,
                    "Check whether the retained test prefix still restores.",
                ),
            )
        finally:
            apc.close()
run_context_sweep()
run_disk_eviction_workload()

Server

Enable in-memory APC for the server with environment variables:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

APC works with KV-cache quantization (--kv-bits):

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --kv-bits 8 --port 8080

APC persists caches to disk by default when enabled, under $MLX_VLM_CACHE_HOME/apc (or ~/.cache/mlx-vlm/apc), with a 20 GiB cap per model namespace. Customize the location and cap:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
APC_DISK_PATH=~/.cache/mlx-vlm/caching \
APC_DISK_MAX_GB=3 \
APC_DISK_SHARD_MAX_BLOCKS=256 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

Repeated requests with the same long prefix will hit APC automatically:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "X-APC-Tenant: demo" \
  -d '{
    "model": "Qwen/Qwen3-VL-4B-Instruct",
    "messages": [{
      "role": "user",
      "content": "Paste a long shared document here.\n\nNow answer question A."
    }],
    "max_tokens": 128
  }'

Use the same X-APC-Tenant value for requests that may share cached prefixes. Use different tenant values to isolate cache entries between users or workspaces.

Inspect and reset APC state:

curl http://localhost:8080/v1/cache/stats
curl -X POST http://localhost:8080/v1/cache/reset

restored_tokens counts prefix tokens successfully materialized for generation across block and exact caches, in memory or on disk. A lookup alone does not increase it, nor does a failed batch restore. A successful restore remains counted if the request later fails or is cancelled. The previous served_tokens block-storage count is now stored_tokens; token_hit_rate retains its existing calculation, matched_tokens / (matched_tokens + stored_tokens).

Configure APC on a running server with PATCH /v1/settings. Use GET /v1/settings to discover supported settings and read their current values:

curl http://localhost:8080/v1/settings
curl -X PATCH http://localhost:8080/v1/settings \
  -H 'Content-Type: application/json' \
  -d '{
    "apc_enabled": true,
    "apc_disk_enabled": true,
    "apc_memory_max_gb": 2,
    "apc_disk_max_gb": 20,
    "apc_checkpoint_interval_tokens": 1024
  }'

The endpoint also accepts apc_memory_reserve_gb, apc_disk_queue_max_gb, apc_disk_path, apc_disk_shard_max_blocks, apc_block_size, apc_num_blocks, apc_checkpoint_entries, and apc_checkpoint_guard_tokens. When MLX_VLM_SERVER_API_KEY is configured, include Authorization: Bearer .

Settings apply on the next text-generation request through the existing model reload path; the server process stays running. Reloading clears resident caches and retires the previous disk writer after its generation worker finishes. Existing disk files remain available. Updating APC options while APC is disabled stages them for the next enable. The response reports applied, rejected, reload_kinds, and the resulting settings; invalid sizes are rejected without changing those values. Unchanged settings do not trigger a reload.

Use null for automatic memory budgets or the default disk path/cap. Zero has specific meanings: apc_memory_max_gb: 0 retains caches only on disk, apc_disk_max_gb: 0 removes the disk cap, and apc_disk_queue_max_gb: 0 writes synchronously. An empty apc_disk_path or apc_disk_enabled: false disables persistence. PATCH merges with current settings; the optional {"op": "replace", "values": {...}} form first restores startup environment defaults. /v1/cache/stats shows the active manager after the next request.

Common APC environment variables:

| Variable | Default | Description | |----------|---------|-------------| | APC_ENABLED | 0 | Set to 1 to enable APC | | APC_NUM_BLOCKS | 2048 | Number of in-memory APC blocks | | APC_BLOCK_SIZE | 16 | Tokens per APC block | | APC_CHECKPOINT_ENTRIES | 2 | In-memory checkpoint entries; also bounds snapshots captured per hybrid prompt (disk-only mode captures two) | | APC_CHECKPOINT_GUARD_TOKENS | 1 | Tokens retained after a reusable hybrid checkpoint boundary; the default preserves the normal final-token prefill boundary | | APC_CHECKPOINT_INTERVAL_TOKENS | 2048 | Spacing of intermediate hybrid checkpoints, rounded up to a multiple of APC_BLOCK_SIZE; observed prompt divergence takes priority within the entry budget; 0 keeps only the final checkpoint | | APC_MEMORY_MAX_GB | auto | Resident block and checkpoint budget in GiB: 10% of Metal's recommended working set, capped at 8 GiB; 0 retains caches only on disk | | APC_MEMORY_RESERVE_GB | auto | Additional memory headroom in GiB: 10% of Metal's recommended working set, at least 1 GiB | | APC_DISK_ENABLED | 1 | Set to 0 to disable disk persistence | | MLX_VLM_CACHE_HOME | ~/.cache/mlx-vlm | Base cache directory; APC uses its apc subdirectory unless APC_DISK_PATH is set | | APC_DISK_PATH | cache directory above | Directory for persistent disk shards; an empty value disables persistence | | APC_DISK_MAX_GB | 20 | Disk cap per model namespace in GiB; 0 means uncapped | | APC_DISK_QUEUE_MAX_GB | 1 | Maximum tensor bytes held by queued disk writes in GiB; larger writes run synchronously; 0 makes all writes synchronous | | APC_DISK_SHARD_MAX_BLOCKS | 256 | Max blocks per disk segment shard | | APC_MAX_POOL_TENSORS | 450000 | Stops adding memory blocks before the Metal resource limit; disk writes continue | | APC_LAYER_MAJOR_MEMORY_MIN_TOKENS | 50000 | Store long warm-memory prefixes as compact layer-major snapshots instead of per-block tensors | | APC_HASH | fast | Set to sha256 for a stable cryptographic hash | | APC_TRACE | unset | Set to 1 for greppable store/reject/self-check log lines |

Custom cache layouts can opt in without APC model-name checks by implementing prefix_cache_snapshot() and prefix_cache_restore(snapshot). In-tree dense, sliding-window, recurrent, composite, VLM, and Omni cache layouts are detected automatically. APC works with --kv-bits (including TurboQuant): the live KV cache stays quantized; pageable APC K/V blocks are stored as dequantized float K/V, so block-pool size does not shrink with quant. When APC is enabled on the server, a non-fatal layout self-check runs at model load.

Requests with a shared document and different questions can reuse their common prefix. Dense K/V caches, including compact layer-major memory snapshots, match complete blocks before the first differing token. Hybrid, recurrent and sliding window caches must restore a state captured before that divergence: their final state cannot be rolled back by slicing K/V tensors.

Hybrid prefill now captures intermediate checkpoints as well as the final guard checkpoint, in streaming, continuous batching and DiffusionGemma generation. For text API requests with transient_context, the server verifies and prioritizes the token boundary before that mutable suffix. Otherwise, observed divergence from resident prompts takes priority. With neither hint, each prompt stores its latest 2,048-token boundary before the final checkpoint and the final checkpoint itself. For example, a 30,000-token document followed by 20 instruction tokens can reuse 28,672 tokens when the instructions change. The same checkpoints persis

... (README truncated for length)