MLX-VLM

• Installation
• Usage

• 1-bit Affine Inference
• Activation Quantization (CUDA)
• Multi-Image Chat Support

• Model-Specific Documentation
• Vision Feature Caching
• TurboQuant KV Cache
• Distributed Inference
• Fine-tuning

Model-Specific Documentation

Some models have detailed documentation with prompt formats, examples, and best practices:

| Model | Documentation | |-------|---------------| | DeepSeek-OCR | Docs | | DeepSeek-OCR-2 | Docs | | Unlimited-OCR | Docs | | DOTS-OCR | Docs | | DOTS-MOCR | Docs | | ERNIE 4.5 VL | Docs | | GLM-OCR | Docs | | Phi-4 Reasoning Vision | Docs | | MiniCPM-o | Docs | | PaddleOCR-VL | Docs | | Phi-4 Multimodal | Docs | | MolmoPoint | Docs | | LocateAnything | Docs | | Moondream2 | Docs | | Moondream3 | Docs | | Gemma 4 | Docs | | MiniMax M3 | Docs | | Falcon-OCR | Docs | | IndicOCR | Docs | | PP-DocLayoutV3 | Docs | | Granite Vision 3.2 | Docs | | Granite 4.0 Vision | Docs | | MiniCPM-V 4.6 | Docs | | GLiNER2.5 | Docs | | LLaVA-OneVision | Docs | | K2-Horizon | Docs | | Z1T-0 | Docs | | Spark-X2.5 | Docs |

Installation

The easiest way to get started is to install the mlx-vlm package using pip:

pip install -U mlx-vlm

The Gradio chat UI needs an extra dependency that is not part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as zsh, do not treat [ui] as a glob pattern.

Agent Skills

This repo ships an agent-skills bundle under skills/ for common MLX-VLM workflows — usage, conversion, development, and support. Skills load into a coding agent (Claude Code, Codex, Gemini) so it follows the right project conventions instead of guessing.

| Skill | Description | |-------|-------------| | cli-inference | Run and debug command-line inference (mlx_vlm.generate) — text/image/audio inputs and image-generation flags. | | server-inference | Run and debug the local server across the models, chat, responses, messages, audio, image, cache, and metrics endpoints. | | convert-quantize | Convert and quantize Hugging Face models to MLX (mlx_vlm.convert) — bits/group size, quant modes, RTN/AWQ, mixed recipes. | | add-new-model | Port a new architecture into mlx_vlm/models — config, weight-name mapping, reuse a similar model, add a test class. | | benchmarking | Produce credible, reproducible perf numbers and fork-vs-main A/B tables for PRs. | | contributing | Shape a change to pass review — code/config/test placement, pre-commit hooks, and PR expectations. | | hf-cache-models | List MLX-VLM-supported (and, with --check-arch, loadable) models in the local Hugging Face cache. | | reproducible-github-issues | Turn CLI or server failures into concise, reproducible GitHub issues. |

Validate the bundle at any time:

python3 skills/scripts/validate_skills.py

Install from a local checkout:

# Claude Code
/plugin marketplace add /path/to/mlx-vlm
/plugin install mlx-vlm-skills@mlx-vlm
Codex CLI
codex plugin marketplace add /path/to/mlx-vlm
codex plugin add mlx-vlm-skills@mlx-vlm
Gemini CLI
gemini extensions install /path/to/mlx-vlm/skills

Usage

Command Line Interface (CLI)

Generate output from a model using the CLI:

# Text generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"
Image generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg
Audio understanding
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav
Multi-modal understanding (Image + Audio)
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wav
Speech generation
mlx_vlm.generate --model openbmb/MiniCPM-o-4_5 --output-modality audio --prompt "Say hello." --ref-audio /path/to/voice.wav --output speech.wav --max-tokens 256

Thinking Budget

For thinking models (e.g., Qwen3.5), you can limit the number of tokens spent in the thinking block:

mlx_vlm.generate --model mlx-community/Qwen3.5-2B-4bit \
  --thinking-budget 50 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>" \
  --enable-thinking \
  --prompt "Solve 2+2"

| Flag | Description | |------|-------------| | --enable-thinking | Activate thinking mode in the chat template | | --thinking-budget | Max tokens allowed inside the thinking block | | --thinking-start-token | Token that opens a thinking block (default: ) | | --thinking-end-token | Token that closes a thinking block (default: ) |

When the budget is exceeded, the model is forced to emit \n and transition to the answer. If --enable-thinking is passed but the model's chat template does not support it, the budget is applied only if the model generates the start token on its own.

On the server, thinking mode is disabled by default. Start the server with --enable-thinking to make thinking mode the default for requests that do not specify it:

mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

You can also set server defaults for the thinking budget and delimiter tokens:

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"

Requests can override the server defaults with enable_thinking, thinking_budget, thinking_start_token, or thinking_end_token.

Speculative Decoding

Speed up generation by drafting several candidate tokens with a small "drafter" model and verifying them in a single target forward pass. Three drafter families are supported.

| Flag | Description | |------|-------------| | --draft-model | HuggingFace repo or local path for the drafter | | --draft-kind | Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP) | | --draft-block-size | Override the drafter's configured block size |

See docs/usage.md for Python API examples including batch generation.

DFlash, DFlash2, and DSpark

A lightweight block-diffusion drafter that predicts multiple tokens per round, typically 2–3× faster.

# Text generation with speculative decoding
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0 --enable-thinking
Also works with images
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --image examples/images/cats.jpg \
  --prompt "Describe this image." \
  --max-tokens 256 --temperature 0 --enable-thinking
Server with speculative decoding
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash

DFlash2 adds dynamic convolutions and a candidate-path selector. The published Qwen3.8-27B checkpoint is auto-detected and uses the shared exact DFlash target verification path. For the fastest quantized setup, convert the drafter to 4-bit; the verifier adapts between three and five rows from recent acceptance:

mlx_vlm.convert --hf-path z-lab/Qwen3.8-27B-DFlash2 \
  --mlx-path Qwen3.8-27B-DFlash2-4bit \
  --quantize --q-bits 4 --q-group-size 64
mlx_vlm.generate --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0
mlx_vlm.server --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit

Liquid AI's DSpark checkpoint uses a Qwen3-style block drafter plus a learned Markov correction head. It is auto-detected and runs through the exact target verification path:

mlx_vlm.generate --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark \
  --prompt "Explain speculative decoding in three sentences." \
  --max-tokens 256 --temperature 0
mlx_vlm.server --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark

The published DSpark block_size: 9 means nine proposals, or ten target rows after adding the anchor token. On MLX, DSpark verifies seven proposals plus the anchor by default: eight rows exactly fill the verifier threadgroup, while nine or ten rows pad to sixteen and run slower. The trained width remains available with --draft-block-size 10. The checkpoint's confidence head is loaded for parity, and DSpark decoding currently requires greedy sampling (temperature=0).

Muse Glimmer's published assistant checkpoint is auto-detected as DFlash:

mlx_vlm.generate --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0
mlx_vlm.server --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant

DFlash draft-cache windowing is available from the Python API. During speculative decoding the target model still verifies every proposed token with its full KV cache; this knob only changes the DFlash drafter cache. When draft_window_size is set, the drafter keeps at most that many recent committed tokens in its own KV cache instead of attending over the full generated prefix. That reduces draft-side cache length and memory, but it can lower acceptance because the drafter has less context than the target verifier. On MLX, the full draft cache is usually faster for Qwen3.5 DFlash, so windowing defaults to None; set it only when you want to experiment with this compact recent-token cache tradeoff:

from mlx_vlm import load
from mlx_vlm.generate import generate
from mlx_vlm.speculative.drafters import load_drafter
model, processor = load("Qwen/Qwen3.5-4B")
draft_model, draft_kind = load_drafter("z-lab/Qwen3.5-4B-DFlash")
draft_model.config.draft_window_size = 256  # None disables windowing
result = generate(
    model,
    processor,
    "Write a quicksort in Python.",
    max_tokens=512,
    temperature=0,
    draft_model=draft_model,
    draft_kind=draft_kind,
)

Gemma 4 MTP

Multi-Token Prediction: Google's 4-layer "assistant" drafter that shares K/V with the target and drafts multiple tokens autoregressively from a constant position. Pass --draft-kind mtp to dispatch the MTP round-loop.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0
Server
mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4

Supported pairings (target ↔ drafter):

| Target | Drafter | |---------------------------------|------------------------------------------| | mlx-community/gemma-4-E2B-it-bf16 | mlx-community/gemma-4-E2B-it-assistant-bf16 | | mlx-community/gemma-4-E4B-it-bf16 | mlx-community/gemma-4-E4B-it-assistant-bf16 | | mlx-community/gemma-4-26B-A4B-it-bf16 | mlx-community/gemma-4-26B-A4B-it-assistant-bf16 | | mlx-community/gemma-4-31B-it-bf16 | mlx-community/gemma-4-31B-it-assistant-bf16 |

Measured speedups (greedy, byte-identical output): up to 3.94× on 26B-A4B and 2.29× on 31B at B=4. See mlx_vlm/speculative/drafters/gemma4_assistant/README.md for full sweeps and architecture notes.

Gemma 4 EAGLE-3

EAGLE-3 drafts from three target hidden-state captures with a lightweight one-layer speculator. The Red Hat Speculators checkpoint auto-detects as --draft-kind eagle3.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0
Server
mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3

MiniMax M3 EAGLE-3

MiniMax M3 supports the released Inferact/MiniMax-M3-EAGLE3 drafter. Convert the target with mlx_vlm.convert because mlx_lm.convert does not know the minimax_m3_vl model type.

mlx_vlm.convert \
  --hf-path MiniMaxAI/MiniMax-M3 \
  --mlx-path ~/MiniMax-M3-4bit \
  --quantize --q-bits 4 \
  --trust-remote-code
mlx_vlm.convert \
  --hf-path Inferact/MiniMax-M3-EAGLE3 \
  --mlx-path ~/MiniMax-M3-EAGLE3
mlx_vlm.generate \
  --model ~/MiniMax-M3-4bit \
  --draft-model ~/MiniMax-M3-EAGLE3 \
  --draft-kind eagle3 \
  --draft-block-size 3 \
  --prompt "Explain MiniMax Sparse Attention in one paragraph." \
  --max-tokens 256 --temperature 0

The public MiniMax M3 BF16 checkpoint advertises MTP metadata but does not publish mtp or nextn tensors, so use the released EAGLE-3 drafter for that checkpoint.

MiniMax M3 also supports image/video prompts, MiniMax thinking tags, MiniMax tool-call parsing, MSA index caches, and MXFP8 config loading. See mlx_vlm/models/minimax_m3_vl/README.md for model-specific conversion and runtime notes.

Chat UI with Gradio

The Gradio chat UI requires the optional ui extra, which the base mlx-vlm install does not include:

pip install -U 'mlx-vlm[ui]'

Then launch the chat interface:

mlx_vlm.chat_ui --model mlx-community/Qwen2-VL-2B-Instruct-4bit

Python Script

Here's an example of how to use MLX-VLM in a Python script:

import mlx.core as mx
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load the model
model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = load_config(model_path)
Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
image = [Image.open("...")] can also be used with PIL.Image.Image objects
prompt = "Describe this image."
Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=len(image)
)
Generate output
output = generate(model, processor, formatted_prompt, image, verbose=False)
print(output)

Audio Example

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load model with audio support
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
Prepare audio input
audio = ["/path/to/audio1.wav", "/path/to/audio2.mp3"]
prompt = "Describe what you hear in these audio files."
Apply chat template with audio
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_audios=len(audio)
)
Generate output with audio
output = generate(model, processor, formatted_prompt, audio=audio, verbose=False)
print(output)

Multi-Modal Example (Image + Audio)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config
Load multi-modal model
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config
Prepare inputs
image = ["/path/to/image.jpg"]
audio = ["/path/to/audio.wav"]
prompt = ""
Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt,
    num_images=len(image),
    num_audios=len(audio)
)
Generate output
output = generate(model, processor, formatted_prompt, image, audio=audio, verbose=False)
print(output)

Server (FastAPI)

Start the server:

mlx_vlm.server --port 8080

Preload a model at startup (Hugging Face repo or local path)
mlx_vlm.server --model <hf_repo_or_local_path>
Preload separate model kinds at startup
mlx_vlm.server --model <language_model> \
  --image-model <image_generation_model> \
  --tts-model <text_to_speech_model> \
  --stt-model <speech_to_text_model>
Preload a model with adapter
mlx_vlm.server --model <hf_repo_or_local_path> --adapter-path <adapter_path>
With trust remote code enabled (required for some models)
mlx_vlm.server --trust-remote-code
Enable thinking mode by default for requests that do not override it
mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking
Configure thinking defaults at startup
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"
Require bearer authentication for API endpoints
mlx_vlm.server --api-key <secret-token>
Opt into shared Hugging Face cache model discovery
mlx_vlm.server --model-discovery hf-cache

Server Options

• --model: Preload a language model at server startup, accepts a Hugging Face repo ID or local path (optional, loads lazily on first request if omitted)
• --image-model: Preload an image generation model at server startup
• --tts-model: Preload a text-to-speech model at server startup
• --stt-model: Preload a speech-to-text model at server startup
• --embedding-model: Preload an embedding model at server startup
• --reranker-model: Preload a supported reranker model at server startup
• --model-discovery: Models exposed by /v1/models; served lists only models loaded by this process (default), while hf-cache also scans the shared Hugging Face cache
• --adapter-path: Path for adapter weights to use with the preloaded model
• --draft-model: Speculative drafter path or HF id (e.g. z-lab/Qwen3.8-27B-DFlash2, z-lab/Qwen3.5-4B-DFlash, RedHatAI/gemma-4-31B-it-speculator.eagle3, google/gemma-4-31B-it-assistant, Inferact/MiniMax-M3-EAGLE3) — enables speculative decoding for ~2× or higher throughput
• --draft-kind: Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP)
• --draft-block-size: Override the drafter's configured block size
• --host: Host address (default: 0.0.0.0)
• --port: Port number (default: 8080)
• --trust-remote-code: Trust remote code when loading models from Hugging Face Hub
• --enable-thinking: Enable thinking mode by default for requests that do not set enable_thinking
• --thinking-budget: Default maximum number of tokens allowed inside a thinking block
• --thinking-start-token: Default token that opens a thinking block
• --thinking-end-token: Default token that closes a thinking block (--thinking-eos-token is also accepted)
• --kv-bits: Number of bits for KV cache quantization (e.g. 8 for uniform, 3.5 for TurboQuant)
• --kv-quant-scheme: KV cache quantization backend (uniform or turboquant)
• --kv-key-bits / --kv-value-bits: Override the bit-width for keys or values individually (see Per-tensor KV quantization)
• --kv-key-scheme / --kv-value-scheme: Override the quantization backend for keys or values individually
• --kv-group-size: Group size for uniform KV cache quantization (default: 64)
• --max-kv-size: Maximum KV cache size in tokens
• --vision-cache-size: Max number of cached vision features (default: 20)
• --log-progress-interval: Decoded tokens between progress log messages; 0 disables periodic decode progress (default: 10)
• --api-key: Bearer token required for inference, model discovery, and management endpoints
• --log-level: Logging level — DEBUG, INFO, WARNING, ERROR, CRITICAL (default: INFO)

OpenAI-compatible streaming responses expose throughput under timings.predicted_per_second. Token-bearing SSE chunks report the instantaneous inter-token rate, while terminal and usage chunks report the aggregate rate as (tokens - 1) / (last_token_time - first_token_time). The first token reports null because it has no preceding token interval.

You can also set trust remote code via environment variable:

MLX_TRUST_REMOTE_CODE=true mlx_vlm.server

The server provides multiple endpoints for different use cases and supports dynamic model loading/unloading with caching (one model at a time).

Continuous Batching

The server supports continuous batching for higher throughput when handling multiple concurrent requests. New requests join the active batch immediately without waiting for existing requests to finish, and mixed batches of image and text-only requests are supported.

Continuous batching is enabled automatically when the server loads a model. You can pre-load a model at startup so it's ready to serve immediately:

mlx_vlm.server --port 8080 --model mlx-community/Qwen2.5-VL-3B-Instruct-4bit

Verify via the health endpoint:

curl http://localhost:8080/health
{"status":"healthy","loaded_model":"...","apc_enabled":false}

If --model is omitted, the model is loaded on the first request.

Automatic Prefix Caching (APC)

Automatic Prefix Caching reuses model cache state across requests that share the same prefix. It is useful for repeated long documents, long chat histories, or retrieval contexts where each request appends a short new suffix.

APC builds a cache plan from model.make_cache() in the same style as vLLM's hybrid cache manager: each layer gets a cache spec, compatible specs form cache groups, and one coordinator selects a common reusable prefix across the groups. Dense attention models use pageable K/V blocks. Hybrid full/sliding-attention, recurrent/SSM, MLA/composite, VLM, and Omni layouts use restorable state checkpoints for the components that cannot be concatenated safely. Generation code uses the same coordinator API for both paths.

APC has two tiers:

• Warm memory: keeps reusable APCBlock tensors in process memory. This is the fastest path, but it keeps both the reusable block pool and the runtime KVCache.
• Warm disk: persists cached prefixes as safetensors shards so they survive process restarts. Warm-disk reads build the layer-major prompt cache directly without promoting restored blocks into the APCBlock pool; writes can still populate both memory and disk tiers.

Python Script

Use APCManager directly when calling stream_generate:

from pathlib import Path
from mlx_vlm import load, stream_generate
from mlx_vlm.apc import APCManager, DiskBlockStore
from mlx_vlm.prompt_utils import apply_chat_template
model_id = "Qwen/Qwen3-VL-4B-Instruct"
model, processor = load(model_id)
disk = DiskBlockStore(
    Path("~/.cache/mlx-vlm/caching").expanduser(),
    namespace=model_id,
    max_bytes=3 * (1 << 30),  # 3 GB disk cap; use None for uncapped
)
apc = APCManager(num_blocks=4096, block_size=16, disk=disk)
Optional diagnostics: inspect the automatically inferred cache groups.
print(apc.coordinator(model).plan.describe())
document = Path("long_document.txt").read_text()
try:
    # First request computes the full prefix and stores reusable K/V blocks.
    prompt1 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nSummarize the key decisions.",
        num_images=0,
    )
    for _ in stream_generate(
        model, processor, prompt1, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        pass
# Second request shares the same document prefix and only prefills the suffix.
    prompt2 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nList the open engineering risks.",
        num_images=0,
    )
    for chunk in stream_generate(
        model, processor, prompt2, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        print(chunk.text, end="", flush=True)
print(apc.stats_snapshot())
finally:
    apc.close()

To compare cold, warm-memory, warm-disk, and disk-eviction behavior with a model, use the same direct API path:

import os
import tempfile
import time
from pathlib import Path
from mlx_vlm import load, stream_generate
from mlx_vlm.apc import APCManager, DiskBlockStore
from mlx_vlm.prompt_utils import apply_chat_template
model_id = "Qwen/Qwen3-VL-4B-Instruct"
contexts = [8000, 20000, 50000, 100000]
disk_cap_gb = 0  # 0 means uncapped
shard_max_blocks = 256
context_sweep_max_tokens = 1  # one token is enough to measure prefill reuse
test_prompt_tokens = 8000
fill_prompts = 80
eviction_disk_cap_gb = 3.0
os.environ["APC_DISK_SHARD_MAX_BLOCKS"] = str(shard_max_blocks)
model, processor = load(model_id)
tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor
def disk_cap_bytes(gb: float):
    return None if gb <= 0 else int(gb * (1 << 30))
def make_context(target_tokens: int, seed: int = 0) -> str:
    line = (
        f"Document {seed}: APC benchmark content with deterministic facts, "
        "dates, identifiers, and repeated technical notes.\n"
    )
    line_tokens = max(1, len(tokenizer.encode(line, add_special_tokens=False)))
    text = line * max(1, target_tokens // line_tokens)
    while len(tokenizer.encode(text, add_special_tokens=False)) < target_tokens:
        text += line
    return text
def make_prompt(context: str, question: str) -> str:
    return apply_chat_template(
        processor,
        model.config,
        prompt=f"{context}\n\n{question}",
        num_images=0,
    )
def run_once(apc: APCManager, context: str, question: str, max_tokens: int = 32):
    prompt = make_prompt(context, question)
    apc.reset_stats()
last = None
    output = []
    start = time.perf_counter()
    for chunk in stream_generate(
        model,
        processor,
        prompt,
        max_tokens=max_tokens,
        temperature=0.0,
        apc_manager=apc,
    ):
        output.append(chunk.text)
        last = chunk
if last is None:
        raise RuntimeError("generation returned no chunks")
return {
        "wall_s": time.perf_counter() - start,
        "prompt_tokens": last.prompt_tokens,
        "prompt_tps": last.prompt_tps,
        "generation_tps": last.generation_tps,
        "apc": apc.stats_snapshot(),
        "text": "".join(output).strip(),
    }
def print_result(label: str, result: dict) -> None:
    stats = result["apc"]
    print(
        f"{label:<12} "
        f"prompt_tokens={result['prompt_tokens']:>7} "
        f"prompt_tps={result['prompt_tps']:>8.1f} "
        f"gen_tps={result['generation_tps']:>7.1f} "
        f"matched={stats.get('matched_tokens', 0):>7} "
        f"disk_hits={stats.get('disk_hits', 0):>5} "
        f"disk_evictions={stats.get('disk_evictions', 0):>5}"
    )
def open_apc(cache_root: Path, namespace: str, disk_gb: float) -> APCManager:
    disk = DiskBlockStore(
        cache_root,
        namespace=namespace,
        max_bytes=disk_cap_bytes(disk_gb),
    )
    return APCManager(num_blocks=4096, block_size=16, disk=disk)
def run_context_sweep() -> None:
    print("cold / warm-memory / warm-disk")
    with tempfile.TemporaryDirectory() as tmp:
        cache_root = Path(tmp)
        for target_tokens in contexts:
            context = make_context(target_tokens)
            namespace = f"{model_id}-context-{target_tokens}"
            apc = open_apc(cache_root, namespace, disk_cap_gb)
            try:
                print(f"\ncontext ~= {target_tokens} text tokens")
                print_result(
                    "cold",
                    run_once(
                        apc,
                        context,
                        "Summarize the key decisions.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
                print_result(
                    "warm-memory",
                    run_once(
                        apc,
                        context,
                        "List the open engineering risks.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
            finally:
                # Closing waits for queued disk writes before reopening the disk tier.
                apc.close()
apc = open_apc(cache_root, namespace, disk_cap_gb)
            try:
                print_result(
                    "warm-disk",
                    run_once(
                        apc,
                        context,
                        "Extract the implementation timeline.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
            finally:
                apc.close()
def run_disk_eviction_workload() -> None:
    print("\ndisk eviction workload")
    with tempfile.TemporaryDirectory() as tmp:
        cache_root = Path(tmp)
        namespace = f"{model_id}-eviction"
        test_context = make_context(test_prompt_tokens, seed=0)
apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
        try:
            print_result(
                "seed",
                run_once(apc, test_context, "Summarize the retained test prefix."),
            )
        finally:
            apc.close()
apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
        try:
            for i in range(fill_prompts):
                fill_context = make_context(test_prompt_tokens, seed=i + 1)
                run_once(
                    apc,
                    fill_context,
                    f"Summarize filler document {i + 1}.",
                    max_tokens=1,
                )
                if (i + 1) % 10 == 0:
                    stats = apc.stats_snapshot()
                    print(
                        f"filled={i + 1:>3} "
                        f"disk_gb={stats.get('disk_bytes', 0) / (1 << 30):.2f} "
                        f"disk_evictions={stats.get('disk_evictions', 0)}"
                    )
        finally:
            apc.close()
apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
        try:
            print_result(
                "post-fill",
                run_once(
                    apc,
                    test_context,
                    "Check whether the retained test prefix still restores.",
                ),
            )
        finally:
            apc.close()
run_context_sweep()
run_disk_eviction_workload()

Server

Enable in-memory APC for the server with environment variables:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

APC works with KV-cache quantization (--kv-bits):

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --kv-bits 8 --port 8080

APC persists caches to disk by default when enabled, under $MLX_VLM_CACHE_HOME/apc (or ~/.cache/mlx-vlm/apc), with a 20 GiB cap per model namespace. Customize the location and cap:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
APC_DISK_PATH=~/.cache/mlx-vlm/caching \
APC_DISK_MAX_GB=3 \
APC_DISK_SHARD_MAX_BLOCKS=256 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

Repeated requests with the same long prefix will hit APC automatically:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "X-APC-Tenant: demo" \
  -d '{
    "model": "Qwen/Qwen3-VL-4B-Instruct",
    "messages": [{
      "role": "user",
      "content": "Paste a long shared document here.\n\nNow answer question A."
    }],
    "max_tokens": 128
  }'

Use the same X-APC-Tenant value for requests that may share cached prefixes. Use different tenant values to isolate cache entries between users or workspaces.

Inspect and reset APC state:

curl http://localhost:8080/v1/cache/stats
curl -X POST http://localhost:8080/v1/cache/reset

Configure APC on a running server with PATCH /v1/settings. Use GET /v1/settings to discover supported settings and read their current values:

curl http://localhost:8080/v1/settings
curl -X PATCH http://localhost:8080/v1/settings \
  -H 'Content-Type: application/json' \
  -d '{
    "apc_enabled": true,
    "apc_disk_enabled": true,
    "apc_memory_max_gb": 2,
    "apc_disk_max_gb": 20,
    "apc_checkpoint_interval_tokens": 1024
  }'

The endpoint also accepts apc_memory_reserve_gb, apc_disk_queue_max_gb, apc_disk_path, apc_disk_shard_max_blocks, apc_block_size, apc_num_blocks, apc_checkpoint_entries, and apc_checkpoint_guard_tokens. When MLX_VLM_SERVER_API_KEY is configured, include Authorization: Bearer .

Settings apply on the next text-generation request through the existing model reload path; the server process stays running. Reloading clears resident caches and retires the previous disk writer after its generation worker finishes. Existing disk files remain available. Updating APC options while APC is disabled stages them for the next enable. The response reports applied, rejected, reload_kinds, and the resulting settings; invalid sizes are rejected without changing those values. Unchanged settings do not trigger a reload.

Use null for automatic memory budgets or the default disk path/cap. Zero has specific meanings: apc_memory_max_gb: 0 retains caches only on disk, apc_disk_max_gb: 0 removes the disk cap, and apc_disk_queue_max_gb: 0 writes synchronously. An empty apc_disk_path or apc_disk_enabled: false disables persistence. PATCH merges with current settings; the optional {"op": "replace", "values": {...}} form first restores startup environment defaults. /v1/cache/stats shows the active manager after the next request.

Common APC environment variables:

| Variable | Default | Description | |----------|---------|-------------| | APC_ENABLED | 0 | Set to 1 to enable APC | | APC_NUM_BLOCKS | 2048 | Number of in-memory APC blocks | | APC_BLOCK_SIZE | 16 | Tokens per APC block | | APC_CHECKPOINT_ENTRIES | 2 | In-memory checkpoint entries; also bounds snapshots captured per hybrid prompt (disk-only mode captures two) | | APC_CHECKPOINT_GUARD_TOKENS | 1 | Tokens retained after a reusable hybrid checkpoint boundary; the default preserves the normal final-token prefill boundary | | APC_CHECKPOINT_INTERVAL_TOKENS | 2048 | Spacing of intermediate hybrid checkpoints, rounded up to a multiple of APC_BLOCK_SIZE; 0 keeps only the final checkpoint | | APC_MEMORY_MAX_GB | auto | Resident block and checkpoint budget in GiB: 10% of Metal's recommended working set, capped at 8 GiB; 0 retains caches only on disk | | APC_MEMORY_RESERVE_GB | auto | Additional memory headroom in GiB: 10% of Metal's recommended working set, at least 1 GiB | | APC_DISK_ENABLED | 1 | Set to 0 to disable disk persistence | | MLX_VLM_CACHE_HOME | ~/.cache/mlx-vlm | Base cache directory; APC uses its apc subdirectory unless APC_DISK_PATH is set | | APC_DISK_PATH | cache directory above | Directory for persistent disk shards; an empty value disables persistence | | APC_DISK_MAX_GB | 20 | Disk cap per model namespace in GiB; 0 means uncapped | | APC_DISK_QUEUE_MAX_GB | 1 | Maximum tensor bytes held by queued disk writes in GiB; larger writes run synchronously; 0 makes all writes synchronous | | APC_DISK_SHARD_MAX_BLOCKS | 256 | Max blocks per disk segment shard | | APC_MAX_POOL_TENSORS | 450000 | Stops adding memory blocks before the Metal resource limit; disk writes continue | | APC_LAYER_MAJOR_MEMORY_MIN_TOKENS | 50000 | Store long warm-memory prefixes as compact layer-major snapshots instead of per-block tensors | | APC_HASH | fast | Set to sha256 for a stable cryptographic hash | | APC_TRACE | unset | Set to 1 for greppable store/reject/self-check log lines |

Custom cache layouts can opt in without APC model-name checks by implementing prefix_cache_snapshot() and prefix_cache_restore(snapshot). In-tree dense, sliding-window, recurrent, composite, VLM, and Omni cache layouts are detected automatically. APC works with --kv-bits (including TurboQuant): the live KV cache stays quantized; pageable APC K/V blocks are stored as dequantized float K/V, so block-pool size does not shrink with quant. When APC is enabled on the server, a non-fatal layout self-check runs at model load.

Requests with a shared document and different questions can reuse their common prefix. Dense K/V caches, including compact layer-major memory snapshots, match complete blocks before the first differing token. Hybrid, recurrent and sliding window caches must restore a state captured before that divergence: their final state cannot be rolled back by slicing K/V tensors.

Hybrid prefill now captures intermediate checkpoints as well as the final guard checkpoint, in streaming, continuous batching and DiffusionGemma generation. With the defaults, each prompt stores its latest 2,048-token boundary before the final checkpoint and the final checkpoint itself. For example, a 30,000-token document followed by 20 instruction tokens can reuse 28,672 tokens when the instructions change. The same checkpoints persist across server restarts when the disk tier is enabled.

Captures are bounded by APC_CHECKPOINT_ENTRIES to avoid copying the growing hybrid cache at every prefill chunk. More entries retain more earlier boundaries; a smaller interval gives finer reuse near the end. Reuse still requires a retained boundary before the divergence. Short prompts below the interval, divergence before the earliest retained checkpoint, and evicted checkpoints can miss. Media checkpoints include all media tokens so the remaining suffix is text. Changing checkpoint boundaries can change floating-point execution shapes, as with other chunked or cached prefill paths.

The resident byte budget includes exact snapshots as well as pageable blocks. Before embeddings and prefill, APC drains pending disk writes and evicts idle checkpoints and blocks in LRU order within each tier. Admission uses current Metal allocations, available system RAM, and the incoming prompt's estimated cache growth, based on observed bytes per token. Leased blocks remain valid for active requests. Snapshots that exceed the budget are written directly to disk without making another resident copy, and disk restores stay out of the memory LRU when promotion would exceed the budget. Disk pressure may therefore trade latency for lower memory use.

These controls limit APC's memory overhead; model weights and an individual request must still fit in memory. Tune the reserve for models with larger prefill or vision temporaries. /v1/cache/stats reports resident bytes, the memory budget, prefill reserve, memory evictions, and pending disk bytes.

KV Cache Quantization

Reduce KV cache memory during continuous batching with --kv-bits. Both uniform quantization and TurboQuant are supported. Compatible with Automatic Prefix Caching (APC_ENABLED=1).

# Uniform 8-bit KV cache quantization
mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 8
TurboQuant 3.5-bit (3-bit keys + 4-bit values)
mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 3.5 --kv-quant-scheme turboquant

Full-attention layers use quantized batch caches while sliding-window layers keep their fixed-size rotating caches. The last full-attention layer stays unquantized (sensitive in deep models).

##### Per-tensor KV quantization

Keys and values do not have to share a bit-width or a backend. A fractional --kv-bits already splits the budget — 3.5 gives 3-bit keys and 4-bit values — and --kv-key-bits / --kv-value-bits override either side:

# 8-bit keys, 3-bit values, both TurboQuant
mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --kv-bits 3.5 --kv-quant-scheme turboquant \
  --kv-key-bits 8 --kv-value-bits 3

--kv-key-scheme / --kv-value-scheme go further and select a different backend per tensor, which builds a hybrid cache:

# uniform 8-bit keys beside TurboQuant 3-bit values
mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --kv-bits 8 --kv-quant-scheme uniform \
  --kv-value-bits 3 --kv-value-scheme turboquant

Two limitations apply to mixed schemes specifically:

• The hybrid cache dequantizes on every step instead of using a fused kernel, so it is slower than either homogeneous path.
• Mixed schemes are not supported during continuous batching or for batch prefix caches, and raise NotImplementedError there. Mixed bit-widths under a single scheme work everywhere.

Tested with gemma-4-26b-a4b-it at 20K context:

| Config | Gen tok/s | KV Cache | KV Reduction | |--------|-----------|----------|--------------| | No quant | 50.3 | 0.624 GB | 1x | | Uniform 8-bit | 52.6 | 0.469 GB | 1.33x | | TurboQuant 3.5-bit | 25.6 | 0.365 GB | 1.71x |

Models with all full-attention layers (e.g. Qwen, LLaMA) see larger reductions — up to 3.6x at 8-bit and 6.4x at 4-bit.

Log Probabilities

The /chat/completions endpoint supports OpenAI-compatible per-token log probabilities. Pass logprobs: true (and optionally top_logprobs: N, up to 20) in the request:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
    "messages": [{"role":"user","content":"Say hi in 3 words."}],
    "max_tokens": 8,
    "logprobs": true,
    "top_logprobs": 3
  }'

Each choice gets a logprobs.content[] list with one entry per generated token: {token, logprob, bytes, top_logprobs: [{token, logprob, bytes}, ...]}. Works for both streaming and non-streaming.

top_logprobs requires the server to be started with a non-zero cap on how many alternatives it will compute per token (default 0 = disabled, max 20). Set it via the --top-logprobs-k flag or the TOP_LOGPROBS_K env var:

mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit --top-logprobs-k 5
or
TOP_LOGPROBS_K=5 mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit

Per-request top_logprobs is clamped to TOP_LOGPROBS_K. When TOP_LOGPROBS_K=0, requests with logprobs: true still return chosen-token logprobs; only the top_logprobs list stays empty. Leaving the cap at 0 keeps the vocab-wide sort out of the decode graph, so deployments that don't need logprobs pay zero overhead.

Structured Outputs

The /v1/chat/completions and /v1/responses endpoints support OpenAI-compatible json_schema structured outputs. The server constrains generation to the supplied JSON schema and supports both streaming and non-streaming responses.

You can define the schema with Pydantic:

from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
class AnimalResult(BaseModel):
    model_config = ConfigDict(extra="forbid")
animal: Literal["dog", "cat", "bird", "unknown"]
    species: str = Field(max_length=60)
    description: str = Field(max_length=200)
schema = AnimalResult.model_json_schema()

Call the local server with the OpenAI Python client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="mlx-community/Qwen3.5-4B-MLX-4bit",
    messages=[
        {"role": "user", "content": "Return a dog object."},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "AnimalResult",
            "strict": True,
            "schema": schema,
        },
    },
)
result = AnimalResult.model_validate_json(response.choices[0].message.content)
print(result)

Example output:

animal='dog' species='Canis lupus familiaris' description='A domesticated canine known for companionship and loyalty.'

Chat completions use top-level response_format. The same format works for text-only and multimodal requests:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3.5-4B-MLX-4bit",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Identify the main animal in this image."},
        {"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}
      ]
    }],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "AnimalResult",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "animal": {"type": "string", "enum": ["dog", "cat", "bird", "unknown"]},
            "species": {"type": "string", "maxLength": 60},
            "description": {"type": "string", "maxLength": 200}
          },
          "required": ["animal", "species", "description"],
          "additionalProperties": false
        }
      }
    },
    "max_tokens": 256
  }'

Structured outputs are also supported with:

• Streaming chat completions by setting "stream": true
• The responses API via text.format on /v1/responses
• Text-only requests using the same response_format sha