One command turns a DGX Spark into a private AI server
Watch the 49-second tour, with sound
**Qwen3.8-27B at 71.4 tok/s with a 1M context, Qwen3.8-Flash-Next 176B on a single box, images, video, Kolibri-1, Nemotron 3.5 Bolt, Nemotron 3.5 Super VL and Ornith 1.5 122B: installed and pinned by one command**, the lane that serves proved before it ends and every other built at its first Load, on a DGX Spark or another GB10 box such as the ASUS Ascent GX10. No prompt, image or video leaves the box.
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bashPrefer to read it before it runs? Clone the repo and run ./install.sh: that is all the
one-liner does (Quickstart).
| Model | Measured on the reference box, an ASUS Ascent GX10 |
|---|---|
| Qwen3.8-27B, NVFP4 + DFlash2 | 71.4 tok/s greedy median, 1M context, 135-148 tok/s at 8 streams (v1.2 battery) |
| Qwen3.8-Flash-Next 176B, on one box | 47.9 tok/s on code (FLASH_TIER=context), 262K context |
| Qwen-Image 2.1, 1024×1024 | 7.5 s with its Turbo, 34 s without |
A second ASUS Ascent GX10, set up from scratch with that one command: what it measured.
Why this rather than a recipe. A recipe starts one model. On a Spark the hard part comes after: nine lanes take turns in one 128 GB pool, SGLang's memory accounting misses 25-40 GB of transient allocations in it, and a pool pushed to zero freezes the box until a power cycle (the unified-memory trap). This repo is what lets them live on one box:
- Pinned and proved. Images by digest, checkpoints by revision; the lane that serves
--all-weights proves every lane it builds), and an
update is the same command.
- One pool, lanes that take turns. The cockpit shows what holds the memory and loads a lane
- Clients that work out of the box. OpenAI- and Anthropic-compatible APIs, opencode wired to
- Measured, not claimed. GSM8K 94.0% on the 27B and 97.41% on the flash, tool calling
Also on the box: Aleph Alpha's Kolibri-1 for German and English (48 tok/s, 166 at 8 streams), NVIDIA's Nemotron 3.5 Bolt, which reads images and video (140 tok/s alone on code, about 200 at 16 streams), its big sibling Nemotron 3.5 Super VL with a 1M-token window (70 tok/s alone on code, 82 at 4 streams), NVIDIA's FP4-Local build of the Flash-Next on vLLM (the flash lane's quality in about half its disk, with twice its KV cache), Ornith AI's Ornith 1.5 122B, a mixture of experts with vision and a 1M-token window (42 tok/s alone, 90 in all at 8 streams), MiniMax-H3 video with sound, opt-in speech on the CPU (Qwen3-ASR hears 30 languages, Kokoro speaks seven), and a System One endpoint that answers with calibrated probabilities instead of text.
Used and cited by others: Kevin Welter's write-up of running Qwen3.8 on a DGX Spark follows this recipe, MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark builds on its published config, heterogeneous-gpu-pd-lab takes its measurements as its DGX Spark reference, and it is listed in awesome-dgx-spark.
The cockpit, installed by default: one pool of unified memory, lanes that take turns in it, and the whole box run from a browser, on a laptop or a phone.
See it in 49 seconds
https://github.com/user-attachments/assets/dce6294e-4fbd-4337-a0e6-360d552eaf29
Everything in it ran on one DGX Spark and was filmed from its own cockpit: the agent wrote its service and ran its test on the 27B, Qwen-Image 2.1 made the image in 40 steps and MiniMax-H3 the video with its sound in 25 min 51 s, sped up where the pill says so.
In detail
One command installs a boot-persistent, hardened serving stack on a single DGX Spark,
and it is not one model: large language models (the Qwen3.8 family, seven
switchable targets, 27B at 71 tok/s and Flash-Next 176B on one box), typed decisions
(a System One endpoint speaking TypeSafe's Jev contract: calibrated probabilities
instead of generated text), image generation (Qwen-Image 2.1) and video generation
(MiniMax-H3), and on vLLM Aleph Alpha's Kolibri-1 since v1.29, **NVIDIA's Nemotron 3.5
Bolt since v1.34, Nemotron 3.5 Super VL since v1.35, NVIDIA's FP4-Local build of the
Flash-Next since v1.37 and Ornith AI's Ornith 1.5 122B** since v1.38. The one command installs
all of it: the lane that serves, proved before the run ends, and every other lane on demand,
built at its first Load from the cockpit (--all-weights builds and proves them all at install). One cockpit drives the nine lanes, one Load button moves
between them, and text answers on the same OpenAI-compatible API.
The two Qwen text lanes come with seven switchable targets, and speculation costs no quality on any of them (NVFP4 is the quantization floor, Qwen's own FP8 is available above it; every speculative path is lossless by construction). Since v1.8 the flash lane serves an official SGLang image for this hardware with nothing added, serves 8 concurrent requests by default where it used to serve one (since v1.19; opencode keeps a 205,000-token window on it since v1.20.3, and eight sessions that outgrow the pool queue rather than fail; FLASH_TIER=context keeps 4, so fewer sessions share the pool), and got 14 to 25% of its decode back from one flag (--speculative-token-map, see below):
| target | model | engine | headline (measured here) |
|---|---|---|---|
| stock (default) | Qwen3.8-27B NVFP4 | SGLang + DFlash2 | 71.4 tok/s greedy median, 135-148 aggregate at 8 streams (v1.2 battery), 1M context by default |
| uncensored | Qwen3.8-27B abliterated NVFP4 | SGLang + DFlash2 | same speed and serving path as stock |
| fp8 | Qwen3.8-27B FP8, Qwen's own release | SGLang + DFlash2 | the quantization reference: 108 tok/s aggregate at 8 streams, ~92K less KV pool |
| uncensored-fp8 | Qwen3.8-27B abliterated FP8 | SGLang + DFlash2 | same serving path and same cost as fp8 |
| flash (default of its lane) | Qwen3.8-Flash-Next 176B hybrid MoE NVFP4 | SGLang + NEXTN | 47.9 tok/s on code, 47.1 on math, 29-31 on prose, 27.0 ms/tok on an agent loop (all at the context tier), prefix caching, vision, 262K on ONE box |
| flash-uncensored | the abliterated build of that same tree | SGLang + NEXTN | 205 of 206 shards identical in size to stock, so the same flags: 45-46 on code, 0 refusals of 5 |
| flash-nvda | the same 176B from NVIDIA's mixed-precision export | SGLang + NEXTN | same N-gram table byte for byte, its own expert calibration: ties flash on every quality probe, with a KV pool about 9% smaller |
The 27B path is the fastest configuration measured so far on GB10 (SGLang + NVFP4 + DFlash2 speculative decoding with deterministic kernels, drafting from a calibrated NVFP4 head 16 deep): 71.4 tok/s greedy median on ./bench.sh, measured 2026-09-17 on SGLang v0.5.19, when this lane moved to the official release image (on v0.5.21 since v1.23: the same 105 to 108 ms per drafter step; on v0.5.22 since v1.39, faster at 1 and 4 streams in the same night's A/B, see the CHANGELOG), against 69.8 on the image it replaced and 65.3 on the v1.9 campaign before it (per probe then: code 64.2-65.3, reasoning 65.3-66.2, math 56.6-71.3, prose 23.0-24.7). Free prose is the slow case on any drafter. 135-148 tok/s aggregate at 8 concurrent streams, 258 at 32 (carried over from the v1.2 battery; the draft only helps concurrency, re-measure on your box with ./bench-matrix.sh). Reproducible to the decimal across boots: see BENCHMARKS.md, "The boot lottery".
The flash path serves a model that does not otherwise fit: the 176B checkpoint's 47.7 GiB FP8 N-gram table is served from a sparse file on NVMe, read row by row by the gather kernel through GB10's host page tables, leaving the unified pool to the compute weights and a real KV cache. Until v1.7 that was a vendored patch of this repo's own; since v1.8 it is upstream (--ple-offload-backend file, sglang#37068) and the overlay is retired, along with the vendored sm_121 QSA kernel and the workaround for the GB10 MTP collapse. Prefix caching works (27k tokens re-served in 2.5 s against 12.0 s cold), decode is 47.9 tok/s on code and 47.1 on math single stream at the context tier (29-31 on prose), prefill ~2,250 tok/s cold, and image input stays available. Since v1.8 it also takes --speculative-token-map, which hands the speculative draft the target's lm_head sliced to 65,536 rows instead of all 248,320: that removes 2.6 GiB from every engine step on a lane that is memory-bandwidth bound, and it is worth 14 to 25% of decode without changing what the model can say, because the target still verifies every drafted token over the whole vocabulary.
Whatever the target, you get the same surface: an OpenAI-compatible API and the Anthropic protocol on port 30001 for every text lane, through a keepalive proxy built for agent CLIs (the engine itself answers on 127.0.0.1:30000, on the box only), and opencode works out of the box (the installer writes a ready-to-use provider config; the chat template ships pre-patched for agentic clients). The stack is built to grow: more targets, engines and drafters will slot into the same switch surface.
Where everything is
The README is the entry point. Everything longer lives next to it, one subject per file.
| If you want | Read |
|---|---|
| Every number this repo publishes, how it was measured, and the failed experiments | BENCHMARKS.md |
| The 1M context mode: what it serves, what it costs, how limits are fitted to your boot | docs/context-1m.md |
| The flash lane in full: how a 176B fits, the three tiers, what each one measured | docs/flash-lane.md |
| The image lane: every number, every refusal and how the runtime is pinned | docs/image-lane.md |
| The video lane: every number, every refusal and how the runtime is pinned | docs/video-lane.md |
| The Kolibri lane: every number, the builds tried and why this one | docs/kolibri-lane.md |
| The Nemotron 3.5 Bolt lane: every number, the settings tried and why these | docs/nemotron-lane.md |
| The Nemotron 3.5 Super VL lane: every number, the builds tried and why this one | docs/nemotron-super-lane.md |
| The Flash FP4-Local lane, NVIDIA's FP4 build of the flash model: every number and the recipe | docs/flash-local-lane.md |
| The Ornith 1.5 122B lane: every number, the engines tried and why this one | docs/ornith-lane.md |
| Models on an external disk: what can live there, what stays on the internal one | docs/external-disk.md |
| The cockpit, view by view, including the Agent view and how it behaves on a phone | docs/cockpit.md |
| Day-to-day commands, the opt-in extras, upgrading from an earlier version | docs/operations.md |
| When an engine stops: the timeline of a crash, the start guard, requests held through a restart | docs/engine-restarts.md |
| What the installer writes for opencode, and how to opt out | docs/opencode.md |
| pi and omp: the same configs in their own formats, generated from the same artifact | docs/pi.md |
| Clients: Claude Code, VS Code Copilot, Open WebUI, Cursor, TLS, per-client identity | docs/clients.md |
| Why --mem-fraction-static decides whether this box stays alive | docs/gb10-memory.md |
| Where the two Qwen text lanes stand against upstream SGLang, re-checked in the images | docs/upstream.md |
| Typed decisions in full: the contract, the four levers, the refusals, the load | docs/systemone.md |
| The speech lane: install, use, the languages, the long recordings, what it was measured with | docs/speech-lane.md |
| The lean reasoning level: method, numbers, negative results | LEAN.md |
| What changed in every release, with the measurement behind each change | CHANGELOG.md |
| The layout, the state files, the invariants CI holds | ARCHITECTURE.md |
| How this repo is tested, and what the tests found in code already in production | TESTING.md |
| Where this is going | ROADMAP.md |
| Trust model and private reporting | SECURITY.md |
| To contribute | CONTRIBUTING.md |
| Mirroring the pinned image and checkpoints for an air-gapped install | MIRROR.md |
| Every capability mapped to a panel or an action, audited at v1.5 | dashboard/CAPABILITIES.md |
| The cockpit's design foundations | dashboard/DESIGN.md |
| The cockpit as it stood at its v1.6 review, kept as history | dashboard/REVIEW.md |
| Provenance and licenses of the retired flash overlay | flash-sglang/ATTRIBUTION.md |
| Reproducing the MMLU and HumanEval numbers, and the two image quirks in the way | evals/README.md |
| Running a GGUF model on this box instead | extras/gguf/README.md |
| Chromium / gstack browse dies with "No usable sandbox!" on Ubuntu 24.04 | extras/chromium-sandbox/README.md |
The whole stack at a glance
+-----------------------------+
agent CLIs / SDKs ---> | keepalive proxy :30001 | keepalives, aborts, guards
(opencode, Claude) +-------------+---------------+
| forwards
+-------------v---------------+
| text engine :30000 | 27B, flash, FP4-Local, Kolibri, Bolt,
| (one lane at a time) | Super VL or Ornith, OpenAI + Anthropic
+-----------------------------+
you, from a browser -->| Spark Cockpit :30090 | health that is real, actions,
(desk or phone) | + Agent view (opencode web) | jobs, registry, recipes, logs
+-----------------------------+The engine answers /health even when it is wedged, so the cockpit runs a real generation canary and reports ready, loading, wedged or stopped from that. Everything privileged the cockpit can do (unit start/stop/restart, lane switch, flush, abort) goes through an exact-argv sudoers allowlist and is audited. The Agent view frames opencode's own web interface behind the cockpit login, so sessions on the box run from a laptop or a phone with no terminal. Full tour: docs/cockpit.md. Since v1.12 it is installed by the one-liner like everything else: when the installer finishes it prints the URL, and that page is where this box is meant to be driven from. --no-cockpit opts out, and the choice sticks.
Quickstart
Requirements: DGX Spark or other GB10 machine (128 GB unified), stock DGX OS (its Docker CE and NVIDIA Container Toolkit: Docker from the Snap package cannot start a GPU container beside that toolkit, and the installer stops on it before it pulls any image). What a new box lacks, the installer puts in itself: DGX OS's first-boot setup leaves the account it creates out of the docker group, and leaves out Python's headers and ffmpeg, which the image and video lanes build and run with. The installer adds the account to the group, saying so and why before it runs sudo, and goes on in the same terminal, with no need to log out. It installs python3-dev and ffmpeg (and python3-venv where it is missing), naming them as it does. Free disk for the whole box: about 930 GB (GB of 10^9 bytes, rounded up), as the lanes measure on the reference box: the 27B lane 58 GB (22 checkpoint, 2 draft, 34 image), the flash lane 223 (136 checkpoint, 35 image, 52 for the PLE table it rewrites at every boot), the image lane 46 (34 checkpoint, 12 runtime and its build), the video lane 174 (145 checkpoint and its Turbo adapter, 12 runtime and its build, 17 for the runtime's own caches), the Kolibri lane 102 (79 checkpoint, 23 image), the Nemotron lane 51 (26 checkpoint and drafter, 25 image), the Nemotron Super lane 77 (its checkpoint and drafter; it runs the Nemotron lane's image, 102 without it), the Flash FP4-Local lane 121 (103 checkpoint, 18 image, and 32 more for the copy of its n-gram table when the cache is on another disk than the home directory's, docs/external-disk.md) and the Ornith 122B lane 78 (its checkpoint; it runs the Flash FP4-Local lane's image, 96 without it). Each lane checks its own room before it downloads anything (a lane on demand, at its first Load), and a lane installed whole that does not fit is left out and named at the end, with the command that adds it later, while the rest installs. What the text lanes check, in detail: for a 27B target, 49 GB on the disk of HF_CACHE (~/.cache/huggingface by default) for the checkpoints and caches, and 52 GB on the disk of Docker's images (/var/lib/docker on a stock box) for its 34 GB image (a pull's peak: Docker's containerd image store keeps the download beside the image); for a flash target, 248 GB on the disk of HF_CACHE (194 for the checkpoint and its caches, 54 for the 52 GB PLE table the lane rewrites at every boot, counted on the disk of PLE_DIR instead when that is another one) and 53 GB there for its 35 GB image. When both are one disk, as on a stock box, the installer asks for the sum there: 101 GB for a 27B target, 301 GB for a flash one. What the cache already holds of a checkpoint comes off its share (a checkpoint that is all there needs 11 GB of working room instead), and an image already pulled needs 6 GB instead of its own size. Caching the other 27B targets adds ~22 GB per NVFP4 target and ~31 GB per FP8 one.
One command, first install and updates alike. It clones or updates ~/dgx-spark-qwen38, then runs the pinned installer, which installs the 27B, which serves, the keepalive proxy with its typed-decision endpoint, the opencode wiring, the cockpit and its Agent view, and every other lane on demand: their units now, and each one prepared at its first Load from the cockpit, as you, with no sudo, then its weights (Flash-Next 176B beside the 27B: 223 GB; images 46, video 174, Kolibri 102, Nemotron 51, Nemotron Super 77 beside it, 102 alone, Flash FP4-Local 121, Ornith 122B 78 beside it, 96 alone). The 27B proves it serves before the run ends; each other lane is built and started by its first Load. A new box downloads about 39 GB for the 27B (about 6 to 7 minutes at 100 MB/s). --all-weights installs the whole box now instead, as every run did before: about an hour and a half of work (92 minutes measured at v1.29.0, before the two Nemotron lanes, the weights cached), plus about 763 GB of downloads (about 127 minutes at 100 MB/s). It ends by printing the cockpit URL, and there is nothing left to run by hand.
Updating a box adds the lanes it does not have yet, on demand. A lane a box never had (the flash beside the 27B, images, video, Kolibri, the two Nemotron lanes, Flash FP4-Local, Ornith 122B) gets its units, and its first Load from the cockpit prepares it; a lane already on the box (installed whole, from any version, or begun by a first Load) is updated whole, as before. --all-weights builds the new ones now and proves them, and --with- does it for one lane. To keep a box as it is, say so once, and later runs remember it: --no-kolibri, --no-nemotron, --no-nemotron-super, --no-flash-local, --no-ornith-122b, --no-flash (or --no-27b on a flash box), --no-image, --no-video, after bash -s -- on the one-liner.
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bashNo sudo in front of it, ever. The installer calls sudo itself, for the privileged steps and for nothing else. Under sudo, $HOME is /root: the clone, the API key, the chat template and the compile cache all land in /root, the units are rendered pointing there, and the engine then installs, starts and serves perfectly well with a key nobody has, so every client reading ~/.config/qwen38/api-key gets 401. Both entry points refuse root before writing anything (tests/test_install_root_refusal.py).
Options ride on the bash side of the pipe (an env prefix on curl would not reach the installer):
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=flash bash
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | CONTEXT_MODE=native bash
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=uncensored bashOr the explicit way:
git clone https://github.com/hasso5703/dgx-spark-qwen38.git
cd dgx-spark-qwen38
./install.sh # image + checkpoints + systemd service, starts at every boot
./bench.sh # verify your tok/sFirst boot takes ~7-9 minutes for a 27B target (CUDA graph capture + kernel compilation, cached afterwards; later boots ~5-7 min) and ~12-15 minutes for a flash target, every boot: the server writes the whole 47.7 GiB N-gram table into its file each time (measured here: 12 min 21 s to /health on a fresh table). The Flash FP4-Local lane writes no table: 17 minutes at its first boot, while b12x tunes its kernels for this GPU, then about 5 and a half. With --all-weights the install then proves each other lane it built, one after the other while the served one waits, and brings it back; a plain install leaves each to its first Load. Then:
- opencode: ready config at
~/.config/qwen38/opencode.json, see opencode integration - Any OpenAI client:
http://, model:30001/v1/chat/completions qwen3.8-27b(flash:qwen3.8-flash-next, Flash FP4-Local:qwen3.8-flash-next-fp4, Kolibri:kolibri-1, Bolt:nemotron-3.5-bolt, Super VL:nemotron-3.5-super-vl, Ornith:ornith-1.5-122b), Bearer key from~/.config/qwen38/api-key - Anthropic protocol:
http://(:30001/v1/messages Authorization: Beareronly, notx-api-key) - Both are the keepalive proxy, not the engine: it relays every route the engine serves and adds the guards for the requests SGLang dies on rather than refuses. Since v1.17 the engine itself binds
127.0.0.1and:30000answers on the box only (docs/clients.md, SECURITY.md) - Don't want a systemd service?
./install.sh --no-service && ./run.sh: the native unit's pins and flags, foreground, no sudo, Ctrl+C and it's gone (27B targets; flash is service-only in this release). That path has no proxy: clients talk to the engine on:30000, without the guards, andrun.shserves it on127.0.0.1like the units (ENGINE_BIND=0.0.0.0 ./run.shputs it on every interface). - Everything is pinned twice (base image digest + checkpoint revisions at download, and the same
--revisionpassed to the server itself, so an upstream push to a checkpoint repo can never change what you serve; the files of the retired flash overlay keep their sha256 manifest, checked in CI, as the record of what upstream replaced,flash-sglang/ATTRIBUTION.md). It still works months from now; the installer is idempotent and every failure path says how to fix itself.MODEL_REV=main ./install.shoverrides the pins;git checkout v1.1 && ./install.shreturns to the DSpark config. - Since 2026-08-21 this same combination (DFLASH2, draft depth 16 since v1.9) is the **official recipe in the
Your choices, and how they combine
Everything below is optional and combinable. Variables ride on the bash side of the one-liner, flags go after bash -s --; with a clone, they go straight on ./install.sh.
| Choice | How | Default | Notes |
|---|---|---|---|
| Model | MODEL_CHOICE=stock, uncensored, fp8, uncensored-fp8, flash, flash-uncensored, flash-nvda | stock | 27B NVFP4 stock or abliterated, the same pair in Qwen's FP8, or Flash-Next 176B in one of three NVFP4 exports (see the seven targets) |
| Reduced draft vocabulary | SPEC_TOKEN_MAP_SIZE=65536, or 0 to serve without it | 65536 | flash only: hands the speculative draft the target's lm_head sliced to that many rows, which is 14 to 25% of decode and cannot change what the model may say |
| Flash serving tier | FLASH_TIER=context, concurrency, throughput | concurrency since 2026-09-30 | flash only: 8 concurrent requests with 2 states per path (default; pool measured 477,056 tokens, so a full 262K prompt fits at 8 too, and opencode gets 205,000 tokens of context and 32,000 of output on it as on context), 4 requests so that long sessions queue less, or 24 without speculation |
| Reasoning effort | lean (default), xhigh, medium, low | lean | the level this repo adds and defaults to: 74 words in the chat template that cost 0.71x the thinking tokens of medium on 364 public problems and 0.436x on 58 underspecified requests, with no measured quality cost. Qwen's three levels are left byte-identical. LEAN_DEFAULT=0 ./install.sh installs it without taking the default. Numbers, method and negative results in LEAN.md |
| Context mode (27B) | CONTEXT_MODE=native or 1m | 1m since v1.12.1 | 1,010,000 window via YaRN, mem-fraction 0.76, proxy required, limits fitted to the real pool at the end of the install (see the 1M section). The flash lane and --no-service are native either way, with no refusal, except --no-service on a box whose installed unit serves 1m: that is refused with both ways out, since it would leave the unit on configs it cannot start from. A re-run keeps whatever is already installed, both directions |
| systemd service | default, or --no-service | service | --no-service: foreground with ./run.sh, no sudo, 27B native only |
| Start now | default, or --no-start | starts | install and enable the engine and the proxy without starting them (sudo systemctl start later, as the installer prints); that path installs the other text lane too, and skips the smoke test, the fit of the 1M limits, the cockpit with its Agent view, and the image, video, Kolibri, both Nemotron, Flash FP4-Local and Ornith lanes |
| The other text lane | default, or --no-flash (on a 27B box), --no-27b (on a flash box) | beside the serving one, on demand: its units, built at its first Load | since v1.20: the same installer, started again for that lane, with its own target kept across updates; since v1.33 its image, checkpoint and draft (or vocabulary) come with its first Load. --with-flash / --with-27b install it whole now and prove it, and bring it back |
| Image lane | default, or --no-image | on demand: its units, built at its first Load | Qwen-Image 2.1, 46 GB; --with-image installs it whole now and proves it, and brings it back after a --no-image |
| Video lane | default, or --no-video | on demand: its units, built at its first Load | MiniMax-H3, about 174 GB of headroom; --with-video installs it whole now and proves it (about 23 min), and brings it back |
| Kolibri lane | default, or --no-kolibri | on demand: its units, built at its first Load | Aleph Alpha's Kolibri-1 on vLLM, 102 GB (79 checkpoint, 23 image); --with-kolibri installs it whole now and proves it (about 10 min), and brings it back |
| Nemotron lane | default, or --no-nemotron | on demand: its units, built at its first Load | NVIDIA's Nemotron 3.5 Bolt on vLLM, 51 GB (26 checkpoint and drafter, 25 image); --with-nemotron installs it whole now and proves it, and brings it back |
| Nemotron Super lane | default, or --no-nemotron-super | on demand: its units, built at its first Load | NVIDIA's Nemotron 3.5 Super VL on vLLM, in the Nemotron lane's image: 77 GB of checkpoint and drafter, 102 with the image when the Nemotron lane is not there; --with-nemotron-super installs it whole now and proves it, and brings it back |
| Flash FP4-Local lane | default, or --no-flash-local | on demand: its units, built at its first Load | NVIDIA's FP4-Local build of Qwen3.8-Flash-Next on vLLM, 121 GB (103 checkpoint, 18 image), 32 more for the copy of its n-gram table when the cache is on another disk. --with-flash-local installs it whole now and proves it, and brings it back |
| Ornith 122B lane | default, or --no-ornith-122b | on demand: its units, built at its first Load | Ornith AI's Ornith 1.5 122B on vLLM, in the Flash FP4-Local lane's image: 78 GB of checkpoint, 96 with the image when the Flash FP4-Local lane is not there; --with-ornith-122b installs it whole now and proves it, and brings it back |
| opencode integration | default, or --no-opencode | on | on = ready config + oc launcher + default model following every switch; off = none of that, your own opencode config is never touched. --with-opencode turns it back on |
| Ports | PORT=, PROXY_PORT= | 30000, 30001 | agent clients use the proxy port |
| Storage | HF_CACHE=, PLE_DIR=, FLASH_LOCAL_PLE_DIR= | ~/.cache/huggingface, ~/flashnext-ple, ~/flash-local-ple when the cache is on another disk | checkpoints, the 52 GB flash PLE backing file, and the copies of the Flash FP4-Local lane's n-gram table (32 GB, none with none, which reads it where the cache is), and each choice holds for the runs after |
| Clone location | DIR= (one-liner only) | ~/dgx-spark-qwen38 | must be a clone of this repo on main |
| Cockpit dashboard | default, or --no-cockpit | installed and enabled; bound to the tailnet address when the box has one, else loopback | installed by install.sh since v1.12, and its URL is the last thing the installer prints. DASH_PORT=/DASH_BIND= on dashboard/install-dashboard.sh change port and bind; a re-run keeps them. Installs a sudoers allowlist, see the cockpit tour |
| Agent view (opencode in the cockpit) | default (the installer puts the pinned opencode in place when there is none, see docs/opencode.md), or --no-cockpit | installed with the cockpit; relay on the tailnet address | skipped with a note when opencode is missing, which costs one view and never the install; dashboard/install-agent.sh with AGENT_PORT=, AGENT_BIND=, OPENCODE_PORT= to retune. opencode itself stays on loopback, see the Agent view |
Combinations that make sense:
# one-liner forms (variables on the bash side, flags after "bash -s --")
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash # 27B stock serving (1M), opencode, cockpit; every other lane prepared at its first Load
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash -s -- --all-weights # the whole box now, every lane built and proved (about 763 GB of downloads, 930 of disk)
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash -s -- --no-flash --no-video --no-kolibri --no-nemotron --no-nemotron-super --no-flash-local --no-ornith-122b # a smaller disk: the 27B and images only
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | CONTEXT_MODE=native bash # the 262144 window instead
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=flash bash # Flash-Next lane (service only)
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash -s -- --no-opencode # API only, no opencode files
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash -s -- --no-cockpit # engine and proxy only, no web UI
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=uncensored bash # abliterated 27B, 1M context
clone forms
./install.sh --no-service && ./run.sh # no systemd, foreground, Ctrl+C stops it
CONTEXT_MODE=native ./install.sh # the 262144 window on a 1M box
./install.sh --with-opencode # turn the opencode integration back on
./switch-model.sh stock | uncensored | flash # change model later, no reinstall (then stop/start the units it prints; a lane installed on demand is built once first: ./prepare-lane.sh <lane>)Re-running the installer (upgrades included) remembers what you chose: the installed model, the context mode, the port, the HF cache, the opencode on/off choice, and the lanes you left out. Pass the variable or flag again only to change something. Since v1.36.0 a value that only an older release's default put on the box, and that nothing records as your choice (the engine on every interface before v1.17.0, the native window before v1.12.1, the flash lane's context tier before v1.19.0, the BF16 drafter before v1.9.0), moves to today's default instead. Every run lists both, what moves and what stays, before its first step, ./install.sh --plan prints that list and changes nothing, and the cockpit says in orange what the next update moves (your own choices are listed in Settings, with no alert). ./install.sh --recommended gives the box what a new one gets instead, your choices included, its port, HF cache and PLE folder kept (--plan --recommended lists what it moves). It restarts the engine only when something the engine reads changed since it started (its unit, image, chat template, checkpoint config, API key): an update that touches only the cockpit, the proxy or opencode keeps it serving, and RESTART_ENGINE=1 ./install.sh forces a restart. The same holds for the services around it since v1.18.7: the proxy, the cockpit and opencode-web restart only when what they run or read changed (the proxy also follows an engine restart), so a run that changes nothing cuts no request in flight and no turn of the Agent view. ./uninstall.sh --list shows everything the repo put on the box before removing anything.
What speed and quality to expect
Speculative decoding accepts predictable tokens, so speed depends on what the model generates, not on one magic number.
Two instruments, both in the box, both reproducible. The headline 71.4 tok/s greedy median is ./bench.sh (streaming decode rate net of TTFT, the repo's historical headline instrument: v1.0-v1.1 measured ~36-40 on it, v1.2 measured 41-57 per probe, v1.9 measured 64-71 per probe with the calibrated draft, and the v1.14 move to the official release image measured 71.4 against 69.8 on the image it replaced). The table below is the harsher one: the frozen battery ./bench-matrix.sh (two-call wall-clock delta, comparable across engines and boxes):
| What you generate (thinking on, battery v1) | v1.9 (NVFP4 draft D16, this repo) | v1.2 (DFlash2, this repo) | v1.1 (DSpark) | Stable-MTP engines |
|---|---|---|---|---|
| Agentic coding (code, diffs, tool calls) | 40 / 34 | 32-40 | 28-36 | 24-28 |
| Math & structured reasoning | 49.5 | 41-44 | 38-42 | 24-33 |
| Technical explanations (FR) | 32 | 26 | 23-25 | ~22 |
| Free-form prose EN / FR / DE | 22 / 20 / 18 | 22 / 20 / 17 | 17 / 14 / 13 | 17-20 |
| 8 concurrent streams, aggregate | 135-148 (v1.2 battery) | 135-148 | 100-104 | ~92 |
| 32 concurrent streams, aggregate | 258 (v1.2 battery) | 258 | not measured | not measured |
v1.9 wins every row of the frozen battery against v1.2 except eval-style math (one skipped sample on the v1.9 run; ./bench.sh math peak 57-71) and holds prose, historically the weak spot of block drafters. Every number above is deterministic across boots (--disable-flashinfer-autotune, see BENCHMARKS.md "The boot lottery") and was re-verified after a full machine reboot, with output-quality canaries passing. This machine serves its own opencode sessions daily on this config (stretched to the 1M preset from the field report below): if something breaks, it breaks here first.
Quality, measured (not claimed). The 27B lane, same box, v1.2.1 (2026-08-20), thinking on:
| Quality check | Result | |---|---| | GSM8K, 200 problems | 94.0% (188/200), exact parity with the DSpark profile | | IFEval, 200 prompts | 81.4% prompt-level / 87.4% instruction-level | | tool-eval-bench, 69 scenarios | 91/100 (Excellent), reproducible to the point across seeds | | Independent users on this config | 92-94/100 tool-calling (forum thread) | | Losslessness | token-identity study vs the pure model in BENCHMARKS.md ("The losslessness study") |
And the flash lane (flash, the RadixArk export), measured on this box on 2026-09-18 at the
context tier, every eval run against the serving surface this repo installs:
| Quality check | Result |
|---|---|
| GSM8K, 1,314 questions (its 1,319 less the 5 shown as examples) | 97.41% |
| MMLU, 500 questions | 90.8% (STEM 95.6, other 94.2, social sciences 89.2, humanities 86.1) |
| HumanEval, 164 problems, pass@1 at temperature 0 | 95.73% (157/164) |
| Tool calling, ./tools-check.py, 15 cases | 15/15 with reasoning off, 15/15 with reasoning on |
| Needle retrieval at 120K and 200K prompt tokens | 2/2 exact |
| conc-check.py, serial and 4 concurrent | 40/40 and 80/80 exact, no false or cross-contaminated answer |
The same battery ran against flash-nvda, NVIDIA's export of the same model, and the two tie
inside the sampling noise on every one of them. Their N-gram tables are the same bytes; the
difference that does hold up is the KV pool, in RadixArk's favour. Method and full table:
BENCHMARKS.md, "RadixArk against NVIDIA, head to head".
Full study (methodology, engine-vs-engine matrix, an independent reproduction, the physics of the GB10 ceiling, and a frozen benchmark battery you can run against any engine, ./bench-matrix.sh): in BENCHMARKS.md.
How the repo is tested, and what the tests found in code that was already in production: in TESTING.md. Measured branch coverage with a floor per module, property-based checks over generated inputs, a fuzzed and state-machine-simulated proxy, and a mutation score, because a suite written alongside its own code has to be asked whether it would notice the code being wrong (lifecycle.py: 74.8% of injected faults caught before that question was asked, 90.8% after). Everything else this repo documents is in the index at the top.
⚠️ The GB10 unified-memory trap (read this before changing anything)
SGLang's memory accounting does not see 25-40 GB of transient allocations on GB10 unified
memory: the flashinfer fp8 autotuner and CUDA graph capture allocate outside the tracked pool.
Push --mem-fraction-static too high, or run SGLang natively outside Docker, and host available
memory can reach zero. On a machine where SSH rides on that same memory, that is a freeze only a
power cycle fixes. This repo learned it the hard way.
What it does about it, with the measurement behind every number, is in docs/gb10-memory.md. The operative rules:
- the installed units pin the fraction for you: 0.76 in 1m mode (the default since v1.12.1,
./run.sh
- 0.80 was measured crashing on the 27B under 3 concurrent requests (the flash lane runs 0.85,
- the Docker cap (
--memory 100g) bounds host RSS only: the cgroup does not see CUDA unified
- the number to watch is GPU-side headroom after graph capture: 21.30 GiB at 0.76 against 19.38
Typed decisions: a System One endpoint
Since v1.15 the keepalive proxy answers POST /v1/systemone with the wire contract of
TypeSafe's Jev, from the Qwen lane
(27B or flash) already running on this box; on the other text lanes the proxy says so (400). A
state and typed questions (choice, score, and noul for yes or no) in, calibrated
probabilities out, with nothing generated, nothing parsed and nothing sent anywhere:
printf 'Authorization: Bearer %s\n' "$(cat ~/.config/qwen38/api-key)" | curl -s http://127.0.0.1:30001/v1/systemone \
-H @- -H 'Content-Type: application/json' -d '{
"state": "Hi, I have been trying to connect my Stripe account for 3 days and it keeps failing.",
"model": "jev-latest",
"questions": {
"department": {"type": "choice", "instructions": "Which team should handle this",
"criteria": {"billing": "Payment issues", "technical": "Bugs", "sales": "Pricing"}},
"is_urgent": {"type": "noul", "instructions": "The message conveys urgency"}
}}'Every question becomes one chat completion of exactly one token: the options are named by single-token letters and the probability of each letter is the probability of its option. A question answers in 0.2 s warm at any state size. The TypeSafe SDK runs against it with one base URL changed, and the cockpit's Decide view exercises the whole contract from a browser, with prefilled examples and the matching curl.
Measured against the hosted model, byte-identical payloads to both: 92.9% against 93.2% on TypeSafe's own 20 public cases, 89.3% against 91.9% on BoolQ with a better calibration (ECE 1.2% against 2.4%), and 62.1% against 83.8% on MMLU-Pro, the gap a single forward pass cannot close on calculation, which a thinking budget closes at 84.5% against 84.0% for 12 s a question.
The contract key by key, the four levers, the refusal parity, the door under load, the label table and every trap on the way: docs/systemone.md.
Images: Qwen-Image 2.1 on the same box
Every plain install includes an image lane since v1.20: text to
image, image editing with up to ten references, and the native RGBA this model is built for.
It costs 46 GB (34 checkpoint, 12 runtime and its build). A plain install puts in its units, and its first Load from the cockpit builds its runtime, as you, with no sudo, and fetches its checkpoint; --with-image installs it whole now and proves it with one image.
--no-image leaves it out, and later runs remember that; --with-image brings it back:
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash -s -- --with-imageThen it is a lane driven like the others: on the cockpit's Lanes view, pick
Qwen-Image 2.1 and press Load: one action that switches the boot, stops the serving
lane and starts this one (about a minute to answer). From a terminal,
./prepare-lane.sh image builds the lane the first time (no sudo), and ./switch-model.sh image
does the switch and prints the rest. Qwen-Image 2.1 Turbo, Qwen's
eight-step distillation of the same model, is a second checkpoint of the lane, loaded the same
way (./switch-model.sh image-turbo, 32.5 GB on its first switch): about four and a half times
faster, close to the base on photographs, portraits, illustrations and short text
(docs/image-lane.md shows where it is not).
31 GB of weights do not fit beside a serving LLM, so the lanes take turns: the cockpit refuses
to start any engine while another is busy, for the nine lanes alike, and the unit's Conflicts= is
a second belt for a systemctl start typed at a terminal.
Measured on a Spark, not copied from the cookbook (SGLang v0.5.21, 2026-10-10): 1024x1024 at 40 steps in 34.3 s (34.8 GiB peak), 7.5 s with the Turbo, 512x512 in 7.8 s (1.8 s), 2048x2048 in 182.8 s (40.6 s), an edit with one reference in 41.9 s (10.3 s), and a transparent generation that comes back with about three quarters of its pixels transparent. Same seed twice is byte-identical.
Three things bite before a client does: width and height must be multiples of 32 (anything else is a bare HTTP 500), an output format must always be sent (left out, the API falls back to JPEG, this model always returns RGBA, and the plainest possible request fails), and **CFG needs both a scale above 1 and a negative prompt** (either alone is ignored byte for byte). The cockpit's Image view refuses all three before they leave the box, exposes every parameter at the model's own defaults with a Reset button, and ships prompts and sample images to try.
One image at a time. The diffusion scheduler has no admission cap, so two concurrent requests do not queue, they each take a working set: measured, one generation holds 31.2 GB and eight in a row hold exactly the same, but two at once held 90.5 GB of this box's 121.6 and the engine stopped answering. The cockpit refuses a second one in under a millisecond and says why, and it refuses a call whose images add up to more pixels than the largest call measured here (one 2752x1536 image): the images of a call are one batch. The runtime cannot abort a generation, so the Image view's Cancel restarts the lane, in about a minute.
Editing redraws the whole picture rather than patching it: the edit you ask for happens, and the rest comes back with about twice the fine detail of what you sent (2.16x, reproduced across every reference, prompt, step count and guidance setting). Licence: Qwen Research, not commercial.
Every number, every refusal and how the runtime is pinned: docs/image-lane.md.
Videos: MiniMax-H3 on the same box
Every plain install includes a video lane beside the others since v1.20: text to video
with joint video-and-audio, plus first/last-frame conditioning. It needs about 174 GB of
headroom. A plain install puts in its units, and its first Load from the cockpit builds its runtime
and fetches its checkpoint; --with-video installs it whole now and proves it with one 4 s video
(about 23 minutes with its boot).
--no-video leaves it out, and later runs remember that; --with-video brings it back:
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash -s -- --with-videoThen it is a lane driven like the others: on the cockpit's Lanes view, pick
MiniMax-H3 and press Load: one action that switches the boot, stops the serving
lane and starts this one (about 12 min to answer). From a terminal,
./prepare-lane.sh video builds the lane the first time (no sudo), and ./switch-model.sh video
does the switch and prints the rest.
It serves the SGLang cookbook's MiniMax-H3 recipe as upstream wrote it, on the v0.5.21 release, with one local patch (an idle lane waits on its socket instead of holding a CPU core): the cookbook verifies about 12.1 s per denoise step, about 40 s of decode and about 12 min per warm 4 s 480P request on the DGX Spark, with no flags at all. One request at a time; the Video view refuses a second one, and a duration outside the cookbook's 4 to 15 s band.
A Turbo switch in the Video view puts the cookbook's recommended speed adapter (larryvrh's MiniMax-H3 Turbo LoRA, 8 steps) on for one video and takes it off after: 4 s at 480P in 127 s against 623 s for the base at its 50 steps, the same prompt, measured here, the speech intact, and the base's next video identical to the byte to one made before. It is admitted at 480P and 4 s for now, the sizes measured with it.
Every number, every refusal and how the runtime is pinned: docs/video-lane.md.
Kolibri-1: Aleph Alpha's German and English model on the same box
Since v1.29 every plain install includes a fifth lane: Kolibri-1, Aleph Alpha's open model (Apache 2.0), a 78B mixture of experts wit
... (README truncated for length)