sparkDash ⚡ — Multi-unit monitoring dashboard for NVIDIA DGX Spark

_{by Mia'a AI Lab}

_{macOS (darwin) SSH collectors authored by C. Michael Gibbs and Nyx Voss (Onyx AI Labs) — commits. sparkDash by Mia's AI Lab.}

sparkDash is a real-time web dashboard for one or more NVIDIA DGX Spark (GB10) machines in a single browser window. It streams GPU, CPU, unified memory, storage, network, and local LLM metrics — and lets you add, edit, reorder, or remove Sparks from the UI without restarts or code changes.

It also supports non-Spark units: any Linux machine with an NVIDIA GPU (e.g. a workstation with a dedicated RTX/L-series card) can be added as a dedicated GPU host and monitored the same way via SSH and nvidia-smi. For these units the dashboard correctly separates RAM (system memory) from VRAM (discrete GPU memory).

Monitored units run Linux, macOS, or Windows (a Windows PC with an NVIDIA GPU — or an AMD-iGPU Windows host like a Strix Halo, whose GPU usage comes from Windows' GPU engine counters; see Windows PCs).

LLM Prompt Showcase

_{Watch the MP4}

Table of contents

• Latest version changelog
• Features
• LLM server presets & backend auto-detection
• ComfyUI monitoring
• Hermes Agent monitoring
• Tailnet monitoring
• Tailscale Serve (HTTPS)
• Mobile PWA (install on a phone)
• Glance integration
• Quality bench
• Prometheus
• Alert notifications
• RoCE / RDMA monitoring
• Intel Arc GPUs
• Windows PCs
• Full changelog
• Quick start
• Architecture
• Tech stack
• Repository layout
• REST API
• Configuration
• Security
• Scripts
• How it works
• Contributing
• License
• Acknowledgements

Latest version changelog

Version 2.70.0

• Decode benchmark: temperature (default 0.7) and, for the Prose type, thinking on or off are selectable; the copied result text and the share card show both.
• NVIDIA Flare is detected and its tok/s monitored from /metrics.
• Tool Eval Bench: compare a run on one Spark with a run from another, and expand the scenario list to a taller view.
• Overview cards show CPU temperature and power; a Cost compare page prices fleet usage as any hosted model; Radiance and Strata engines are recognised.

Version 2.50.0

• Tool Eval Bench: Show card turns a finished result into a share image (score, rating, outcomes, safety warnings, categories) you can copy or save as a PNG.
• Tool Eval Bench: comparison card: tick two runs in History and get one image of the change between them.
• GPU clock cap can be run on the Spark yourself, or the dashboard asks for the sudo password when a unit has no passwordless sudo.

Version 2.40.1

• Temps page with day and week GPU temperature graphs for every unit, backed by a minute-resolution history kept for 7 days.
• GPU clock cap set with nvidia-smi -lgc shows on each unit (read-only).
• Mac units come online and report CPU, GPU and RAM through macmon.
• Fixes: GPU and energy data on remote units whose login shell is zsh, and the TensorFold Slots tile now counts occupied lanes.

Version 2.40.0

• Hide any Overview panel: Settings has switches for Recent activity and for each of the four tiles (Fleet decode, Fleet power, Memory in use, Tokens in last 24 hours), next to Fleet energy and LLM token totals.
• LM Studio and Ollama are detected by their own endpoints, so they are no longer labelled SGLang or vLLM; LLM panels match their port, and the Add wizard and LLM panels have LM Studio / Ollama / TensorFold port presets.
• Windows PCs with an AMD iGPU (Strix Halo) show GPU use and memory from Windows' GPU counters. Windows setup docs now cover the administrators_authorized_keys gotcha.

Version 2.32.0

• LLM panel: an engine that only SSH can reach (a remote unit behind a NAT that forwards just SSH) is now monitored through an SSH forward, with a via SSH chip; TensorFold prefill tok/s counts only computed tokens, so prefix-cache hits no longer read as 100k+.
• Decode benchmark: live generation tok/s is measured from the run's own streams, so it no longer shows 0.
• Health: a finding is held for 90 s, so a reading sitting on a threshold does not flap and spam Activity.
• iPhone app: the header clears the status bar and the home-screen icon is the logo. Dropping a Spark after dragging it in the sidebar no longer opens it.

Version 2.31.0

• RoCE / RDMA panel for Sparks: link state and rate, traffic, drops, PFC and QoS settings and the RDMA counters per port, with two health findings (a link went down, packets being dropped). See RoCE / RDMA monitoring.
• Time zone setting for the Tokens, Energy and Activity charts, so the 24 h charts line up with your local time.
• /metrics now shows when each collector last returned real data, so a stuck GPU poll can be alerted on while SSH liveness still passes.
• Spark page: Unified memory, CPU and Models share one row; CPU and RAM sit side by side on GPU hosts; the GPU history chart fills its panel; Windows PCs got a first real-hardware pass.

Version 2.30.0

• Windows PCs as units: a Windows PC with an NVIDIA GPU can be monitored over its built-in OpenSSH server (GPU, RAM, CPU load, disks, network, shutdown). See Windows PCs.
• Mobile app: sparkDash installs as a PWA over HTTPS (for example Tailscale Serve).
• Monthly energy history with a month/year view, Prometheus /metrics (opt-in), per-GPU graphs on multi-GPU hosts, a model picker in the decode benchmark, per-Spark llmHost, and Restart sparkDash from the UI after a fleet change.
• The Settings poll interval now really sets the polling rate; the Fleet energy card explains partial coverage.
• Fixes: "Frontend not built" after a Docker upgrade, the SSH tunnel for benchmarks, model launchers inheriting PORT=5555, and a token-protected remote bind showing a blank page. Everything since 2.0.0 is in the CHANGELOG.

Version 2.0.0 — a new sparkDash

• New shell: sidebar, command palette (Ctrl/⌘ K), four themes (White, Light, Dark, OLED), smooth page transitions and real links everywhere (right-click, new tab).
• Rebuilt Overview: per-Spark cards with the model row, head/worker relations, a Model/System memory bar, and a launcher to load a model from the card.
• Benchmarks section: Decode, Prefill, Quality and the new Tool Eval Bench (trials, side-by-side compare, one-click upgrade).
• Showcase is now an in-app page with much smoother streaming; Token totals, Fleet energy and Activity get their own pages.
• Start and stop your own LLMs from the dashboard; live prefill tok/s during long prefills; Quality bench with GSM8K and MMLU.
• Health findings per Spark (hot GPU, low unified memory under 3 GB, Xid/OOM errors, slow links) and Activity events for them; Clear / Reset for Activity, Fleet energy and Token totals.
• Mobile: docked tab bar with iOS-style motion, a Stats tab (Token totals, Fleet energy, Activity) and a rebuilt Spark page.

Features

| Area | What you get | |------|----------------| | Multi-unit | Any number of units; each has a tabbed detail page plus a shared Overview | | Non-Spark GPU hosts | Linux boxes with a dedicated NVIDIA GPU are first-class units: same nvidia-smi collectors over SSH, detected hardware summary, and separate RAM / VRAM panels. Detail page: GPU (left) + RAM → Network → Storage (right column); Overview cards show RAM and VRAM bars | | Windows hosts | Windows machines with an AMD iGPU (Strix Halo and friends) are first-class units too: agentless PowerShell over OpenSSH, CPU/RAM/disks/network collectors, and GPU usage straight from the engine performance counters (reads like Task Manager). The Add/Edit wizard has a Strix Halo / Windows host unit type | | Live streaming | WebSocket metrics with configurable poll intervals; central history store for sparklines across tab switches | | Local + remote | Host metrics via sysfs/proc/nvidia-smi; remotes over SSH (key or password) | | LLM probe | Auto-detects llama.cpp, vLLM, sglang, LM Studio, Ollama, ds4-server, EXL3, TensorFold, FreeToken, Strata, q27, or NVIDIA Flare; live decode/prefill tok/s from the engine's own counters where it publishes them (not LM Studio or Ollama); cached vs uncached prefill on ds4, llama.cpp, SGLang, and q27; daily peak history on the LLM card | | LLM server presets | One-click LM Studio :1234 / Ollama :11434 / TensorFold :8888 presets in the Add wizard's LLM-ports step, the "+ Add LLM port" panel, and each LLM panel's settings (they fill the port number — they install nothing) | | ComfyUI | Opt-in probe: queue/jobs, progress, cancel, Open link, inventory, overview chip | | Hermes Agent | Opt-in per unit: background update check (10 min), status badges, one-click or batch hermes update | | Tailnet | Opt-in probe: flags a unit that is healthy on the LAN but off its tailnet | | Decode benchmark | Multi-concurrency streaming decode tok/s; type picker (Structured / Prose / Code / JSON). Code is a different Python task per stream. Lab protocol (temp 0, thinking off); persisted last run. Remote units: LAN HTTP, or SSH tunnel to loopback. Remote button for an on-demand HTTPS/host:port target | | Prefill benchmark | Context-size sweep (1k–300k) of prefill tok/s and TTFT; unique prefix per size; persisted last run. Same remote targeting as decode | | Quality benchmark | Fixed, seeded quality suite (QA, reasoning, arithmetic, state tracking, GSM8K, MMLU, plus optional instruction following and long-context recall) at temperature 0; per-category and overall score; last 30 runs kept; paired compare with McNemar p-value. Same remote targeting as decode | | Prompt Showcase | Full-page multi-terminal LLM streaming demo (up to 32 prompts) with live tok/s and copy-out | | LLM inference health | KV cache free/pool (vLLM cache_config_info token and byte size; usage % when the pool size is absent), run/wait queue, TTFT/E2E/ITL p95, preemptions, prefix cache, MTP accept from Prometheus /metrics (vLLM and q27; q27 FIFO-queues so the Requests tile reads “N run” without a wait gauge) | | Multiple LLM ports | Monitor several LLM servers on different ports simultaneously — each gets its own panel with independent backend detection and metrics | | GPU processes | See the top GPU processes by VRAM usage directly in the GPU panel, including process name and memory allocation | | Multi-GPU hosts | A dedicated GPU host with several NVIDIA cards reports each one: the header names every card, the GPU panel adds a block per card (throttle chip, usage and temperature sparklines, power, VRAM bar) and the API exposes gpu.gpus[]. The headline gpu numbers stay an aggregate of all cards, so Overview cards and alerts need no change | | Spark uptime | System uptime displayed inline on each Spark header for at-a-glance availability | | Power controls | Graceful shutdown (SSH host script). Wake-on-LAN is for dedicated GPU hosts whose NIC supports it. DGX Spark does not wake from a magic packet | | Spark roles | Head / Worker / Standalone — worker label + head link; standalone can disable LLM monitoring; optional hide workers from Overview and tabs | | Unified memory | GB10 128 GB LPDDR5X pool (~273 GB/s), GPU/CPU split, bandwidth via nvidia-smi dmon. Non-Spark hosts show discrete VRAM (nvidia-smi) and system RAM separately | | Health findings | Small explainable rules over metrics already collected: GPU running hot (85 / 90 °C), GPU stuck in a low-power state, unified memory low (under 3 GB warns, under 1.5 GB critical), NVIDIA Xid errors and kernel OOM kills, several models loaded while memory is busy, a link slower than 1 Gb/s, and RoCE link / loss. Shown as chips on the cards and in full on the unit page; changes are logged to Activity | | Fleet pages | Token totals (per model, hourly / daily history), Fleet energy (rolling 24 h / 31 d plus a permanent monthly archive, optional cost), and Activity (event log with filters). Each page can be cleared or reset from its Clear / Reset… menu | | RoCE / RDMA | Per-port link state and rate, traffic, drops, PFC and QoS (trust mode, DSCP), and RDMA counters, read from sysfs / ethtool / mlnx_qos on the Spark (see RoCE / RDMA monitoring) | | Windows PCs | A Windows PC with an NVIDIA GPU is a first-class unit over the built-in OpenSSH server; an AMD-iGPU Windows host (Strix Halo) works too — its GPU usage comes from Windows' GPU engine counters (see Windows PCs) | | Mobile + PWA | A phone layout with a docked tab bar (Overview, Sparks, Stats, Settings); installable to the home screen over HTTPS (see Mobile PWA) | | Time zone | Settings → Time zone sets hour and day boundaries and labels on the Tokens, Energy and Activity charts, whatever device you view them on | | Prometheus | Opt-in GET /metrics export (see Prometheus) | | Metrics history | Opt-in (Settings → Metrics history): 48 h of 1-minute and 30 days of 15-minute avg/max per unit — GPU, memory headroom, CPU, RAM, network, LLM tok/s and KV fill — kept by the server across reloads and restarts and charted in a History panel on the unit page (1h–30d, shared crosshair). The 8 h GPU chart and 7-day Temps history are always on | | Restart from the UI | When the fleet changes and Energy needs a restart, a Restart sparkDash button restarts the Docker container (only offered when it can) | | Themes | Dark, light, cool white, OLED — neutral palettes, persisted in localStorage | | Secrets | SSH passwords AES-256-GCM encrypted; never in sparks.json or API responses | | Docker-first | Single privileged container for host metrics; prod and dev Compose files | | Hot config | Add / edit / remove / reorder Sparks from the UI with no process restart |

ComfyUI monitoring

sparkDash can optionally monitor a ComfyUI instance on each Spark — the same way it probes local LLMs, but focused on jobs and queue, not a second copy of GPU/RAM bars (those stay on the GPU / CPU panels).

What is supported

| Capability | Details | |------------|---------| | Opt-in per Spark | comfyMonitoring (default off) + comfyPort (default 8188) | | Any role | Head, worker, and standalone can enable ComfyUI independently of LLM cluster role | | Liveness | GET /system_stats — online, ComfyUI / PyTorch version, device type (cpu/cuda) | | Queue / jobs | GET /queue — running + pending items; workflow title, model/LoRA filenames from the graph, footprint (resolution · steps · sampler · batch · node count) | | Progress | Progress bar on the active job — Comfy WebSocket when events are available; otherwise elapsed / average-duration estimate | | Last finished job | Status + duration via /api/jobs (fallback /history) | | Queue ETA | Estimate from recent job durations × pending (+ progress remainder when known) | | Cancel / remove | From the Comfy card: interrupt a running job or dequeue a pending one (POST /api/sparks/:id/comfy/cancel) | | Open ComfyUI | One-click link to http://{lanIp}:{comfyPort} (LAN IP preferred so remote browsers do not hit localhost) | | Model inventory | Checkpoints + LoRAs from /models/* (UI section only when at least one file is listed) | | Overview chip | When monitoring is on: Comfy · idle / run / Nq / muted if unreachable | | Layout | Under Services: primary LLM + Comfy side-by-side when both are enabled; collapsible Resources / Services sections |

Not claimed: true per-job VRAM (Comfy does not expose that cleanly over HTTP). Host GPU/VRAM remains on the GPU panel. Live step progress depends on Comfy broadcasting WS events; stock Comfy often scopes detailed progress to the client that submitted the prompt.

How to enable (per Spark)

1. Open the Spark tab → Edit (pencil).
2. Enable ComfyUI monitoring.
3. Set port if needed (default 8188).
4. Save.

Connectivity Test (in Edit) includes ComfyUI when monitoring is enabled.

ComfyUI side requirements

• ComfyUI must be reachable from the sparkDash server on the probe host:

• For Open from another machine’s browser, Comfy should listen on a reachable interface (e.g. --listen 0.0.0.0), not only loopback, and the Spark’s LAN IP must be set correctly in Edit.

Config fields (persisted on the Spark)

| Field | Default | Description | |-------|---------|-------------| | comfyMonitoring | false | Probe ComfyUI and show the card / overview chip | | comfyPort | 8188 | ComfyUI HTTP port |

Related API

| Method | Path | Purpose | |--------|------|---------| | POST | /api/sparks/:id/comfy/cancel | Cancel a job ({ "promptId": "" }) — interrupt running and/or remove from queue |

Env (optional): COMFY_PORT (default 8188), COMFY_PROBE_TIMEOUT_MS, POLL_INTERVAL_COMFY.

Hermes Agent monitoring

sparkDash can optionally monitor Hermes Agent (nousresearch/hermes-agent) on each unit and run one-click updates for you over SSH.

What is supported

| Capability | Details | |------------|---------| | Opt-in per Spark | hermesMonitoring (default off) in Edit Spark | | Auto update check | Background hermes update --check over SSH (default every 10 min) — returns update availability + pending commits | | Status badges | In the Spark header: Hermes (installed version), Hermes not found if the binary is missing | | One-click update | Update Hermes button opens a dialog with live status, real pending commits, and release notes; Update now runs hermes update via SSH (non-interactive go) | | Update state | Running / success / error surfaced live (button turns into a “Hermes updating… / failed” state) | | Batch update | Update Hermes on Overview runs hermes update on every monitored unit, with a live per-unit progress bar |

How to enable (per Spark)

1. Open the Spark tab → Edit (pencil).
2. Enable Hermes Agent.
3. Save — background checks start immediately.

Connectivity check note: local units run the check as the host user (via setpriv/nsenter, never as container root); remote units run it over SSH. Either way, the logged-in user needs permission to read the Hermes repo.

Side requirements

• Hermes Agent must be installed on the target machine — sparkDash only checks & updates; it does not install it. The binary is looked up in ~/.local/bin and /usr/local/bin.
• SSH user must be able to run hermes update --check / hermes update non-interactively (key auth recommended).
• An update can take a few minutes (repo pull + dependency reinstall); a stale *.lock file from a crashed run is cleared before each attempt.

Config fields (persisted on the Spark)

| Field | Default | Description | |-------|---------|-------------| | hermesMonitoring | false | Check/update Hermes Agent on this machine |

Related API

| Method | Path | Purpose | |--------|------|---------| | POST | /api/sparks/hermes/update-all | Batch hermes update on every monitored Spark (Overview button) | | POST | /api/sparks/:id/hermes/check | Force hermes update --check now | | POST | /api/sparks/:id/hermes/update | Run hermes update in the background (202) | | GET | /api/sparks/:id/hermes/updates | Update preview: latest release + installed version + real pending commits + resolved view |

Env (optional): POLL_INTERVAL_HERMES (default 600000 ms), HERMES_UPDATE_TIMEOUT_MS (default 600000 ms).

Tailnet monitoring

Opt-in per unit (default off). Runs tailscale status --json on the host and shows a Tailnet card under Resources.

This closes a blind spot every LAN-based check shares, including sparkDash's own SSH liveness. When tailscaled loses its session with the coordination server, SSH/GPU/LLM can all stay healthy while the box is unreachable from off-LAN.

What is supported

| Capability | Details | |------------|---------| | Opt-in per unit | tailscaleMonitoring (default off) in Edit Spark | | Off-tailnet detection | Self.Online — the node's own view of the coordination server | | Reason, not just state | Tailscale Health messages, backend state, tailnet IP, DERP relay, version, expired-key warning |

Asked of each node about itself. Peer state is never the verdict. The probe is read-only (tailscale up / down / login are never run).

How to enable

1. Open Edit Spark.
2. Tick Tailnet monitoring.
3. Save. The Tailnet card appears under Resources.

Host requirements

• tailscale CLI on the monitored host, and tailscaled running.
• Remote units: existing SSH. Local Docker: nsenter into the host mount namespace (same as nvidia-smi; /host/proc is already bind-mounted).

Config fields

| Field | Default | Description | |-------|---------|-------------| | tailscaleMonitoring | false | Run tailscale status --json and show the Tailnet card |

Env (optional): POLL_INTERVAL_TAILSCALE (default 30000), TAILSCALE_PROBE_TIMEOUT_MS (default 8000).

Tailscale Serve (HTTPS)

Tailscale Serve publishes the loopback-only dashboard over HTTPS to your tailnet — no ports opened, no reverse proxy, and a real certificate (https://..ts.net). It is also the easiest way to satisfy the HTTPS requirement for the Mobile PWA, which browsers refuse to install over plain HTTP.

One-time setup

1. Enable HTTPS certificates and Serve for the tailnet (admin does this once):

1. On the sparkDash host, publish the dashboard (the dashboard must be running — Docker or npm start):

sudo tailscale serve --bg --https=443 http://127.0.0.1:5555

If the container binds a specific address instead of loopback, proxy that address (check docker logs sparkDash for the bind= line), e.g. sudo tailscale serve --bg --https=443 http://100.x.y.z:5555.

1. Verify and get your URL:

tailscale serve status
   # https://your-node.your-tailnet.ts.net  ->  proxy http://127.0.0.1:5555

Notes:

• The Tailscale hostname works with the default loopback bind — no SPARKDASH_ALLOWED_HOSTS entry needed (that list is only for custom reverse-proxy domains).
• Access is tailnet-only. Any device signed in to the tailnet (phone, laptop) can open the URL; nobody else can. For public exposure use tailscale funnel and an authenticated front door (see Remote access) — but prefer keeping it tailnet-only.
• The proxy forwards /api/* and /ws including WebSocket upgrades; nothing else to configure.
• To remove it later: sudo tailscale serve --https=443 off.
• Snap installs run Tailscale as root; if the plain command reports Access denied, use sudo tailscale serve ….

Mobile PWA (install on a phone)

sparkDash is a PWA: from a supported browser it installs to the home screen and runs fullscreen like a native app, with its own icon and no browser chrome.

What ships

| File | Purpose | |------|---------| | public/manifest.webmanifest | App name, standalone display, theme/background colors (#0a0c0f), icons | | public/icons/ | 192/512 px icons, maskable variants (Android adaptive icons), apple-touch-icon.png | | public/sw.js | Service worker: caches the app shell; never caches /api/* or /ws (live data always comes from the network) | | src/components/shell/InstallPrompt.tsx | In-app install banner (see below) |

Requirements

• HTTPS is mandatory for install (or localhost for local testing). Use Tailscale Serve or an authenticated TLS reverse proxy — a plain http://:5555 page cannot be installed.
• The manifest and icons are static files served by the same Express server (dist/ after npm run build); no extra configuration.

Installing

The app shows its own banner when opened in a browser it can be installed from (hidden once installed, or after dismissing):

• Android / desktop Chrome: an Install banner appears near the bottom of the page; tapping it opens the native confirm dialog. Also available in the ⋮ menu → Install app / Add to Home screen.
• iPhone / iPad (Safari): iOS allows no programmatic prompt; the app shows a one-time hint, then: Share ⬆︎ → Add to Home Screen → Add.
• The banner only appears on a fresh page load — reload once if you don't see it.

Updating the installed app

Service worker caching: hashed assets/* are cache-first (immutable), index.html and / are network-first with a cache fallback. After deploying a new build, the installed app picks it up on the next open; no manual cache clearing is needed. If you ever need to force it, bump the CACHE version in public/sw.js.

Glance integration

Glance can show the cluster on a self-hosted dashboard with the community widget sparkdash-dgx-cluster (contributed by @linxichen). One custom-api card renders GPU temperature and usage, VRAM, generation tok/s, KV-cache usage, and uptime for a head node plus one worker, reading /api/sparks/:id/metrics directly:

- type: custom-api
  title: sparkDash · DGX Cluster
  cache: 30s
  url: ${SPARKDASH_HEAD_URL}
  subrequests:
    node2:
      url: ${SPARKDASH_NODE2_URL}
  options:
    dashboardUrl: ${SPARKDASH_DASHBOARD_URL}
  template: |
    # paste the template from the widget's README

| Variable | Value | |----------|-------| | SPARKDASH_HEAD_URL | Metrics URL of one unit — https://sparkdash.example.com/api/sparks//metrics | | SPARKDASH_NODE2_URL | Metrics URL of a second unit. For a single Spark, drop the subrequests map and the widget's second node block | | SPARKDASH_DASHBOARD_URL | Dashboard base URL, used for the card's title link |

/api/sparks/:id/metrics is unauthenticated on loopback, the same as the rest of the dashboard. Read Security before exposing it beyond a trusted network.

Quality bench

The Quality button on the LLM card scores whatever model the port is serving on a fixed suite, so you can compare models, quantizations and KV-cache formats over time. Every item comes from a seeded PRNG and has a stable id: two runs are the same questions, paired item by item.

Every request is an OpenAI chat completion at temperature 0 with a fixed seed. Thinking is switched with the same flags the other benches use.

| Category | Items | Thinking | max_tokens | Scored by | |----------|-------|----------|------------|-----------| | QA | 150 (30 fixed + 120 generated: multiplication, add/subtract, string reversal, letter counts, sorting, binary, date offsets, LCM) | off | ≥ 64 | Accepted answer as a case-insensitive substring; counts, binary and LCM must be the last integer in the reply | | Reasoning | 40 multi-step word problems | on | 8192 | Last Answer: , line in the final answer (not the reasoning) | | Arithmetic chain | 40 ten-step chains (multiply, add, subtract, remainder, floor divide) | on | 12288 | Last Answer: (markdown and thousands commas allowed) | | State tracking | 40 token-transfer stories, 5 people, 25 events | on | 12288 | Last Answer: | | GSM8K | 200 grade-school maths problems, a fixed seeded sample of the public GSM8K test set (MIT) | on | 8192 | Last Answer: ($, markdown and thousands commas allowed) | | MMLU | 285 multiple-choice questions, 5 per subject across all 57 MMLU subjects, a fixed seeded sample of the public test set (MIT) | off | 512 | Last Answer: (or a bare letter) | | Instruction following | 40 prompts with 2–4 verifiable formatting rules (case, length, bullets, paragraphs, keywords, endings…) | off | 1024 | Each rule is checked by code; an item passes only when every rule holds. Off by default so older overall scores stay comparable | | Long-context recall | Items per size (default 2) at 8k–256k (default 32k, off by default) | off | 512 | 16 animal codes spread through filler text; 4 are corrected near the end. Passes when all 16 latest codes come back; overwritten codes are counted as stale |

The overall score is the mean of the category percentages. Long sizes that do not fit the model context (from the live probe or /v1/models) are skipped and listed.

The dialog shows live progress, the per-category table (mean completion tokens and how many replies hit max_tokens for the thinking categories, keys found and stale answers for long recall), and a collapsible per-item table with reply excerpts. Pick an earlier run under Compare with to see both scores, how many replies were identical, how many items only one run got right, and the exact two-sided McNemar p-value. When p ≥ 0.05 the row reads difference within noise. Copy results copies a plain-text summary.

The last 30 runs per unit are kept in config/quality-bench-history.json, with the label, model id, settings, and per-item pass/fail and reply hash. One quality run per unit at a time; it cannot overlap a decode or prefill bench or a showcase. A full default run is about 755 requests, and the thinking categories can take a while on slow models.

_{Quality bench by Mia's AI Lab.}

Prometheus

Opt-in (default off). Settings → Prometheus metrics serves every unit's metrics at GET /metrics in the Prometheus text format (0.0.4), so Prometheus, VictoriaMetrics or Grafana Agent can scrape sparkDash and keep history beyond what the dashboard shows. While the setting is off the path answers 404.

/metrics goes through the same auth as the REST API: open on a loopback install, and on a remote bind with SPARKDASH_TOKEN set it needs the token as a bearer header.

scrape_configs:
  - job_name: sparkdash
    scrape_interval: 15s
    static_configs:
      - targets: ["sparkdash.lan:5555"]
    # Only when SPARKDASH_TOKEN is set on a non-loopback bind:
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/sparkdash.token

The exporter reads the snapshot sparkDash already holds, so scraping adds no SSH or nvidia-smi calls; values are as fresh as the poll interval. Names follow node_exporter / DCGM-exporter practice: base units (bytes, seconds, °C, watts, 0–1 ratios), _total only on counters. Every series carries unit (the unit id), unit_name, kind (spark / host) and role (head / worker / standalone). Memory and disk sizes are collected in MiB and converted to bytes.

| Family | Type | Extra labels | Notes | |--------|------|--------------|-------| | sparkdash_up | gauge | | 1 when the unit answered its last liveness check. An unreachable unit exports sparkdash_up 0 and nothing else | | sparkdash_uptime_seconds | gauge | | Host uptime | | sparkdash_collector_last_success_timestamp_seconds | gauge | collector | Unix time of the last poll of each collector (gpu, cpu, ram, memory, network, storage, llm, roce) that returned real data, not zeroed defaults. A stuck collector stops advancing while sparkdash_up stays 1, so alert on time() - sparkdash_collector_last_success_timestamp_seconds > 60. Absent until a collector has succeeded | | sparkdash_gpu_info | gauge | gpu, name, uuid | Always 1. gpu is the nvidia-smi index for an NVIDIA card; an AMD card (amd-smi) is its index plus 1000, so gpu="1000" is the first AMD card. Per-card chart history uses the same number (gpu.1000.*) | | sparkdash_gpu_utilization_ratio | gauge | gpu | 0–1 | | sparkdash_gpu_temperature_celsius | gauge | gpu | | | sparkdash_gpu_power_watts, sparkdash_gpu_power_limit_watts | gauge | gpu | | | sparkdash_gpu_memory_used_bytes, sparkdash_gpu_memory_total_bytes | gauge | gpu | GB10: the GPU's share of / the whole unified pool | | sparkdash_gpu_throttled | gauge | gpu, reason | 1 while clocks are limited for thermal, power or hw | | sparkdash_memory_available_bytes | gauge | | Headroom for GPU work: unified-pool MemAvailable on a GB10, free VRAM on a discrete GPU | | sparkdash_cpu_utilization_ratio | gauge | | 0–1 | | sparkdash_cpu_temperature_celsius | gauge | sensor | A GB10 has no CPU package sensor; sensor="acpitz" says it is a board zone | | sparkdash_ram_used_bytes, sparkdash_ram_total_bytes | gauge | | | | sparkdash_network_receive_bytes_per_second, …_transmit_bytes_per_second | gauge | interface | Interfaces hidden in sparkDash are skipped | | sparkdash_disk_used_bytes, …_available_bytes, …_total_bytes | gauge | mount, device | Devices hidden in sparkDash are skipped | | sparkdash_llm_up | gauge | port, backend, model | One per LLM endpoint (head and standalone units) | | sparkdash_llm_generation_tokens_per_second, sparkdash_llm_prefill_tokens_per_second | gauge | port, backend, model | | | sparkdash_llm_kv_cache_usage_ratio | gauge | port, backend, model | Where the backend reports it | | sparkdash_llm_requests_running, sparkdash_llm_requests_waiting | gauge | port, backend, model | Where the backend reports it | | sparkdash_llm_generated_tokens_total, sparkdash_llm_prompt_tokens_total, sparkdash_llm_cached_prompt_tokens_total | counter | port, backend, model | The engine's own lifetime counters — they reset when the engine restarts, which rate() handles |

Unknown is left out rather than written as zero: a failed GPU read, a sensor that does not exist, or a field the LLM backend does not publish produces no sample.

Alert notifications

Opt-in (default off): Settings → Alert notifications → Configure…. The server sends a notification to ntfy, Discord, Slack or a JSON webhook when a problem has held for a while, and again when it clears — whether or not a dashboard is open.

What notifies

Nothing new is detected here. The conditions are the ones sparkDash already shows, with their own thresholds:

• the Health findings on the unit cards (server/health/HealthEvaluator.js: GPU temperature, unified memory, stuck low-power state, Xid / OOM, RoCE link and loss, slow link, several models loaded), and
• the Overview exceptions (src/shared/fleetExceptions.js, the strip's own code): unit offline, thermal or hardware throttle (never the software power cap), storage at or above 90 %, LLM unavailable, tailnet unavailable.

The Activity log keeps recording what happened, as before; on top, every notification sent or failed is an alert.notify.sent / alert.notify.failed event (category Alert notifications), which names the channel and the alert — never the URL, never what the server answered.

Channels

| Type | What is sent | |------|--------------| | ntfy | POST plain text to the topic URL with Title, Priority (5 critical, 4 warning, 3 resolved) and Tags headers — ntfy.sh or self-hosted | | discord | Discord webhook JSON: content plus one embed per alert (mentions disabled) | | slack | Slack incoming-webhook JSON { "text": … }; &, < and > are escaped, so unit names and remote text cannot add mentions or links | | webhook | Generic JSON { source, status, severity, title, reminder, sentAt, alerts: [{ status, severity, rule, ruleName, unit, unitId, label, summary, startsAt, endsAt }] } |

Everything that fires or clears in one evaluation goes out as one message per channel, filtered by the channel's minimum severity. Remind while still firing (off by default) repeats firing alerts every 1 h / 4 h / 12 h / 24 h. Text that comes from elsewhere (unit names, SSH errors, kernel Xid lines, model ids) is cut to 120 characters.

How sends are made:

• Outbound policy. A channel URL must be http(s) and pass the same target policy as units and LLM endpoints (server/validate.js): LAN, Tailscale and loopback addresses are fine (a self-hosted ntfy), link-local / cloud-metadata, multicast and unspecified addresses are refused. The host is checked when the channel is saved and resolved again on every send; the connection goes to exactly the address that was checked, so a DNS record that changes in between cannot redirect it.
• No redirects. A 3xx answer is a failed send (redirect not followed).
• No response bodies. A failure is reported as a status code or a fixed reason (HTTP 403, timed out after 10s, connection failed (ECONNREFUSED)) — in the dialog, the log and Activity alike.
• No retries. Each message is sent once. A failed send is logged, shown on the channel and recorded in Activity (alert.notify.failed), but the alert still counts as notified: the channel next hears about it when it escalates, clears, or — with Remind while still firing on — at the next reminder. Use Send test to check a channel after changing it.
• Channel URLs may point at your LAN. Like unit and LLM targets, a channel can be a private or loopback address (a self-hosted ntfy), so POST /api/alerts/test shows, to anyone who holds the dashboard token, whether a host and port on those networks answers (a status code or a fixed reason, never the response body). Link-local, metadata, unspecified and multicast addresses are refused, and the connection is pinned to the addresses that were checked. Set SPARKDASH_TOKEN if the dashboard is reachable beyond your own machine.

Where the configuration lives

Channel URLs are credentials, so they are stored encrypted in config/sparks-secrets.json next to the unit passwords and LLM API keys, and the API only ever shows them masked (https://ntfy.sh/…: the host and a fixed mask, nothing of the topic or token; saving the masked value unchanged keeps the stored URL). config/alerts.json (ALERTS_JSON_PATH, mode 0600) holds the rest: rule switches and waits, channel names and types, and the firing set. A plaintext URL left in alerts.json by an earlier build is moved into the encrypted store when the server starts.

While notifications are off, nothing is evaluated or sent, and PUT /api/alerts/config and POST /api/alerts/test answer 409. While on, the WebSocket snapshot carries alerts.active and the Overview exceptions strip lists the server's alerts with their real durations.

In the browser

Settings → Tab badge / Desktop notifications (per browser, both off by default): the tab title shows how many fleet alerts are active — (2) sparkDash — with a red (critical) or amber dot on the icon, and a desktop notification pops up when one becomes active while the page is open. Both count every alert the strip knows of — the server's alerts while notifications are on, otherwise the strip's own conditions with the temperature unit applied — including offline units, storage, LLM and Health while Show active fleet exceptions is off and the strip itself shows throttling only. Conditions the browser derives itself must hold for 30 s before they count and stay clear for 30 s before they stop, so a flapping reading notifies once. Browsers only allow notifications on HTTPS or localhost — on a plain-HTTP LAN address, use an SSH tunnel, Tailscale Serve or an HTTPS proxy, or an ntfy channel.

RoCE / RDMA monitoring

Spark units with RDMA devices (the CX7 ports) get a RoCE / RDMA panel on their page, with no switch integration and no root. Per port: link state and rate, MTU, RX/TX traffic, interface errors and drops, PFC priorities, QoS trust mode (PCP or DSCP) and the DSCP map, global pause settings, and the RDMA counters (out of buffer, sequence errors, ACK timeouts, ECN-marked packets, CNPs sent and handled) plus ethtool discards, CRC errors and pause frames, with the change since the last sample.

Everything comes from the Spark itself: /sys/class/infiniband, /sys/class/net, ethtool -S / -a and mlnx_qos -i. The fast part (link, counters, traffic) is read about every 5 s; the slow part (ethtool counters, PFC and trust) every 30 s. Units without RDMA devices are skipped and re-checked every 10 minutes. The API exposes it as metrics.roce in /api/sparks/:id/metrics.

Two health findings use it: a RoCE link went down (a port that was up, critical) and RoCE is dropping packets (RDMA loss counters such as out_of_buffer or sequence errors rising for three samples in a row, or port discards / CRC errors rising on two consecutive 30 s readings; a single dropped frame is not reported). Ports that never came up are not reported. Switch telemetry (PFC and ECN statistics on the switch side) is not part of this; it could be added later as an optional integration.

Intel Arc GPUs

Discrete Intel GPUs bound to the Linux xe driver show up in the GPU panel next to NVIDIA and AMD cards, locally and on SSH units. Everything is read from sysfs and hwmon, with no vendor tool: temperature, fan, clock and power cap directly; power in watts from the card's energy counter (a delta between polls); usage from the GT idle residency. Integrated GPUs (PCI bus 00) and i915 cards are not read. Intel cards have their own index range (2000 and up) and vendor: "intel".

VRAM. Exact VRAM use lives in root-only debugfs (/sys/kernel/debug/dri//tile/vram_mm). Reading it is off by default: with Settings → Intel Arc VRAM via sudo off, no sudo runs on any host and the card shows its PCI BAR size as the total (vramSource: "pci-bar", usage unknown). When it is on, the probe runs sudo -n cat for tile0 first and then tile1 to tile3, one path per call, and the first file that answers wins. The result is cached for 45 seconds, and a host where sudo -n fails is not asked again for 10 minutes, so neither case floods the host's auth log.

The sudoers rule. Grant the SSH user that one command on the exact files and nothing wider. A bare cat grant (or ALL) would let that user read any file as root. Pin each card's debugfs files by its PCI address (lspci -D | grep -i -E 'vga|display', for example 0000:3a:00.0):

# /etc/sudoers.d/sparkdash-intel-vram   (edit with: sudo visudo -f /etc/sudoers.d/sparkdash-intel-vram)
Cmnd_Alias SPARKDASH_INTEL_VRAM = \
    /usr/bin/cat /sys/kernel/debug/dri/0000\:3a\:00.0/tile0/vram_mm, \
    /usr/bin/cat /sys/kernel/debug/dri/0000\:3a\:00.0/tile1/vram_mm, \
    /usr/bin/cat /sys/kernel/debug/dri/0000\:3a\:00.0/tile2/vram_mm, \
    /usr/bin/cat /sys/kernel/debug/dri/0000\:3a\:00.0/tile3/vram_mm
sparky ALL=(root) NOPASSWD: SPARKDASH_INTEL_VRAM

Use the full path of cat that command -v cat prints for that user (/usr/bin/cat or /bin/cat; they are the same file on merged-/usr systems) and escape the colons of the PCI address with a backslash, as above. Do not use in the paths: in sudoers a wildcard also matches spaces and /, so .../dri//tile0/vram_mm would also allow cat of other files. On sudo 1.9.10 or newer a regular expression pins the same set: /usr/bin/cat ^/sys/kernel/debug/dri/[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]/tile[0-3]/vram_mm$. Check the rule with sudo -n cat /sys/kernel/debug/dri//tile0/vram_mm as that user. The local unit (the container with pid: host) runs as root and needs no rule.

Health. Intel cards are shown and counted in the power and VRAM totals but are not judged by the NVIDIA-tuned health thresholds. If nvidia-smi is installed but returns no devices on a host that has an Intel card, the GPU collection is reported as failed (the Intel card is still shown), so a broken NVIDIA driver does not look healthy; a host with no nvidia-smi at all is treated as Intel-only.

Windows PCs

A Windows PC with an NVIDIA GPU can be monitored with no agent. On the PC:

1. Install the OpenSSH Server (Settings → Apps → Optional features → OpenSSH Server), then start it and set it to start automatically: Start-Service sshd; Set-Service sshd -StartupType Automatic in an elevated PowerShell. Windows Firewall opens port 22 for it.
2. Make sure the NVIDIA driver is installed (nvidia-smi works in a terminal).
3. In sparkDash choose Add Spark / GPU host, set Unit type to Windows PC with an NVIDIA GPU, enter the PC's LAN IP and your Windows user (a password works; for key login see the administrator-account note below).

icacls.exe "$env:ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"

A refused key on a Windows unit also shows this hint in the connection error.

sparkDash then runs two short PowerShell scripts over SSH per poll (no files are installed): GPU temperature, utilisation, power, VRAM and GPU processes from nvidia-smi, and RAM, uptime, CPU load, disks (fixed drives) and network adapters from Windows' CIM classes. Shutdown works (shutdown.exe /s). Not available on Windows: CPU temperature (Windows exposes no unprivileged sensor; CPU power is estimated from load), automatic Wake-on-LAN MA

... (README truncated for length)