• Node.js 20 or newer — free, one installer from . This

• Google Chrome or Microsoft Edge, in a real browser window — not an

• Optional: an ElevenLabs API key — a good add-on, not a requirement. It

Run npm run setup after cloning and it checks all of this for you, in plain language.

Quick start

First, install, then start it:

npm install
npm start          # runs the brain and the face together

Then open the URL it prints (http://localhost:5173) in Chrome, click INITIALISE, and say “Hey Jarvis”.

Prefer two terminals? Run them separately instead:

npm install

Terminal 1 — the brain:

npm run bridge

Terminal 2 — the face:

npm run dev

Then open the app in a real Chrome or Edge window:

open http://localhost:5173

Click INITIALISE, allow the microphone when asked, and say "Hey Jarvis".

It has to be a real browser window. Embedded preview panes block the

microphone, so JARVIS will look perfectly alive and simply never respond.

How it works

JARVIS is two processes. The browser is the face and the voice; the bridge is the brain and the hands.

┌─ browser (the face) ───────────────┐        ┌─ bridge (the brain) ─────────────┐
  │  "Hey Jarvis" wake word            │        │  Node · bridge/server.mjs        │
  │  local VAD  →  speech to text      │   ws   │  Claude Agent SDK                │
  │  reactor UI (Three.js + GLSL)      │◄─────► │   = Claude Code, headless        │
  │  text to speech                    │  8787  │  spawns your MCP servers         │
  │  heads-up display                  │        │  permission gate (decideTool)    │
  └────────────────────────────────────┘        └──────────────────────────────────┘

Everything you see and hear happens in the browser. The bridge is a single Node process (bridge/server.mjs) that runs the Claude Agent SDK (@anthropic-ai/claude-agent-sdk) — this spawns the real claude CLI as a child process, so the brain literally is Claude Code, headless. They talk over a WebSocket (plus a few HTTP endpoints) on ws://localhost:8787.

Why a bridge at all? A browser tab cannot spawn the local stdio MCP servers — higgsfield, elevenlabs, android, playwright, exa, serper, and the rest. The bridge can. And because it is the Agent SDK, it authenticates off your existing Claude Code login: no API key, billed to that same Claude account.

The model. claude-opus-5 at effort medium by default. Override with the JARVIS_MODEL and JARVIS_EFFORT environment variables. On startup the bridge prints its choice, e.g. [jarvis] model claude-opus-5 · effort medium.

The voice pipeline

The loop is designed so that nothing silently dies and barge-in feels natural.

• Detection is local. An energy-based voice-activity detector

• Transcription has two tiers, chosen automatically at boot. The browser asks

• Speaking uses the ElevenLabs voice when a key is present, and the

So it works with no keys and auto-upgrades when a key appears — there is no flag to set. Capability detection lives in src/lib/capabilities.ts, which probes the bridge's GET /health (returning { ok, tts, stt }, both tracking the ElevenLabs key) once at boot and picks the engines.

What JARVIS can do

Beyond answering, JARVIS reaches every MCP server in your Claude Code configuration, and can drive his own interface.

Your tools

Every server in your ~/.claude.json is handed to the SDK explicitly. Depending on what you have installed, that is roughly:

• Web & search — exa, serper, serpapi
• Images & video — higgsfield, openrouter-image, palmier-pro
• Voice — elevenlabs
• Your phone — android
• The browser — playwright

• "What's happening in AI this week?"
• "Generate an image of the Mark VII suit."
• "Take a screenshot of my phone."
• "Open my GitHub notifications."

Note on account connectors. Servers you added through your **claude.ai

account** (mail, drive, calendar and other account connectors) are not in ~/.claude.json,

but the CLI fetches them on its own and they join the session from the second

turn. The bridge keeps them out by default; set JARVIS_ACCOUNT_CONNECTORS=1

to let them in, and JARVIS_MCP_SERVERS to choose which ~/.claude.json

servers JARVIS gets.

JARVIS controls the interface

He drives the UI through MCP tools the bridge exposes:

• ui_theme — accent, background, per-phase colours
• ui_reactor — colour, scale, intensity, spin, and style (ring | sphere | wire), visibility
• ui_orbit — put images in orbit around the reactor
• ui_chrome — show or hide rails, transcript, badges
• ui_effect — glitch | pulse | scan | shake | flash
• ui_screen — clear
• ui_reset — back to defaults

The heads-up display

JARVIS authors panels with a display tool against a fixed .hud-* design system. The browser sanitises the markup (DOMPurify, a class allowlist and a strict CSP) before rendering. Rich media works — images, , and YouTube/Vimeo embeds. Remote images and video are fetched server-side through the bridge (/img and /media, both SSRF-guarded), so hotlink-blocked news thumbnails still appear and the page never beacons your IP to a host the model chose.

Controls

| Key / phrase | Does | |---|---| | "Hey Jarvis" | Wake him | | Space | Talk without the wake word | | Just speak | Interrupt him mid-sentence (barge-in) | | V | Cycle the browser voice | | Escape | Stand down | | D | Live diagnostics panel | | T | One-line audio self-test |

The boot sequence

Power-up plays a four-beat Iron Man start-up (src/ui/Boot.tsx): an "INITIATING SYSTEM" status bar with a segmented progress bar and boot log; then concentric reticle rings resolving into "J.A.R.V.I.S"; then a suit schematic; then the triangular arc reactor lighting up — with a start-up sound under it (public/audio/boot-music.mp3).

Configuration

Everything is optional in bridge mode. Frontend settings live in .env.local (copy .env.example); bridge settings are environment variables.

Bridge

| Variable | Default | Effect | |---|---|---| | JARVIS_BRIDGE_PORT | 8787 | Port for the WebSocket + HTTP endpoints | | JARVIS_MODEL | claude-opus-5 | Model to run | | JARVIS_EFFORT | medium | Reasoning effort | | JARVIS_ALLOW_WRITES | off | 1 allows effectful tools (see below) | | JARVIS_MCP_SERVERS | all configured | Comma-separated ~/.claude.json servers to pass on; empty passes none | | JARVIS_ACCOUNT_CONNECTORS | off | 1 lets your claude.ai account's connectors join the session | | JARVIS_ALLOWED_ORIGINS | local dev | Extra WebSocket origins to accept | | JARVIS_ALLOW_NO_ORIGIN | off | Accept connections with no Origin header | | JARVIS_WORKSPACE | ~/.jarvis/workspace | The agent's working directory; file tools are confined to it and the temp directory | | JARVIS_FILE_ROOTS | — | Extra folders the file tools and the /file endpoint may use | | JARVIS_VOICE_ID | — | ElevenLabs voice id | | ELEVENLABS_API_KEY | — | Optional; enables the ElevenLabs voice + Scribe |

Frontend (.env.local)

| Variable | Effect | |---|---| | VITE_BACKEND | bridge (default) or direct | | VITE_BRIDGE_URL | Where to reach the bridge | | VITE_TTS_ENGINE | system or kokoro | | VITE_KOKORO_VOICE | Voice for the Kokoro engine | | VITE_USE_ELEVENLABS | Force the ElevenLabs voice on | | VITE_ANTHROPIC_API_KEY | Direct mode only |

Adding an ElevenLabs key

You do not have to touch a flag. Either:

• Set ELEVENLABS_API_KEY on the bridge before starting it, or
• Add the key to your elevenlabs MCP server's env in ~/.claude.json — the

Either way, /health starts reporting the capability, the browser picks it up on the next boot, and both the voice and transcription upgrade automatically.

Enabling actions

The tool gate starts read-only. Search, generation and lookups run freely; anything effectful — send, tap, delete, install, pay — is denied. Voice is a poor interface for a confirmation dialog, so the decision is made ahead of time in decideTool() in bridge/server.mjs, not at the moment of use. The bridge sets settingSources: [], which makes its own gate the only authority — filesystem settings and any global bypassPermissions cannot override it.

To allow effectful tools (phone, browser driving, sending), run the bridge this way instead:

npm run bridge:writes

Read decideTool() before you do. "Hey Jarvis, clean up my downloads folder"

means something rather different with writes enabled.

Troubleshooting

I can't hear him, or he can't hear me. Press D for the diagnostics panel — it states plainly whether he is hearing you and whether he is producing sound. Press T for a one-line audio self-test.

No voice at all. You must be in Chrome or Edge, in a **real browser window (not an embedded preview), and you must have allowed the microphone**.

Bridge not reachable. Check that npm run bridge is still running in its terminal, and that nothing else is holding port 8787.

Running the bridge in WSL, with Chrome on Windows. Browser control works: the Windows native host exposes a named pipe rather than a Unix socket, and the bridge reaches it through a short PowerShell relay (WSL interop must be on, which it is by default). Open the app at http://127.0.0.1:5173 if localhost is slow to load — Windows tries IPv6 loopback first, and WSL does not forward it.

Security

All of this lives in bridge/server.mjs:

• The WebSocket accepts only local dev origins (add more with

• /file, /img and /media validate the scheme, confine to allowed roots,

• The tool gate (decideTool) is default-deny for effectful MCP tools.
• A strict CSP in index.html; model-authored panel HTML is sanitised.

Credits & licence

MIT.

The boot sound and any tracks in public/audio/ ship with the project for the demo. If you go on to monetise something built on this, clearing the rights to that audio is your responsibility.