Profile
Back to NewsBack
GitHub Trending 22 min
Reader Mode
google/three-live: AI-first Native 3D, CAD & spatial computing. Runtime and JIT compiler for Android (Filament & Jetpack XR), iOS/visionOS (RealityKit), and WebGL2/WebXR.

google/three-live: AI-first Native 3D, CAD & spatial computing. Runtime and JIT compiler for Android (Filament & Jetpack XR), iOS/visionOS (RealityKit), and WebGL2/WebXR.

Disclaimer: This is not an officially supported Google product.

Three-Live

AI-First Native 3D, CAD & Spatial Computing: On-device JIT compiler turning live, AI-generated Three.js into native Android (Google Filament & Jetpack XR), iOS & visionOS (Apple RealityKit), and WebXR.

CI</a> Release</a> License</a>

🧭 Why Three-Live? • ⚡ Quickstarts (Android • iOS • CLI • Web) • 📦 Install (JitPack • SwiftPM • Cargo • npm) • ▶ Live 3D Studio • 🧩 Domain Kits Catalog • 🥽 Live Spatial XR Studio • 📚 Docs

iOS & visionOS · RealityKit | Android XR · Google Filament :-----------------------------: | :------------------------------: ![iOS RealityKit Demo][ios-gif] | ![Android XR Demo][android-gif]

[ios-gif]: examples/assets/slide-product-ios.gif [android-gif]: examples/assets/slide-xr-android.gif

LLMs and developers excel at writing concise Three.js code—not hundreds of lines of platform-specific native 3D, shader, and AR boilerplate. Three-Live compiles standard Three.js (THREE.Mesh, THREE.CSG, userData.xr, KHR_interactivity) on-device in a sandboxed Rust VM and renders natively in Google Filament and Jetpack XR SceneCore on Android, **Apple RealityKit and Quick Look on iOS and visionOS, headless 4-up PNG contact sheets (three-live-cli) for AI coding agents, and WebGL2 / WebXR** on the Web.

--------------------------------------------------------------------------------

Quickstarts

1. Run Three.js natively on Android (Google Filament & Jetpack XR SceneCore)

Compile standard Three.js code over zero-copy JNI and drive native **Google Filament rendering, 60 Hz physics, and Jetpack XR SceneCore** spatial entities:

import com.google.threelive.FilamentHelper
import com.google.threelive.XRLiveRuntimeSession
import com.google.threelive.XRLiveSceneCoreHelper

val threeJsCode = """ const core = new THREE.Mesh( new THREE.IcosahedronGeometry(0.55, 3), new THREE.MeshStandardMaterial({ color: 0x38bdf8, metalness: 0.85, roughness: 0.15 }) ); core.userData.xr = { grab: '2-hand', orbit: { speed: -0.4 } }; scene.add(core); """.trimIndent()

// Compile Three.js to a zero-copy GLB ByteBuffer and load directly into Google Filament val holder = FilamentHelper.loadModelZeroCopy(threeJsCode) { buffer -> assetLoader.createAssetFromBinary(buffer) }

// Step live KHR_interactivity graphs, 60 Hz rigid-body physics, and spatial traits per frame val runtime = XRLiveRuntimeSession.fromGlb(holder.byteBuffer) runtime.step(dtSeconds = 1.0 / 60.0)

// Or attach interactive spatial traits (grab, scale, billboard, anchor) in Android XR SceneCore XRLiveSceneCoreHelper.attachTraitsFromGlb(xrSession, entity, holder.byteBuffer)

2. Run Three.js natively on iOS, macOS, and visionOS (Apple RealityKit)

Compile the exact same Three.js code into Apple RealityKit (ARView / RealityView) or .usdz:

import ThreeLiveCompiler
import RealityKit

@MainActor func loadSpatialScene(from threeJsCode: String, into arView: ARView) async throws -> XRLiveRealityKitSession? { let glb = try ThreeCompiler.compileToGLB(script: threeJsCode) let entity = try await RealityKitHelper.loadEntity(fromGLB: glb) XRLiveRealityKit.installTraits(on: entity, in: arView)

// Bind live KHR_interactivity, 60 Hz physics, and spatial traits to RealityKit SceneEvents.Update let session = XRLiveRealityKitSession(glbData: glb, root: entity) session?.installInputTargets() session?.start() return session }

3. Verify & compile Three.js headlessly for AI agents (three-live-cli & MCP server)

Render a 4-up orthographic/camera contact sheet (zero GPU required), run the spatial linter for JSON frustum/overlap telemetry, compile scene.js to .glb / .usdz, generate a scene from a reference image (from-image), or attach three-live-cli mcp as a zero-dependency Model Context Protocol (MCP) stdio server (three_live_validate, three_live_render, three_live_convert, three_live_from_image, three_live_bake_textures, three_live_sim, three_live_link, and the three_live_gym_reset / three_live_gym_step / three_live_gym_close agent gym) in Gemini CLI, Google Antigravity, Claude Code, or any MCP client. The agent gym lets an agent play a scene step by step (act, observe, react) to check a level is winnable; "gym" is the reinforcement-learning term for such a reset/step environment, and no ML is involved. Not using MCP? three-live-cli gym reset level.js --session ep.json and three-live-cli gym step --session ep.json --actions '[...]' do the same, one command per step. three-live-cli link scene.js --qr prints a playground link (and a terminal QR code) that opens the scene on a phone, one tap from AR. Building from a photo, mockup, sketch, or blueprint? Pass --reference (or reference in MCP) to get a side-by-side + aligned-overlay + silhouette-diff sheet with IoU and proportion guidance, or run from-image to synthesize and verify a procedural Three.js scene directly from an image:

$ npx three-live-cli render scene.js --view=quad --grid --annotations -o contact_sheet.png
$ npx three-live-cli render scene.js --reference=blueprint.png --view=front -o compare.png
$ npx three-live-cli from-image photo.jpg -o scene.js --visual-turns 2 --render compare.png
$ npx three-live-cli validate scene.js --json
$ npx three-live-cli link scene.js --qr
$ npx three-live-cli convert scene.js -o model.usdz
$ npx three-live-cli textures bake scene.js -o scene.textures.json   # AI textures (needs THREE_LIVE_AI_IMAGE_ENDPOINT)
$ npx three-live-cli convert scene.js --texture-pack scene.textures.json -o model.glb

Or register the built-in MCP server for closed-loop visual + spatial AI verification:

$ claude mcp add three-live -- npx -y -p three-live three-live-cli mcp

4. Run on the Web ( and )

On the web, (< 35 KB gz viewer, < 88 KB gz with the live Three.js compiler) and (< 100 KB gz) render the same Three.js scenes in WebGL2 and 6DoF WebXR with 1-tap mobile AR handoff:

// Standard Three.js scene (runs identically in Android, iOS, CLI, and Web)
const mesh = new THREE.Mesh(
  new THREE.TorusKnotGeometry(0.4, 0.14, 64, 16),
  new THREE.MeshStandardMaterial({ color: 0x38bdf8, metalness: 0.8, roughness: 0.15 })
);
mesh.userData.xr = { grab: '1-hand', billboard: 'capsule' };
scene.add(mesh);

Add three-live-ai.min.js (< 35 KB gz; built into the full bundles) to let users reshape a scene with any LLM: every edit is compiled in the sandbox first, auto-repaired once on failure, and only then hot-swapped, so a bad generation never breaks the live scene.

<three-live capabilities="scripts ai network:https@https://your-backend.example"
    ai-endpoint="https://your-backend.example/chat"  ai-model="gemini-2.5-flash">
  <script type="text/three">/ your scene /</script>
</three-live>
<script type="module">
  const el = document.querySelector('three-live');
  const result = await el.editScene('make the roof red and add a chimney');
  if (!result.applied) console.warn(result.error);  // live scene unchanged
  // el.undoEdit() reverts; el.aiProvider = { complete: async ({messages}) => '...' } plugs in any model.
</script>

ai-endpoint speaks the widely supported Chat Completions protocol (ai-provider="ai-chat"): your own backend proxy, Ollama or LM Studio (http://localhost:..., no account needed), vLLM, OpenRouter, or Gemini's compatibility endpoint. Keep API keys on your server; never put them in pages.

Tap & Speak: let users tap a mesh and say what to change. The voice model talks, and your code model writes the edit through an edit_scene tool, so edit quality never depends on the voice model.

<three-live capabilities="scripts ai microphone network"
    ai-endpoint="https://your-backend.example/chat"
    ai-select-on-tap ai-voice-button
    ai-voice-url="wss://your-backend.example/realtime">
  <script type="text/three">/ your scene /</script>
</three-live>
  • ai-select-on-tap: tapping a mesh sets selected-mesh (tap again to
clear), so "make this red" knows what "this" is.
  • ai-voice-button: a mic button. With ai-voice-url it toggles a realtime
voice conversation (ai-voice-provider="ai-realtime", the default, for any server speaking the Realtime WebSocket protocol, or "gemini-live"). Without it, it runs push-to-talk with the browser's speech recognizer (el.startVoiceEdit()).
  • From code: await el.connectVoice({provider: 'gemini-live', url}) or
await el.startVoiceEdit(). Progress arrives as three-live-voice events (listening, editing, transcripts, results).

Talk-to-Select: users can select by voice instead of tapping, so they can check that the right objects are highlighted before an edit.

  • Say "select the two red chairs on the left", "also select the lamp",
"deselect the small one", or "clear the selection". References are resolved locally with no model call: names (with synonyms and plurals), material colors, left / right / front / back from the current camera, top / bottom, size, ordinals ("the second mug"), counts ("both"), "all … except …", and lists ("the lamp and the sofa").
  • "Select the lamp and make it blue" selects first, then edits right away
when the reference was unambiguous. When it isn't, it asks ("Which one: Chair Left or Chair Right?") and changes nothing.
  • References the local matcher can't resolve ("the cup next to the sofa",
other languages) go to your code model, which picks from a compact scene index or asks a question.
  • Shift / Ctrl / Cmd-tap or a long press adds to the selection.
el.selectedMeshes holds it; three-live-select events report it.
  • Push-to-talk shows confirmation chips: each selected mesh with ×, buttons
for the candidates of a question, and Apply for an edit that is waiting on a model-resolved selection. Set ai-select-chips="off" to hide them. Realtime voice sessions read the selection back instead, using the select_meshes tool.
  • From code: await el.selectByVoice('the red chairs', {mode: 'add'}).
Image to Scene (compileFromImage): turn a photo, sketch, or blueprint into a live procedural Three.js scene across Web (await el.compileFromImage(image, {prompt, priors, visualTurns: 2})), Android (editor.compileFromImage(bitmap, prompt, priors)), iOS (await editor.compileFromImage(imageData: ...)), CLI (three-live-cli from-image --visual-turns 2), and MCP (three_live_from_image with visualTurns). Images are automatically downscaled and JPEG-normalized, compiled in the sandbox, verified against spatial quality gates (minimum 2 meshes, finite bounds, non-empty frustum coverage, and zero critical overlap errors) with one-turn auto-repair, optionally refined closed-loop against rendered REFERENCE | RENDER | OVERLAY 50/50 (+ 4-up orthographic/iso) sheets (visualTurns: 0..5), and hot-swapped into the scene (el.undoEdit() reverts). Optional metric spatial priors (userHeight, floorNormal, and 3D detected object bounding boxes from ARCore / ARKit / world-understanding, or a whole RoomPlan scan on iOS via AISpatialPriors.fromCapturedRoom(room), or any room scan via AISpatialPriors.fromRoomScan on Android) anchor real-world scale and placement.

AI textures (userData.xr.material.prompt): add a texture prompt next to a material preset. The preset is the offline fallback, so the scene always renders the same way without a network:

<three-live capabilities="scripts remote_content ai network" texture-pack="scene.textures.json"
    ai-image-endpoint="https://your-backend.example/v1/images/generations">
  <script type="text/three">
    const wall = new THREE.Mesh(new THREE.BoxGeometry(4, 2.5, 0.2),
        new THREE.MeshStandardMaterial());
    wall.name = 'TempleWall';
    wall.userData.xr = {material: {preset: 'brick',
        prompt: 'weathered Kyoto temple stone with moss', size: 512}};
    scene.add(wall);
  </script>
</three-live>

The compiler never calls an image model. Textures come from a *texture pack** (texture-pack="…json", el.texturePack = json, or an inline Chat with me