Profile
Back to NewsBack
GitHub Trending 8 min
Reader Mode
ngxson/wllama: WebAssembly binding for llama.cpp - Enabling on-browser LLM inference

ngxson/wllama: WebAssembly binding for llama.cpp - Enabling on-browser LLM inference

18 hours ago

wllama - Wasm binding for llama.cpp

WebAssembly binding for llama.cpp

👉 Try the demo app

👉 See the blog post introducing WebGPU support in llama.cpp and wllama

📄 Documentation

For changelog, please visit releases page

[!IMPORTANT]
> 🔥🔥 V3 is out, with WebGPU, multimodal and tool calling support. Read the V3 release guide
> For compatibility issues, please refer to @wllama/wllama-compat

Features

  • 🔌 OpenAI-compatible API (fully-typed built-in)
  • 🚀 WebGPU support
  • 🔥 Multimodal support (image and audio file input)
  • Speech support: text to speech, transcription and realtime audio
  • 🔥 Tool calling support
  • Can run inference directly on browser (using WebAssembly SIMD), no backend or GPU is needed!
  • No runtime dependency (see package.json)
  • Load a GGUF file as-is, even bigger than 4GB, no need to split it (models already split with llama-gguf-split work too)
  • Auto switch between single-thread and multi-thread build based on browser support
  • Inference is done inside a worker, does not block UI render
  • Pre-built npm package @wllama/wllama
Limitations:
  • To enable multi-thread, you must add Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy headers. See this discussion for more details.
  • One wasm instance has 4GB of memory. If the part of the model that is not on WebGPU is bigger, it is split across several wasm instances in workers (rpc_workers, automatic by default), which needs multi-thread, and JSPI or the compat build (on Firefox, see setCompat).

Code demo and documentation

Demo:

  • Basic usages with completions and embeddings: https://github.ngxson.com/wllama/examples/basic/ (source code)
  • Embedding and cosine distance: https://github.ngxson.com/wllama/examples/embeddings/ (source code)
  • Multimodal (vision) completion: https://github.ngxson.com/wllama/examples/multimodal/ (source code)
  • Tool calling: https://github.ngxson.com/wllama/examples/tools/ (source code)
  • Decision models: https://github.ngxson.com/wllama/examples/decision/ (source code)
  • Omni (voice chat, text to speech, realtime speech to text): https://github.ngxson.com/wllama/examples/omni/ (source code)

How to use

Use Wllama inside React Typescript project

Install it:

npm i @wllama/wllama

Install from this git repo

Wasm binaries do not come pre-built with this repo. You need docker installed on your machine to build them:

# recommend to clone as git submodule
git submodule add https://github.com/ngxson/wllama.git wllama
git submodule update --init --recursive

run the build

cd wllama npm ci npm run build:wasm && npm run build

Then, import the module:

import { Wllama } from '@wllama/wllama';
let wllamaInstance = new Wllama(WLLAMA_CONFIG_PATHS, ...);
// (the rest is the same with earlier example)

For complete code example, see examples/main/src/utils/wllama.context.tsx

NOTE: this example only covers completions usage. For embeddings, please see examples/embeddings/index.html. For audio, see Speech and realtime audio

Speech and realtime audio

These APIs need a model with audio support and its mmproj file, for example LFM3-Omni. Use getLoadedContextInfo() to see what the model can do (has_audio_input, has_audio_output, has_audio_stream, has_live_asr, audio_voices).

// text to speech, returns a WAV file. Use stream: true to get chunks as they are generated
const { audio, sample_rate } = await wllama.createSpeech({
  input: 'Hello world',
  voice: 'female',
});

// transcribe an audio file const { text } = await wllama.createTranscription({ file: await audioFile.arrayBuffer(), language: 'en', });

A realtime session uses the same events as the OpenAI realtime API and the /v1/realtime route of llama-server. It holds one slot until close() is called, and inference only runs while events() is being read.

const session = await wllama.createRealtimeSession();
await session.send({ type: 'session.update', session: { type: 'transcription' } });

// mono float samples at getLoadedContextInfo().audio_input_rate await session.appendAudio(samples); await session.send({ type: 'input_audio_buffer.commit' });

for await (const event of session.events()) { if (event.type === 'conversation.item.input_audio_transcription.delta') { console.log(event.delta); } else if (event.type === 'conversation.item.input_audio_transcription.completed') { break; } } await session.close();

There are 3 kinds of session:

  • Default: a conversation. Send response.create after the audio, the answer comes as response.output_text.delta and response.output_audio.delta events. Use decodeRealtimeAudio() to read the audio
  • session.update with type: 'transcription': each committed audio buffer is transcribed on its own
  • createRealtimeSession({ intent: 'transcription' }): live ASR, the text comes out while the audio goes in. Needs has_live_asr, LFM3-Omni does not have it

WebGPU support

WebGPU support is introduced via PR #215.

Upon updating to V3.1, WebGPU will be enabled automatically. By default, all layers will be offloaded to GPU. If the model is too big to fit into VRAM, you can manually adjust the number of layers via the n_gpu_layers parameter of LoadModelParams. Example:

// (optionally) will allow running WebGPU on Firefox via compat mode; performance will be significantly degraded
wllama.setCompat('default', 'firefox_safari');

await wllama.loadModel(files, { n_gpu_layers: 4, // meaning 4 layers are offloaded to GPU; set to 0 to disable GPU inference });

Prepare your model

  • It is recommended to use the GGUF file as-is, there is no need to split it anymore. A file of 4GB or more is split in memory by wllama, and a model that does not fit in one wasm instance is spread across workers (see Limitations above). See the "Split model" section below if your model is already split.
  • It is recommended to use quantized Q4, Q5 or Q6 for balance among performance, file size and quality. Using IQ (with imatrix) is not recommended, may result in slow inference and low quality.

Simple usage with ES6 module

For complete code, see examples/basic/index.html

import { Wllama } from './esm/index.js';

(async () => { const CONFIG_PATHS = { default: './esm/wasm/wllama.wasm', }; // Automatically switch between single-thread and multi-thread version based on browser support // If you want to enforce single-thread, add { "n_threads": 1 } to LoadModelConfig const wllama = new Wllama(CONFIG_PATHS); // Define a function for tracking the model download progress const progressCallback = ({ loaded, total }) => { // Calculate the progress as a percentage const progressPercentage = Math.round((loaded / total) * 100); // Log the progress in a user-friendly format console.log(Downloading... ${progressPercentage}%); }; // Load GGUF from Hugging Face hub // (alternatively, you can use loadModelFromUrl if the model is not from HF hub) await wllama.loadModelFromHF( { repo: 'ggml-org/models', file: 'tinyllamas/stories260K.gguf' }, { progressCallback } ); const response = await wllama.createChatCompletion({ messages: [{ role: 'user', content: elemInput.value }], max_tokens: 50, temperature: 0.5, top_k: 40, top_p: 0.9, }); console.log(response.choices[0].message.content); })();

Alternatively, you can use the *.wasm files from CDN:

import WasmFromCDN from '@wllama/wllama/esm/wasm-from-cdn.js';
const wllama = new Wllama(WasmFromCDN);
// NOTE: this is not recommended, only use when you can't embed wasm files in your project

Split model

Splitting the model is not needed anymore: wllama loads a single GGUF file, even bigger than 4GB, so we recommend using the file as-is. Models that are already split with llama-gguf-split (files ending with -00001-of-00003.gguf, -00002-of-00003.gguf, and so on) are still supported.

To load a split model, pass to loadModelFromUrl or loadModelFromHF the URL of the first file and it will automatically load all the chunks:

const wllama = new Wllama(CONFIG_PATHS, {
  parallelDownloads: 5, // optional: maximum files to download in parallel (default: 3)
});
await wllama.loadModelFromHF({
  repo: 'ngxson/tinyllama_split_test',
  file: 'stories15M-q8_0-00001-of-00003.gguf',
});

Custom logger (suppress debug messages)

When initializing Wllama, you can pass a custom logger to Wllama.

Example 1: Suppress debug message

import { Wllama, LoggerWithoutDebug } from '@wllama/wllama';

const wllama = new Wllama(pathConfig, { // LoggerWithoutDebug is predefined inside wllama logger: LoggerWithoutDebug, });

Example 2: Add emoji prefix to log messages

const wllama = new Wllama(pathConfig, {
  logger: {
    debug: (...args) => console.debug('🔧', ...args),
    log: (...args) => console.log('ℹ️', ...args),
    warn: (...args) => console.warn('⚠️', ...args),
    error: (...args) => console.error('☠️', ...args),
  },
});

How to compile the binary yourself

This repository already come with pre-built binary from llama.cpp source code. However, in some cases you may want to compile it yourself:

  • You don't trust the pre-built one.
  • You want to try out latest - bleeding-edge changes from upstream llama.cpp source code.
You can use the commands below to compile it yourself:

# /!\ IMPORTANT: Require having docker compose installed

Clone the repository with submodule

git clone --recurse-submodules https://github.com/ngxson/wllama.git cd wllama

Optionally, you can run this command to update llama.cpp to latest upstream version (bleeding-edge, use with your own risk!)

git submodule update --remote --merge

Install the required modules

npm i

Firstly, build llama.cpp into wasm

npm run build:wasm

Then, build ES module

npm run build

TODO

  • Add support for LoRA adapter
  • Support multi-sequences: knowing the resource limitation when using WASM, I don't think having multi-sequences is a good idea

Acknowledgments

Wllama was created and is maintained by Xuan-Son Nguyen. The WebGPU backend for llama.cpp is maintained by Reese Levine. We thank all other contributors to both wllama and llama.cpp, whose work made this project possible.

Chat with me