wllama - Wasm binding for llama.cpp
WebAssembly binding for llama.cpp
👉 See the blog post introducing WebGPU support in llama.cpp and wllama
For changelog, please visit releases page
[!IMPORTANT]
> 🔥🔥 V3 is out, with WebGPU, multimodal and tool calling support. Read the V3 release guide
> For compatibility issues, please refer to @wllama/wllama-compat
Features
- 🔌 OpenAI-compatible API (fully-typed built-in)
- 🚀 WebGPU support
- 🔥 Multimodal support (image and audio file input)
- Speech support: text to speech, transcription and realtime audio
- 🔥 Tool calling support
- Can run inference directly on browser (using WebAssembly SIMD), no backend or GPU is needed!
- No runtime dependency (see package.json)
- Load a GGUF file as-is, even bigger than 4GB, no need to split it (models already split with
llama-gguf-splitwork too) - Auto switch between single-thread and multi-thread build based on browser support
- Inference is done inside a worker, does not block UI render
- Pre-built npm package @wllama/wllama
- To enable multi-thread, you must add
Cross-Origin-Embedder-PolicyandCross-Origin-Opener-Policyheaders. See this discussion for more details. - One wasm instance has 4GB of memory. If the part of the model that is not on WebGPU is bigger, it is split across several wasm instances in workers (
rpc_workers, automatic by default), which needs multi-thread, and JSPI or the compat build (on Firefox, seesetCompat).
Code demo and documentation
Demo:
- Basic usages with completions and embeddings: https://github.ngxson.com/wllama/examples/basic/ (source code)
- Embedding and cosine distance: https://github.ngxson.com/wllama/examples/embeddings/ (source code)
- Multimodal (vision) completion: https://github.ngxson.com/wllama/examples/multimodal/ (source code)
- Tool calling: https://github.ngxson.com/wllama/examples/tools/ (source code)
- Decision models: https://github.ngxson.com/wllama/examples/decision/ (source code)
- Omni (voice chat, text to speech, realtime speech to text): https://github.ngxson.com/wllama/examples/omni/ (source code)
How to use
Use Wllama inside React Typescript project
Install it:
npm i @wllama/wllama
Wasm binaries do not come pre-built with this repo. You need docker installed on your machine to build them:
# recommend to clone as git submodule
git submodule add https://github.com/ngxson/wllama.git wllama
git submodule update --init --recursive
run the build
cd wllama
npm ci
npm run build:wasm && npm run build
Then, import the module:
import { Wllama } from '@wllama/wllama';
let wllamaInstance = new Wllama(WLLAMA_CONFIG_PATHS, ...);
// (the rest is the same with earlier example)
For complete code example, see examples/main/src/utils/wllama.context.tsx
NOTE: this example only covers completions usage. For embeddings, please see examples/embeddings/index.html. For audio, see Speech and realtime audio
Speech and realtime audio
These APIs need a model with audio support and its mmproj file, for example LFM3-Omni. Use getLoadedContextInfo() to see what the model can do (has_audio_input, has_audio_output, has_audio_stream, has_live_asr, audio_voices).
// text to speech, returns a WAV file. Use stream: true to get chunks as they are generated
const { audio, sample_rate } = await wllama.createSpeech({
input: 'Hello world',
voice: 'female',
});
// transcribe an audio file
const { text } = await wllama.createTranscription({
file: await audioFile.arrayBuffer(),
language: 'en',
});
A realtime session uses the same events as the OpenAI realtime API and the /v1/realtime route of llama-server. It holds one slot until close() is called, and inference only runs while events() is being read.
const session = await wllama.createRealtimeSession();
await session.send({ type: 'session.update', session: { type: 'transcription' } });
// mono float samples at getLoadedContextInfo().audio_input_rate
await session.appendAudio(samples);
await session.send({ type: 'input_audio_buffer.commit' });
for await (const event of session.events()) {
if (event.type === 'conversation.item.input_audio_transcription.delta') {
console.log(event.delta);
} else if (event.type === 'conversation.item.input_audio_transcription.completed') {
break;
}
}
await session.close();
There are 3 kinds of session:
- Default: a conversation. Send
response.createafter the audio, the answer comes asresponse.output_text.deltaandresponse.output_audio.deltaevents. UsedecodeRealtimeAudio()to read the audio session.updatewithtype: 'transcription': each committed audio buffer is transcribed on its owncreateRealtimeSession({ intent: 'transcription' }): live ASR, the text comes out while the audio goes in. Needshas_live_asr, LFM3-Omni does not have it
WebGPU support
WebGPU support is introduced via PR #215.
Upon updating to V3.1, WebGPU will be enabled automatically. By default, all layers will be offloaded to GPU. If the model is too big to fit into VRAM, you can manually adjust the number of layers via the n_gpu_layers parameter of LoadModelParams. Example:
// (optionally) will allow running WebGPU on Firefox via compat mode; performance will be significantly degraded
wllama.setCompat('default', 'firefox_safari');
await wllama.loadModel(files, {
n_gpu_layers: 4, // meaning 4 layers are offloaded to GPU; set to 0 to disable GPU inference
});
Prepare your model
- It is recommended to use the GGUF file as-is, there is no need to split it anymore. A file of 4GB or more is split in memory by wllama, and a model that does not fit in one wasm instance is spread across workers (see Limitations above). See the "Split model" section below if your model is already split.
- It is recommended to use quantized Q4, Q5 or Q6 for balance among performance, file size and quality. Using IQ (with imatrix) is not recommended, may result in slow inference and low quality.
Simple usage with ES6 module
For complete code, see examples/basic/index.html
import { Wllama } from './esm/index.js';
(async () => {
const CONFIG_PATHS = {
default: './esm/wasm/wllama.wasm',
};
// Automatically switch between single-thread and multi-thread version based on browser support
// If you want to enforce single-thread, add { "n_threads": 1 } to LoadModelConfig
const wllama = new Wllama(CONFIG_PATHS);
// Define a function for tracking the model download progress
const progressCallback = ({ loaded, total }) => {
// Calculate the progress as a percentage
const progressPercentage = Math.round((loaded / total) * 100);
// Log the progress in a user-friendly format
console.log(Downloading... ${progressPercentage}%);
};
// Load GGUF from Hugging Face hub
// (alternatively, you can use loadModelFromUrl if the model is not from HF hub)
await wllama.loadModelFromHF(
{ repo: 'ggml-org/models', file: 'tinyllamas/stories260K.gguf' },
{ progressCallback }
);
const response = await wllama.createChatCompletion({
messages: [{ role: 'user', content: elemInput.value }],
max_tokens: 50,
temperature: 0.5,
top_k: 40,
top_p: 0.9,
});
console.log(response.choices[0].message.content);
})();
Alternatively, you can use the *.wasm files from CDN:
import WasmFromCDN from '@wllama/wllama/esm/wasm-from-cdn.js';
const wllama = new Wllama(WasmFromCDN);
// NOTE: this is not recommended, only use when you can't embed wasm files in your project
Split model
Splitting the model is not needed anymore: wllama loads a single GGUF file, even bigger than 4GB, so we recommend using the file as-is. Models that are already split with llama-gguf-split (files ending with -00001-of-00003.gguf, -00002-of-00003.gguf, and so on) are still supported.
To load a split model, pass to loadModelFromUrl or loadModelFromHF the URL of the first file and it will automatically load all the chunks:
const wllama = new Wllama(CONFIG_PATHS, {
parallelDownloads: 5, // optional: maximum files to download in parallel (default: 3)
});
await wllama.loadModelFromHF({
repo: 'ngxson/tinyllama_split_test',
file: 'stories15M-q8_0-00001-of-00003.gguf',
});
Custom logger (suppress debug messages)
When initializing Wllama, you can pass a custom logger to Wllama.
Example 1: Suppress debug message
import { Wllama, LoggerWithoutDebug } from '@wllama/wllama';
const wllama = new Wllama(pathConfig, {
// LoggerWithoutDebug is predefined inside wllama
logger: LoggerWithoutDebug,
});
Example 2: Add emoji prefix to log messages
const wllama = new Wllama(pathConfig, {
logger: {
debug: (...args) => console.debug('🔧', ...args),
log: (...args) => console.log('ℹ️', ...args),
warn: (...args) => console.warn('⚠️', ...args),
error: (...args) => console.error('☠️', ...args),
},
});
How to compile the binary yourself
This repository already come with pre-built binary from llama.cpp source code. However, in some cases you may want to compile it yourself:
- You don't trust the pre-built one.
- You want to try out latest - bleeding-edge changes from upstream llama.cpp source code.
# /!\ IMPORTANT: Require having docker compose installed
Clone the repository with submodule
git clone --recurse-submodules https://github.com/ngxson/wllama.git
cd wllama
Optionally, you can run this command to update llama.cpp to latest upstream version (bleeding-edge, use with your own risk!)
git submodule update --remote --merge
Install the required modules
npm i
Firstly, build llama.cpp into wasm
npm run build:wasm
Then, build ES module
npm run build
TODO
- Add support for LoRA adapter
- Support multi-sequences: knowing the resource limitation when using WASM, I don't think having multi-sequences is a good idea
Acknowledgments
Wllama was created and is maintained by Xuan-Son Nguyen. The WebGPU backend for llama.cpp is maintained by Reese Levine. We thank all other contributors to both wllama and llama.cpp, whose work made this project possible.