litert-lm-web-runtime

LiteRT-LM Web Runtime (@litert-lm/core 0.18.1)

WebAssembly + WebGPU runtime binaries and the matching @litert-lm/core TypeScript SDK for running Gemma 4 (E2B, E4B, 12B, 26B-A4B, 31B) in the browser with:

• Text decode with shared-KV neural Multi-Token Prediction (MTP) speculative decoding (GPU_ARTISAN backend, gamma 1-3, greedy token-for-token equivalence with standard decode).
• Multimodal vision (SigLIP encoder on WebGPU, 70 / 140 / 280 / 560 / 1120 token budgets, zero-copy RGBA8 ImageData preprocessing, per-budget executor memory release).
• Multimodal audio (Conformer encoder, streaming markers, zero-copy audio ingestion).
• Streamed model loading for 2-20 GB .litertlm bundles under wasm32 (MAXIMUM_MEMORY=4GB), with section filtering and inter-section padding discard.
• Bounded-in-flight WebGPU prefill command buffers (RunPendingCommands with $\le 2$ command buffers in flight and group-size-scaled command batching min(128, (16 * 1024) / group_size)) so long text and multimodal prefills yield cleanly to the browser compositor and respond to mid-prefill cancellation within ~100 ms.
• Decode telemetry on the console log for measurement tooling: [MTP_TELEMETRY] cycles=<N> accepted=<M> cbs/cycle=.. gather_cycles=.. ple_fallbacks=.. step=<T> win_cycles=<k> win_tok_per_cycle=<x> win_cycle_ms=<y> win_draft_ms=.. win_score_ms=.. win_readback_ms=.. (32-cycle window) and [DECODE_TELEMETRY] step=<T> win_tokens=32 win_ms_per_token=<z> for standard decode.

Repository layout

Exactly one runtime set is kept on main. Every release is also addressable by commit SHA (.../resolve/<commit>/wasm/), and older releases remain available through their commits.

Variant selection: internal = JSPI + threads + SIMD (Chrome 137+), compat_internal = JSPI + threads without SIMD, asyncify_internal / compat_asyncify_internal = Asyncify fallbacks for browsers without JSPI.

Usage

import {loadLiteRtLm, Backend} from '@litert-lm/core';

const WASM_BASE =
  'https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wasm/';
const litertlm = await loadLiteRtLm(WASM_BASE);
const engine = await litertlm.createEngine({
  modelPath: 'https://huggingface.co/jfan/gemma-4-e4b-web-mtp-vision-litert-lm/resolve/main/gemma-4-E4B-it-web-mtp-vision-audio.litertlm',
  backend: Backend.GPU_ARTISAN,
  speculativeDraftTokens: 3,   // MTP gamma; omit for standard decode
});

Pin a commit SHA instead of main in production so the binaries and the SDK stay in lock-step.

Model bundles

• jfan/gemma-4-e2b-web-mtp-vision-litert-lm, jfan/gemma-4-e4b-web-mtp-vision-litert-lm (text / +vision / +audio / +vision+audio, with the eevee MTP drafter and centroid head)
• jfan/gemma-4-12b-web-mtp-vision-litert-lm, jfan/gemma-4-26b-web-mtp-vision-litert-lm, jfan/gemma-4-31b-web-mtp-vision-litert-lm

Release notes

0.18.1 (2026-10-15, this commit)

• Bounded in-flight WebGPU prefill command buffers (RunPendingCommands, $\le 2$ command buffers in flight, command_batch_size = min(128, (16 * 1024) / group_size)) on Emscripten to eliminate browser compositor frame starvation and support mid-prefill cancellation.
• Dual Engine.createLegacyEngine / Engine.createStreaming Embind and TypeScript SDK compatibility across p4headCL 999856454.
• Dynamic per-budget vision executor memory release (280 / 560 / 1120 tokens) and pre-reserved sequence lengths (qwtzvnkm, putwztvn, mszqroqq).
• Repository cleaned to a single runtime set (wasm/) and a single package tarball (packages/), removing temporary wasm_dev_jank_v* sweep directories.

Earlier 0.18.1 builds (2026-10-09 .. 2026-10-14)

• Windowed [MTP_TELEMETRY] / [DECODE_TELEMETRY] decode telemetry, 26B/31B standardized vision adapter support, trailing <end_of_image> causal mask fix.
• Minijinja Chainable undefined-lookup fix and empty-messages guard in the Gemma 4 chat templates.
• Non-blocking thinking-channel KV refill, SWA ring-eviction rewind fallback, MTP token accounting, native GPU profiler.
• Host-parity shader pipeline drain / warm-up at initialization, fused residual norms, in-place ringed uint8 KV reads, batched prefill ConvGeneric with local-memory dequantization, 4x4 GEMM spatial tiling.

0.18.0 / 0.17.x

• Shared-KV MTP drafter on WebGPU / OpenCL / Metal, centroid decoding, zero-readback draft chaining; multimodal vision (1120-token budget) and audio; streamed loading of >4 GB bundles.