sea-asr-id

sea-asr-id — fast Indonesian speech recognition on EmbeddingGemma 2 (CTC)

Pixel ML builds video intelligence for Southeast Asian media. These are the speech models we run for Southeast Asian media: captions, search, clips. Open weights, open data, runs on your GPU — or on a CPU.

Google's EmbeddingGemma 2 audio encoder with a CTC head, trained by Pixel ML on openly licensed Indonesian speech (YODAS, FLEURS, Common Voice). An optional n-gram language model (folder lm/, 0.27 GB) built from Indonesian Wikipedia comes with the weights.

Speed (one H100 80GB, side by side with Whisper on the same GPU type and container size; sea_asr v0.2.1): greedy decoding is about 18x faster than batched Whisper large-v3 and 9x faster than large-v3-turbo on a 79-minute recording (long-form), and 30x / 20x faster on FLEURS test (short clips). With the n-gram LM or rescoring, the beam search on CPU sets the speed: on short clips both modes are still faster than Whisper, but on the long-form recording + n-gram LM is slower than turbo and + rescoring is slower than large-v3 (table in "Max accuracy" below). It also runs on a CPU. One pass, no autoregressive decode, streaming in 0.48 s chunks. Word timestamps and SRT/VTT output included.

Trained on CC-BY YouTube audio (via YODAS) and read speech. No purchased data.

Results

Same H100 for every open model, one normaliser for all (quanphid/text.py: lower case, no punctuation, numbers as words), WER % (lower is better). Avg = mean of FLEURS and CV27. Speed = audio seconds / wall seconds on FLEURS id test (short clips), one H100 80GB, batched, model load and warm-up excluded; ours sea_asr v0.2.1, Whisper on the same GPU type and container size. Params = Hugging Face safetensors total (ours: counted from this repo).

Size class. Under 1B: sea-asr-id with its n-gram LM ties Whisper large-v3-turbo on read speech, lower WER on conversational Indonesian (GigaSpeech 2) than every other open model we tested. Paired 95 % CI of the WER difference (sea-asr-id + LM − rival): vs Qwen3-ASR-0.6B (2.6× larger) FLEURS −1.8 [−2.3, −1.3], CV27 −1.4 [−1.8, −1.0], GigaSpeech 2 −0.3 [−0.4, −0.1]; vs Whisper large-v3-turbo FLEURS +0.4 [−0.1, +0.8], CV27 −0.2 [−0.7, +0.2] (tie), GigaSpeech 2 −0.6 (CI below 0); vs Whisper large-v3 FLEURS +0.9 [+0.4, +1.3], CV27 +0.7 [+0.3, +1.1]. Whisper large-v3 and Qwen3-ASR-1.7B are more accurate. Greedy, without the LM: ties Whisper medium on FLEURS (−0.1 [−0.7, +0.5]) and is better on CV27 (−2.1 [−2.7, −1.6]).

• GS2: test video 45 excluded for all systems; it overlaps our YODAS training audio.
• — = not measured in this container. Qwen3-ASR, Whisper medium / small and omniASR were not re-run here. In an earlier H100 run (different container, vLLM max batch 128), Qwen3-ASR-1.7B ran at 1,650× real time on FLEURS id test and Qwen3-ASR-0.6B at 810–1,597× across our test sets: faster than sea-asr-id + n-gram LM (389×), slower than greedy.
• Speed with the LM: beam 128 (the shipped default), one beam-search worker (the v0.2.1 default; 4 workers change it by less than 1 %). The beam search runs on CPU. Long-form speed: see "Max accuracy (opt-in)" below.
• LM settings (alpha 0.4, beta 1.5, beam 128) and the LM itself were chosen on FLEURS id validation only. The LM text has every sentence removed that equals or shares an 8-gram with a FLEURS, CV27 or GigaSpeech 2 test or dev reference; an LM built without that filter gives CV27 6.99 vs 7.00.
• CV27 test clips and sentences were held out of all our training data. omniASR was trained on FLEURS and Common Voice (Meta, arXiv 2511.09690): its FLEURS and CV27 rows are not held-out scores.

Max accuracy (opt-in): 1B rescoring LM

With a 1B rescoring LM, on par with Whisper large-v3 (1.55B) on FLEURS and Common Voice, and better on GigaSpeech 2.

A second pass rescores the n-gram beam search's 32 best hypotheses with Sailor2-1B (Sea AI Lab, Apache-2.0). Weights of the scores were chosen on FLEURS id validation only. System: 364M acoustic model + 0.27 GB n-gram LM + 1B rescoring LM. Paired 95 % CI of the WER difference (ours + rescoring − Whisper large-v3): FLEURS +0.28 [−0.20, +0.74], CV27 −0.03 [−0.41, +0.36] (both: CI includes 0, on par), GigaSpeech 2 (video 45 excluded) −0.62 [−0.79, −0.45] (better).

WER %, and speed in × real time on one H100 80GB (sea_asr v0.2.1; Whisper on the same GPU type and container size, 32 vCPU):

Speed: whole transcribe() wall time, load and warm-up excluded; FLEURS test = 684 clips, 2.34 h; CV27 test = 3,711 clips, 4.27 h; long-form = one 79 min 10 s Indonesian parliament plenary recording, whole file per call. The beam search runs on CPU (one worker, the v0.2.1 default) and is most of the cost with the LM. On long-form audio + n-gram LM is slower than Whisper large-v3-turbo and + rescoring is slower than Whisper large-v3. Rescoring needs about 2 GB of extra GPU memory (bf16).

Error tail: CTC cannot loop (no autoregressive decoder); utterances with WER > 100 %: sea-asr-id 0 % on FLEURS, 0.08 % on CV27; Whisper large-v3 0.22 % on CV27.

Same-GPU setup: Whisper via faster-whisper 1.2.1 fp16, beam 5, Silero VAD, batch 16 (clips, or the batched long-form pipeline); ours 600 s of padded audio per forward pass. CAUTION: Whisper here spends part of its wall time on per-clip CPU work (Silero VAD + log-mel), and batch 16 is conservative for an H100. A Whisper setup tuned for throughput would narrow the gap.

Gemini is a closed benchmark row only; its outputs were not used for training (Gemini API terms).

Use

pip install "sea-asr[lm] @ git+https://github.com/PixelML/[email protected]"        # drop [lm] for greedy only
pip install "sea-asr[rescore] @ git+https://github.com/PixelML/[email protected]"   # opt-in max accuracy

from sea_asr import SeaASR, to_srt

asr = SeaASR.from_pretrained("PixelML/sea-asr-id")                            # greedy; device="cuda" or "cpu"
asr = SeaASR.from_pretrained("PixelML/sea-asr-id", lm="PixelML/sea-asr-id")   # + n-gram LM from lm/
asr = SeaASR.from_pretrained("PixelML/sea-asr-id", lm="PixelML/sea-asr-id",
                             rescore=True)                                     # + Sailor2-1B rescoring (opt-in)
print(asr.transcribe("clip.wav"))

out = asr.transcribe("newscast.wav", return_words=True)    # {"text", "words": [{"word", "start", "end"}]}
open("newscast.srt", "w").write(to_srt(out["words"]))

1
00:00:00,880 --> 00:00:04,120
tim tim virtual memiliki standar
keunggulan yang sama dengan tim

2
00:00:04,320 --> 00:00:10,320
konvensional tetapi ada sedikit
perbedaan segera setelah

Cues: at most 2 lines × 32 characters, at most 6 s. Input: 16 kHz mono (other rates are resampled). Inputs longer than 30 s are cut at quiet points; timestamps stay on the original timeline. CLI: python -m sea_asr --model PixelML/sea-asr-id --lm PixelML/sea-asr-id clip.wav --srt.

Limits

• Lower case, no punctuation; caption-ready output needs a punctuation + casing restorer (next, on the EG2 text backbone).
• The n-gram LM is built from Wikipedia: it helps most on read, formal speech. On casual speech the gain is smaller.
• Machine labels from Qwen3-ASR-1.7B and Whisper large-v3: the model inherits their shared errors and does not beat them on read speech.
• Code-switching to English and regional languages (Javanese, Sundanese) is not evaluated. Speaker, accent and gender balance are not measured.
• Do not use for speaker identification or surveillance.

Training

YODAS id000 + id100, scored with an earlier model (CER ≤ 0.6), relabelled with Qwen3-ASR + Whisper consensus (keep where CER(qwen, whisper) ≤ 0.2, label = Qwen): 1,901 h of segments plus 386 h of 1–3 s clips, + Common Voice 27.0 train ×5 (41 h) + FLEURS train ×5 (45 h) = 2,373 h. Fine-tuned from the previous round on one NVIDIA DGX Spark (GB10); best checkpoint at step 2,000 on FLEURS validation. Labels: PixelML/sea-asr-id-data. Code and recipe: PixelML/sea-asr.

The round-1 data filter used a pilot model that saw CC BY-NC (WorldSpeech) audio. That model only scored data; no NC audio or labels entered training. WorldSpeech is used as a test set only.

The n-gram LM: KenLM 4-gram on Indonesian Wikipedia (dump 20231101, 10.3 M sentences, 156 M words after the test filter), pruned to 0.27 GB; build details in lm/manifest.json.

Roadmap: punctuation + casing head; Malay, Vietnamese, Thai.

Credits

• Base encoder: Google DeepMind, EmbeddingGemma 2 (914f7f89)
• Data: YODAS (ESPnet/CMU, 52c5a1b9); FLEURS (Google); CC-BY YouTube creators listed in ATTRIBUTION
• Data (pins added for this release): FLEURS train (google/fleurs, 70bb2e84, CC-BY-4.0); Common Voice 27.0 Indonesian train (Mozilla, CC0, via Mozilla Data Collective); the creator list is ATTRIBUTION.tsv
• Teachers: Qwen3-ASR-1.7B (Alibaba Qwen, 7278e1e7); Whisper large-v3 (OpenAI) via faster-whisper (SYSTRAN)
• Language model text: Indonesian Wikipedia, Wikipedia contributors / Wikimedia Foundation (wikimedia/wikipedia 20231101.id, b04c8d1c, CC-BY-SA-3.0 + GFDL)
• Language model tools: KenLM by Kenneth Heafield (kpu/kenlm 4cb443e6); pyctcdecode 0.5.0 (Kensho Technologies, kensho-technologies/pyctcdecode)
• Rescoring LM (opt-in): Sailor2-1B, Sea AI Lab (sail/Sailor2-1B 75c2856f, Apache-2.0)
• Test sets: GigaSpeechBench (SpeechColab); Common Voice 27.0 Indonesian test (Mozilla, CC0); WorldSpeech (disco-eth)
• Test sets (this release): GigaSpeech 2 Indonesian test (SpeechColab, non-commercial terms, test only); FLEURS Indonesian test (same revision as above)
• Recipe: Quan Nguyen (@qnguyen3), QUAN-Transcribe — https://github.com/qnguyen3/QUAN-Transcribe (4f45e02)

Licence

Model weights and code: Apache-2.0. Base encoder EmbeddingGemma 2 is Apache-2.0; deployments must adhere to the Gemma Prohibited Use Policy.

The n-gram LM in lm/ is derived from Wikipedia text and is licensed CC BY-SA 3.0 (see lm/LICENSE.md). If you ship or modify lm/, keep the attribution and share it under the same licence. Greedy decoding does not use lm/.