ep-hindi-asr

Hugging Face | Base model | Vaani Benchmark
License: Apache 2.0 | Authors: Epoch Discovery

ep-hindi-asr is a Hindi speech recognition model built on the audio encoder of EmbeddingGemma 2 . A CTC head reads a learned mix of the encoder's layers and predicts Gemma tokenizer units. Only the top two encoder layers, the layer mix and the head are trained; the rest of the encoder is used as released by Google DeepMind.

The model was trained in two stages on a single NVIDIA A10: 300 hours of IndicVoices Hindi, then continued fine-tuning on IndicVoices, Vaani, Kathbath and Gram Vaani with noise augmentation. It decodes in one pass with no autoregressive decoder, and transcribes about 800 times faster than real time with batched input.

Model Overview

Benchmark Results

All results use the released checkpoint (stage 2, step 28,000) in bf16. The checkpoint was selected on dev sets only. The KenLM decoder settings are the stage-1 settings, tuned on our IndicVoices dev split.

Except for Vaani, WER is single-reference after normalize.py (strict) on reference and hypothesis.

Stage 1 (previous release) and stage 2

Inference

1. Get access and download

The repository and the base model are gated. Accept the terms on both pages first: epoch-discovery/ep-hindi-asr and google/embeddinggemma-2.

pip install -U "huggingface_hub[cli]"
hf auth login
hf download epoch-discovery/ep-hindi-asr --local-dir ep-hindi-asr
cd ep-hindi-asr
pip install -r requirements.txt

transcribe.py loads the weights from model.safetensors and config.json. From google/embeddinggemma-2 it downloads only the configuration files and the tokenizer.

2. Command line

# greedy decoding (fastest)
python transcribe.py --ckpt model.safetensors audio.wav

# several files in one call (the model loads once)
python transcribe.py --ckpt model.safetensors a.wav b.flac c.mp3

# beam search with the 4-gram KenLM (lower WER, slower; the search runs on the CPU)
python transcribe.py --ckpt model.safetensors --decode lm \
    --lm hi_train_noheldout_4gram.bin --unigrams unigrams_noheldout.txt audio.wav

# CPU only
python transcribe.py --ckpt model.safetensors --device cpu audio.wav

Each output line is <file>\t<text>\t(RTF <real-time factor>). Add --normalize to apply the scoring normaliser (normalize.py, strict mode). The KenLM settings (alpha 0.3, beta 1.0, beam 64) come from config.json; --alpha, --beta and --beam override them.

3. Python

import torch
from transcribe import load, read_audio, log_probs, greedy

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model, units, ck = load("model.safetensors", device)

wav = read_audio("audio.wav")          # mono float32, resampled to 16 kHz
lp = log_probs(model, wav, device)     # CTC log-probabilities, 25 frames per second
print(greedy(lp, units))

4. Long recordings

The model was trained on segments of 30 s or less, and transcribe.py decodes each file in one pass. Split longer audio into segments of up to 30 s before transcription. Split at pauses (for example with a voice activity detector), not at fixed times, so that words are not cut.

5. Speed and hardware

Peak GPU memory is 5.6 GB with batched input. KenLM beam search runs on the CPU and adds decode time.

6. Notes

• Run in bf16 (GPU) or fp32 (CPU). fp16 is not supported.
• Use SDPA attention (the default in eg2_model.py). Eager attention in transformers 5.19.0 gives wrong outputs.
• Output is Devanagari without punctuation or digits. English words are written in Devanagari.

7. Reproduce the Vaani score

python scripts/vaani_eval.py --pkg . --data <Vaani-Benchmark-V1.0 snapshot> \
    --toolkit <Vaani-Benchmark-Evaluation-Toolkit> --out out/
python run_eval.py --model epoch-discovery/ep-hindi-asr --mode score --predictions out/predictions_beam_lm.json

The second command is from the official toolkit.

Training

Training scripts for stage 1 are in scripts/.

Usage and Limitations

Intended Usage

Transcription of Hindi speech to Devanagari text: voice notes, interviews, call and meeting recordings, and batch transcription of audio archives where throughput matters.

Limitations

• Language and script: Hindi only. English and other Latin-script words are written in Devanagari or missed; there is no romanised or code-mixed output.
• Numbers: the training transcripts spell numbers as words, so the model does not output digits. Test sets that write numbers as digits (for example FLEURS) count these as errors.
• Formatting: no punctuation or capitalisation.
• Accents and audio: accuracy varies by region; on Vaani, greedy WER per state ranges from 7.2 to 22.6. Rural telephone speech (Gram Vaani) remains the hardest domain.
• Streaming: not supported; each window uses its full context.
• Language model: the KenLM is built from IndicVoices transcripts only and has a limited vocabulary for formal or technical text.

License and Acknowledgements

The fine-tuned weights and code are released under Apache 2.0. Use of the base model is subject to the terms of google/embeddinggemma-2. Training data: IndicVoices and Kathbath (AI4Bharat), Vaani (ARTPARK, IISc) and Gram Vaani (OpenSLR 118), each under its own terms. Noise augmentation uses MUSAN and simulated room impulse responses (OpenSLR 28). The KenLM text comes from IndicVoices.

The recipe follows QUAN-Transcribe and the idea of transcribing with the EmbeddingGemma 2 audio encoder from Embedding Gemma Transcribe.