whisper-heb-v7-multi (Hebrew / English / Arabic / Russian / Yiddish with built-in language ID)
plus speech-free negatives: the model has learned to answer with nothing on line noise, hold
music, tones and silence instead of inventing words. Same architecture (Whisper large-v3-turbo,
809M), same languages, same usage; drop-in replacement for v7.
Why v8
On telephony audio Whisper fine-tunes hallucinate short phrases when there is no speech. v7
produced a transcript for 98% of real line-noise clips (27% of them "תודה רבה") and 46% of
synthetic noise. v8 was warm-started from v7 and trained 5,000 steps on the v7 corpus plus
83,920 speech-free rows (real Voicenter line noise ×4, synthetic noise/music/tones, real speech
with trailing noise) whose target is Whisper's native <|nospeech|> token, with the 458
"phantom thanks" training rows and ~2.6k mis-cut labels removed.
| speech-free set |
v7 any output |
v8 any output |
v7 "תודה" |
v8 "תודה" |
| real Voicenter line noise (59 clips, human-labelled, held out) |
98.3% |
6.8% |
27.1% |
0.0% |
| synthetic noise, 12 kinds (1,500 clips) |
46.2% |
1.3% |
8.5% |
0.0% |
Speech is unaffected: on 1,517 real telephony clips ≤4 s (56 of them a bare "תודה רבה") v8
scores WER 0.033 vs v7's 0.034 and silences 0.4% of them.
Usage
from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="VoicenterTeam/whisper-heb-v8-multi",
torch_dtype="bfloat16", device="cuda")
asr("call.wav") # language auto-detected; "" on noise
asr("call.wav", generate_kwargs={"language": "he"}) # force one of he/en/ar/ru/yi
The stock generation_config is kept (begin_suppress_tokens=[220, 50257], language=None).
On speech-free input the model emits <|nospeech|> followed by <|endoftext|>, which decodes
to an empty string; in faster-whisper / CTranslate2 this surfaces as a high no_speech_prob,
so the usual no_speech_threshold drops the segment. Converts with ct2-transformers-converter
unchanged.
Results (clean audio, auto language detection, WER)
Normalised text (nikud/tashkeel/punctuation stripped, case folded, ё→е, Arabic alef/ta
marbuta/ya folded), greedy, bf16. Full tables incl. 8 kHz μ-law condition, forced mode, CER,
LID confusion: results/whisper-heb-v8-multi/SUMMARY.md in the training repo.
| lang |
sets |
v8 |
v7 |
openai large-v3-turbo |
LID acc |
| he |
ILSpeech, 10k telephony |
0.128 |
0.128 |
0.240 |
1.00 |
| en |
FLEURS+CV+AMI, AppTek call-center |
0.141 |
0.138 |
0.133 |
0.98 |
| ar |
FLEURS, CV17, Casablanca jo/ps |
0.368 |
0.376 |
0.292 |
0.96 |
| ru |
FLEURS+CV+Golos, ru.test |
0.111 |
0.114 |
0.122 |
0.99 |
| yi |
Omnilingual ydd (read) |
0.336 |
0.347 |
1.196 |
1.00 |
| macro |
0.217 |
0.221 |
0.397 |
0.99 |
Selected sets:
| set |
refs |
v8 |
v7 |
| he telephony 10k (1000 sample) |
soniox |
0.155 |
0.155 |
| he telephony, soniox+witness agree (5,915) |
soniox |
0.078 |
0.079 |
| he short clips ≤4 s (1,517) |
2-provider |
0.033 |
0.034 |
| he ILSpeech speaker 1 |
human |
0.124 |
0.124 |
| he whatsapp (long-form) |
human |
0.070 |
0.071 |
| he suite macro (d1, whatsapp, fleurs, CV, kan, saspeech; forced he) |
human |
0.102 |
0.102 |
| en AppTek call-center (3,000) |
human |
0.182 |
0.181 |
| ru real phone calls (Open STT, eval-only), auto / forced |
human |
0.412 / 0.355 |
0.388 / 0.355 |
| yi prompt-disjoint held-out (761) |
human |
0.345 |
0.356 |
Regressions to know about: Russian real phone calls in auto-LID mode (0.412 vs 0.388; identical
when the language is forced — a few more calls are misdetected), and the Hebrew kan set in
forced mode (0.088 vs 0.073; equal in auto mode). Everything else is equal or better.
Known limitations
- Arabic trails
openai/whisper-large-v3-turbo (0.368 vs 0.292): Arabic was 5% of the training
mix with a Levantine-biased teacher. Russian telephony also trails on real calls.
- LID drops on very short clips (<2 s: ar 0.58, yi 0.83, ru 0.89) and ~5% of Hebrew read
speech (FLEURS, CV) is detected as Arabic — force
language="he" when the language is known.
- About 7% of real line-noise clips still get a short output (mostly English fillers). A VAD in
front of the model remains cheap insurance.
- Yiddish numbers are read speech; no conversational Yiddish test set exists.
Training
- Warm start:
VoicenterTeam/whisper-heb-v7-multi (step 14500). 5,000 steps, effective batch 256,
lr 1.5e-6 (encoder 0.3×), cosine, 200 warmup, bf16, label smoothing 0.1, SpecAugment, speed
perturbation, telephone band-pass. Best checkpoint (step 3500) selected on macro dev WER over
the five languages with auto language detection.
- Data: v7 corpus (3.76M clips; he 53%, en 19%, ru 19%, ar 5%, yi 3%, commercially-licensed
sources only) + 83,920 speech-free rows with target
<|nospeech|><|endoftext|> − 458 phantom
"תודה רבה" rows − ~2.6k decoder-loop / mis-cut / subtitle-credit labels.
- Knowledge distillation as in v7 (
0.2·CE + 0.8·T²·KL, T=2; teachers he: whisper-heb-v5,
en: TheStageAI/thewhisper-large-v3-turbo, ru: openai/whisper-large-v3, ar:
VoicenterTeam/whisper-levantine-hf, yi: ivrit-ai/yi-whisper-large-v3). Negatives are CE-only:
every teacher hallucinates on noise, so distilling them there would teach the phantom back.
Code: the whisper-finetune repo (scripts/build_multilingual_manifest_v8.py,
plans/v8-data/NOTES.md, src/kd.py, scripts/bench_v7.sh).
Data attribution
As for v7: ivrit.ai (crowd-recital, crowd-transcribe, audio-v2, crowd-recital-yi,
crowd-whatsapp-yi; ivrit.ai license v2 — not for voice cloning), VoxKnesset, Voicenter telephony;
Common Voice (CC0), People's Speech, Granary/YODAS, VoxPopuli, AMI, ICSI, LibriSpeech, EdAcc,
NOTSOFAR-1, DiPCo, SLUE, Vystadial, MLS, speechocean762, MInDS-14; Golos (SberDevices license),
Russian LibriSpeech, SOVA RuDevices, FLEURS, ToneWebinars, Dialogs; Omnilingual ASR corpus
(CC-BY-4.0), Lahgtna Levantine TTS, SADA, ClArTTS, ArVoice, SVQ; REYD (CC-BY-4.0).
Negatives: Voicenter line noise, MUSAN (CC-BY-4.0) noise and music. Teachers: TheStage AI
(CC-BY-4.0), OpenAI Whisper (MIT), ivrit.ai (Apache-2.0).
