VibeVoice-ASR (7B), the original BF16 weights as Microsoft published them, with its
DFlash 2 drafter bundled in drafter/:
one download, and vibevoice.c
decodes with speculative decoding -- the drafter proposes 8 tokens in one
pass, the model checks them in one pass and keeps the ones it agrees with.
The check is exact: every checked row is computed with the arithmetic of
the model's own decode step, so the transcript is byte-for-byte the one
without the drafter.
(With a drafter vibevoice.c attends with flashinfer, which checks rows
together; the transcript is then --attn flashinfer's.)
Use
Needs vibevoice.c with DFlash 2 support: branch dflash2
(PR #48), in the next
release. A model directory's drafter/ is used without asking:
vv_cli --model ./VibeVoice-ASR-DFlash2 --audio talk.wav # with the drafter
vv_cli --model ./VibeVoice-ASR-DFlash2 --audio talk.wav --draft none # plain decoding
vv_cli serve --model ./VibeVoice-ASR-DFlash2 --slots 4
vibevoice.c runs BF16 weights as dense FP16 (about 19 GB of VRAM for a 7B,
drafter included); --quant int4 quantizes them at load, and the drafter
works with that too.
Results
vibevoice.c b72be15 (branch dflash2), RTX 3090, greedy decoding, decode tokens per second:
|
plain |
drafted |
speedup |
tokens per block |
same transcript |
| 2-minute file |
57 tok/s |
156 tok/s |
2.73x |
3.52 |
yes |
| 32-minute file |
52 tok/s |
124 tok/s |
2.36x |
3.07 |
yes |
BF16 weights run as dense FP16. Plain = the same model with --draft none --attn flashinfer (with a drafter vibevoice.c attends with flashinfer); against the default without a drafter (fa2): 2.79x on the 2-minute file, 2.51x on the 32-minute file. --draft-check exact, the check width measured as it runs.
Inside
- The model: the files of microsoft/VibeVoice-ASR
at revision
d0c9efdb, unchanged (17.37 GB) -- its card has the model, the evaluation and the license.
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, special_tokens_map.json, preprocessor_config.json: from Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM at revision 22b44c85 -- the upstream repository leaves them out (they are Qwen2.5's tokenizer and the audio preprocessor's settings), and vibevoice.c needs them next to the weights.drafter/: Ar4ikov/VibeVoice-ASR-DFlash2-Drafter at revision
795e2a5d (1.66 GB): 5 Qwen3-style layers reading the model's
layers 1/7/13/19/25, a candidate selector, a 32768-id draft vocabulary;
BF16, held as INT4 by default at load (--draft-quant f16 keeps FP16). Its card has the architecture and the training.
License
MIT, like VibeVoice.
