VibeVoice-ASR (7B), AWQ W4A16 (asymmetric, groups of 128), with its
DFlash 2 drafter bundled in drafter/:
one download, and vibevoice.c
decodes with speculative decoding -- the drafter proposes 8 tokens in one
pass, the model checks them in one pass and keeps the ones it agrees with.
The check is exact: every checked row is computed with the arithmetic of
the model's own decode step, so the transcript is byte-for-byte the one
without the drafter.
(With a drafter vibevoice.c attends with flashinfer, which checks rows
together; the transcript is then --attn flashinfer's.)
Use
Needs vibevoice.c with DFlash 2 support: branch dflash2
(PR #48), in the next
release. A model directory's drafter/ is used without asking:
vv_cli --model ./VibeVoice-ASR-AWQ-W4A16-ASYM-DFlash2 --audio talk.wav # with the drafter
vv_cli --model ./VibeVoice-ASR-AWQ-W4A16-ASYM-DFlash2 --audio talk.wav --draft none # plain decoding
vv_cli serve --model ./VibeVoice-ASR-AWQ-W4A16-ASYM-DFlash2 --slots 4
Results
vibevoice.c b72be15 (branch dflash2), RTX 3090, greedy decoding, decode tokens per second:
|
plain |
drafted |
speedup |
tokens per block |
same transcript |
| 20 held-out clips (AMI, VoxConverse, earnings calls, LibriSpeech, FLEURS) |
148 tok/s |
565 tok/s |
3.82x |
5.32 |
20/20 |
| 2-minute file, 2 speakers |
151 tok/s |
523 tok/s |
3.47x |
4.64 |
yes |
| 32-minute file |
123 tok/s |
296 tok/s |
2.41x |
3.63 |
yes |
Plain = the same binary without --draft, both with flashinfer attention, which --draft picks by default; --draft-check exact with the check width it measures. Against the default without a drafter (fa2): 3.65x on the 2-minute file, 2.78x on the 32-minute file.
Inside
- The model: the files of Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM
at revision
22b44c85, unchanged (7.02 GB) -- its card has the quantization, the calibration and the WER.
drafter/: Ar4ikov/VibeVoice-ASR-DFlash2-Drafter-AWQ-W4A16-ASYM at revision
cff4a1db (0.55 GB): 5 Qwen3-style layers reading the model's
layers 1/7/13/19/25, a candidate selector, a 32768-id draft vocabulary;
its projections stored as INT4 (compressed-tensors pack-quantized). Its card has the architecture and the training.
License
MIT, like VibeVoice.
