A DFlash 2 block drafter for VibeVoice-ASR (7B)
(Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM), for speculative decoding in
vibevoice.c. It proposes 8 tokens in
one pass; vibevoice.c checks them in one pass of the model and keeps the ones
the model agrees with, plus one of its own.
The check is exact: every checked row is computed with the arithmetic of
the model's own one-token decode step, so the transcript with this drafter is
byte-for-byte the transcript without it — words, timestamps, speakers. The
drafter only changes how many passes it takes.
One thing changes with --draft: vibevoice.c then attends with flashinfer
(--attn auto picks it), because flashinfer checks several rows in one pass
over the cache where the default fa2 walks it once per row. The transcript is
then byte-for-byte that of --attn flashinfer without a drafter — on this
model the default's words, with a timestamp 10 ms apart on a 32-minute file.
--attn fa2 --draft keeps the default's transcript bit for bit, and gains
less on long recordings.
Use
Needs vibevoice.c with DFlash 2 support: branch dflash2
(PR #48), in the next
release. The model and this drafter in one download:
Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM-DFlash2 (the INT4 drafter in
drafter/, used without --draft).
vv_cli --model ./VibeVoice-ASR-AWQ-W4A16-ASYM --audio talk.wav --draft ./VibeVoice-ASR-DFlash2-Drafter
vv_cli serve --model ./VibeVoice-ASR-AWQ-W4A16-ASYM --draft ./VibeVoice-ASR-DFlash2-Drafter --slots 4
| flag |
|
--draft <dir> |
this repository |
--draft-block <n> |
rows of the drafted block the model checks per pass (the drafter drafts 8; without the flag: 4 or 8, whichever keeps more tokens per ms) |
--draft-check exact\|fast |
exact (default): the transcript without a drafter on the same --attn, bit for bit; fast: flashinfer attention and prefill projections for the check, the greedy transcript within rounding |
--draft-quant int4\|f16 |
how the runtime holds a BF16 drafter (INT4 by default) |
It drafts for any VibeVoice-ASR (7B) checkpoint of the same model —
microsoft/VibeVoice-ASR, scerz/VibeVoice-ASR-4bit — since it reads
only the model's hidden states, embedding and LM head; it was trained on the
AWQ checkpoint's transcripts, so that is where it drafts best.
On NF4 (bitsandbytes) and INT8 weights only --draft-check fast pays: their
decode step has no multi-row twin, so an exact check would cost a step per
row, and vibevoice.c declines the drafter there with a pointer to fast.
Results
vibevoice.c b72be15 (branch dflash2), RTX 3090, VibeVoice-ASR-AWQ-W4A16-ASYM, greedy decoding, decode tokens per second:
|
plain |
drafted |
speedup |
tokens per block |
same transcript |
| 20 held-out clips (AMI, VoxConverse, earnings calls, LibriSpeech, FLEURS) |
148 tok/s |
566 tok/s |
3.84x |
5.33 |
20/20 |
| 2-minute file, 2 speakers |
151 tok/s |
547 tok/s |
3.63x |
4.87 |
yes |
| 32-minute file |
123 tok/s |
290 tok/s |
2.36x |
3.57 |
yes |
Plain = the same binary without --draft, both with flashinfer attention, which --draft picks by default; --draft-check exact with the check width it measures. Against the default without a drafter (fa2): 3.82x on the 2-minute file, 2.73x on the 32-minute file.
The drafter
- DFlash 2: 5 Qwen3-style layers (hidden 3584, 28/4 heads,
intermediate 9472), 831M parameters, block 8.
- KV injection: for every position the model has processed, the outputs of
its layers 1, 7, 13, 19 and 25 are concatenated, projected (
fc) and
normalized; every drafter layer turns them into keys and values. The drafter
reads the model's own view of the audio and of the transcript so far.
- Two-tap dynamic convolution around attention and MLP (per-row kernels
predicted from the row), and a candidate selector that re-ranks the
model head's top 16 per row with a pairwise predecessor/successor term
(rank 256).
- The embedding and the LM head are the model's, frozen. The drafter scores a
draft vocabulary of 32768 ids (
draft_vocab): the ids its training
transcripts use, plus every stop id.
- Accepted drafts on 43 held-out clips' traces: 5.273 tokens per
block of 8 (the selector's, with every earlier draft right).
Training
Self-distillation: the drafter learns what this model writes, timestamps and
speaker ids included. tools/dflash in vibevoice.c holds the pipeline:
- ~390 hours of audio, 8228 training clips and 43 held out, from LibriSpeech,
FLEURS (8 languages), SOVA, AMI, earnings calls and VoxConverse — audio
only; the datasets' own text is never used.
- The model's own greedy transcripts of every clip (
vv_dflash_data gen).
- Traces: every position's token, its role and the five tapped layers'
outputs, replayed through the runtime (
vv_dflash_data trace).
train.py (PyTorch, flex attention): anchors at generated positions, rows
1..7 scored against the next 7 tokens with weights e^(-k/4); cross-entropy
of the head plus the selector's; AdamW, cosine schedule, BF16.
Limits
- vibevoice.c only: the checkpoint keeps the published DFlash 2 names, but the
KV injection, the selector and the draft vocabulary are as vibevoice.c
implements them (
docs/DFLASH.md).
- CUDA only (the CPU path and Metal decode without the drafter). Greedy only.
- Audio the corpus covered little (Mandarin, Russian) keeps fewer drafts; the
runtime then falls back to plain steps and stays near plain speed.
License
MIT, like VibeVoice.
