sonari-wav2vec2-phoneme-int8

An int8, ONNX version of facebook/wav2vec2-lv-60-espeak-cv-ft, a wav2vec 2.0 model that outputs espeak IPA phoneme labels (not words). It is the file the speech service of Sonari, an English course for Vietnamese learners, runs on CPU to score pronunciation: CTC forced alignment of the learner's recording against the expected phonemes, then a goodness-of-pronunciation (GOP) score per phoneme. It is not used to transcribe speech.

Only the weights' format and precision changed. No training or fine-tuning was done.

The file

curl -L -o wav2vec2_int8.onnx \
  https://huggingface.co/duyentq/sonari-wav2vec2-phoneme-int8/resolve/f0652c415244e59f472e23fe3c629cea1e694a11/wav2vec2_int8.onnx
sha256sum wav2vec2_int8.onnx   # must print the digest above

Use the commit-pinned URL above, not main: Sonari pins this exact file by SHA-256 and refuses any other, because its thresholds are calibrated on it. Later commits to this repository (this card, the licence) leave that URL and its content unchanged.

Base model

Upstream describes it as wav2vec2-large-lv60 fine-tuned on multilingual Common Voice to recognise phonetic labels (Xu, Baevski, Auli, 2021; see Citation). This repository does not redistribute the vocabulary; take it from the upstream revision above.

How it was made

1. Export to ONNX (fp32, 1,263,740,983 bytes, SHA-256 8abaa871f21d481b217c43505f94892db2c103c853e970b784c118bc61842926): Wav2Vec2ForCTC with eager attention, traced with PyTorch 2.14.0 (legacy TorchScript exporter, dynamo=False) on 3 s of noise, opset 20, weights inline. The graph has one output, logits, without the log-softmax. Dynamic axes: batch and samples in, batch and frames out.
2. Dynamic int8 quantisation with onnxruntime.quantization.quantize_dynamic (onnxruntime 1.30.0, onnx 1.23.1): weight_type=QInt8, op_types_to_quantize=["MatMul"], and the defaults per_channel=False, reduce_range=False. quant_pre_process was skipped because its symbolic shape inference fails on this graph.

What the result contains (counted in the graph): 146 MatMulInteger nodes with 98 DynamicQuantizeLinear nodes, 292 int8 weight tensors; 48 MatMul nodes and all 8 Conv nodes stay fp32 (the feature-encoder and positional convolutions). Quantising every op type fails on the weight-normalised positional convolution, which is why only MatMul was requested.

The scripts are export.py and quantize.py --ops MatMul under spikes/gop/onnx/ in the Sonari repository. quantize_dynamic peaks at about 4 GB of memory on this model.

Calibration data

None. Dynamic quantisation fixes the int8 weights offline and computes the activation scales at run time from each input, so no calibration set was used or is needed.

Four clips were used to check the result, not to fit it (all from LibriSpeech dev-clean, CC BY 4.0, Panayotov, Chen, Povey, Khudanpur, ICASSP 2015): two 0.8 s clips, each around one target word (one starting with /θ/, one with /t/, chosen by script), and the first 3 s and 8 s of one fixed utterance.

Interface

• Input input_values: float32 [batch, samples], 16 kHz mono, normalised per utterance to zero mean and unit variance ((x - mean) / sqrt(var + 1e-7), what Wav2Vec2FeatureExtractor does).
• Output logits: float32 [batch, frames, 392], one frame per 320 samples (20 ms). Apply log-softmax yourself (Sonari does it in float64 for a stable difference).
• Token ids follow the upstream vocab.json. CTC blank is <pad> (id 0).
• Runs with onnxruntime's CPUExecutionProvider; Sonari uses one intra-op thread per request.

Measured behaviour

Against the PyTorch model on the same audio (log-posteriors):

Single-frame log-posteriors move by up to 2.8 nats, so thresholds must be calibrated on this exact file, not on the fp32 model (the fp32 ONNX export itself matches PyTorch to 3.3e-4 nats). On the one word pair used as a gate, the GOP of /θ/ in the correct clip went from +3.886 (PyTorch) to +3.596, and the GOP of /t/ when the reference said /θ/ went from -5.713 to -5.537, with /θ/ named as the competitor by both models; the largest change of any phoneme's GOP was 0.375 nats.

CPU cost on an i5-11400H laptop under WSL2 (onnxruntime 1.30.0), int8 against fp32:

These are laptop figures, not a server's.

Intended use and limits

• Intended: scoring which phonemes a learner produced, by alignment against a known reference, on CPU. The labels are espeak's phone symbols, which differ from textbook IPA in places.
• Not intended: transcribing speech into words, or any use that needs word-level language modelling.
• Checked on four clips and one word pair: enough to reject a broken export, not a certification of the int8 model in general. Accented and non-native speech, noise and long clips were not evaluated here.
• Inherits whatever biases and limits the upstream model has.

Licence and notice

This repository is distributed under the Apache License, Version 2.0 (file LICENSE), the licence of the upstream model, of which it is a derivative.

Changes made to the upstream work (Apache-2.0 section 4(b)): the weights were exported to ONNX and the MatMul weights were quantised to int8, as described above. Nothing else was changed.

NOTICE. At the pinned upstream revision (ae45363bf3413b374fecd9dc8bc1df0e24c3b7f4) the repository facebook/wav2vec2-lv-60-espeak-cv-ft contains no NOTICE file and no copyright statement: its files are README.md, config.json, preprocessor_config.json, pytorch_model.bin, special_tokens_map.json, tokenizer_config.json, vocab.json, and its card declares only license: apache-2.0. There is therefore no upstream NOTICE text to reproduce (section 4(d)). Attribution, as the upstream card gives it: Wav2Vec2-Large-LV60 finetuned on multi-lingual Common Voice, by Qiantong Xu, Alexei Baevski and Michael Auli, published under the facebook organisation on Hugging Face, original code in pytorch/fairseq (examples/wav2vec).

Citation

@article{xu2021simple,
  title   = {Simple and Effective Zero-shot Cross-lingual Phoneme Recognition},
  author  = {Xu, Qiantong and Baevski, Alexei and Auli, Michael},
  journal = {arXiv preprint arXiv:2109.11680},
  year    = {2021}
}