parakeet-tdt_ctc-0.6b-ja — ONNX, INT8 (MatMul-only)
An INT8 ONNX build of nvidia/parakeet-tdt_ctc-0.6b-ja (TDT branch). It is the Japanese ASR model used by euhadra.
Only the encoder's constant-weight MatMuls are quantised. On FLEURS-ja this keeps the CER of the FP32 export. A build that quantises every op loses 35 %.
| Build |
Size |
CER, FLEURS-ja test (321) |
CER, euhadra CI subset (30) |
RTF (CPU) |
| FP32 export (source of this build) |
~2.5 GB |
0.0566 |
0.0724 |
0.057 |
INT8, all ops (OpenVoiceOS *.int8.onnx) |
~664 MB |
0.0765 |
— |
0.049 |
| This build: INT8, MatMul only, per-channel |
~885 MB |
0.0559 |
0.0737 |
0.030 |
Measurement conditions:
- Data: the FLEURS ja_jp test split, one recording per sentence id (321 utterances).
- Decoding: greedy TDT decoding via parakeet-rs-featsize, a fork of parakeet-rs that reads the mel size from the encoder. Upstream assumes 128 mels and can't load this 80-mel model.
- Scoring: mean per-utterance CER with euhadra's cer_lenient, which strips punctuation and canonicalises numerals.
- Hardware: one Apple Silicon machine, CPU execution provider. FP32 was measured twice: CER was identical and RTF 0.057 / 0.058.
On the 321 utterances, this build's hypothesis is identical to FP32's on 257. Of the rest, 21 score worse and 23 better. Measurements and discussion: euhadra#173.
Files
| File |
Notes |
encoder-model.int8.onnx + .data |
Encoder, MatMul weights in INT8 (per-channel) |
decoder_joint-model.onnx + .data |
Fused decoder/joint, FP32, unchanged from the source export |
vocab.txt |
Unchanged from the source export |
config.json |
Source config with encoder_path pointing at the INT8 encoder |
SHA256SUMS |
Hashes of every file above |
The layout is the one parakeet-rs-featsize loads: it looks for encoder-model.onnx, then encoder-model.int8.onnx. Don't put FP32 encoder files in the same directory.
Provenance
- Original model:
nvidia/parakeet-tdt_ctc-0.6b-ja, revision 44edb27eea9317daf89333e75eb830db4b1cc298, CC-BY-4.0.
- ONNX export used as input:
sunilmahendrakar/parakeet-tdt-0.6b-ja-onnx, revision 90b7d30b38ba48adf897d44b4f2e39894751b112, CC-BY-4.0 (inherited). This build was not exported from the .nemo checkpoint directly. The FP32 export's CER above is the reference it is checked against.
SHA-256 of the input files:
512862b7ab613a38131ecdc4027b93b3e7a811ea15023de84f63dd7a38cd77d0 encoder-model.onnx
4c9e2aac1b8c6c3bb3d50d8423af98d8c2b68aedaab78e5739dd2f910a78ae49 encoder-model.onnx.data
860ccf516531d6d1c5786f5a956d4c4d54c95f16095db61b29ca4789a7844918 decoder_joint-model.onnx
b0541fc4f837b67c08e32ead08823108b8ba1c8d80a620cf88610393a033aafa decoder_joint-model.onnx.data
e7a9c3a35c1e9e2b01424a6b3bf8ae23bfe456c9c90c546a57c9837ff2f5d496 vocab.txt
3. Quantisation: scripts/quantize_parakeet_ja.py in euhadra. The tool versions are pinned in the script (onnxruntime 1.31.0, onnx 1.23.2, numpy 2.4.6, Python 3.11):
python
quantize_dynamic(
"encoder-model.onnx", "encoder-model.int8.onnx",
op_types_to_quantize=["MatMul"],
weight_type=QuantType.QInt8,
per_channel=True,
use_external_data_format=True,
)
The output is byte-for-byte reproducible: running it again on the same inputs gives the same hashes as SHA256SUMS.
Modifications
Relative to the source export:
- Encoder: constant-weight MatMul nodes were dynamically quantised to INT8 (per-channel). All other encoder ops are FP32.
- config.json: encoder_path was changed to encoder-model.int8.onnx.
- Not changed: the decoder/joint and vocab.txt are byte-identical to the source.
- No fine-tuning or other training was done.
Known limitations
- Leading speech can be dropped. When a long clip (≳ 15 s) starts with a long non-speech lead (≳ 3.5 s), the model can drop the opening sentences. The FP32 export does the same, so the quantisation didn't introduce it. Trimming the lead, for example with a voice activity detector, avoids it. Details: euhadra#172.
- The numbers come from read speech. FLEURS is read speech on a single machine. Conversational or noisy audio has not been measured.
License and attribution
This model is a modified version of NVIDIA's parakeet-tdt_ctc-0.6b-ja. It is distributed under the Creative Commons Attribution 4.0 International License, the same license as the original. The changes made are listed under Modifications above.
