ReLMCodec

ReLMCodec · Inference checkpoints

Single-stream, 50 Hz speech coding at 650 / 800 bps.

Paper · Project page · Reproduction results

This model repository contains the complete ReLMCodec inference release: both codec checkpoints, their matching YAML configs, the frozen W2v-BERT 2.0 parameters embedded in each checkpoint, and the code needed to reconstruct audio. The weights have no optimizer, scheduler, discriminator, or RNG state.

The 8K checkpoint is the 200K perceptual-stage export. The 64K checkpoint is the available 50K additional fine-tune, selected by Log-Mel/PESQ/STOI on a separate 50-file dev set. The closest retained record to the paper's 64K table used an unavailable 30K fine-tune weight. Checkpoint provenance records this distinction and the actual Euclidean VQ assignment rule used by the released weights.

Quick inference

Use Python 3.10 or 3.11 on Linux. From a local snapshot of this model repository:

python -m venv .venv
source .venv/bin/activate
python -m pip install torch==2.9.0 torchaudio==2.9.0 \
  --index-url https://download.pytorch.org/whl/cu128
python -m pip install -e .
export USE_TF=0

python scripts/smoke_test.py --variant 64k
python infer.py reconstruct \
  --config configs/relmcodec_64k.yaml \
  --checkpoint models/relmcodec_64k.pt \
  --input input.wav --output reconstructed.wav

The model file and its config must match. Input audio is downmixed and resampled to 16 kHz. infer.py encode saves token IDs; infer.py decode reconstructs them. See Inference for commands and token format.

From the paper

Across 24 representations under a matched quantizer and language model, pre-quantization phoneme separability correlates with next-token accuracy (Spearman ρ = 0.911). Figure 3 shows how ReLMCodec preserves frozen SSL phoneme structure, controls acoustic adaptation with PAPA, and refines the latent space with a training-only WavLM teacher.

These images are cropped from the paper PDF. Table 2 gives the paper's LibriSpeech reconstruction results. The released checkpoints were rerun on all 5,559 test-clean + test-other utterances; verified release results include exact metrics, per-file JSON, and checkpoint hashes. Evaluation gives the reproduction protocol and external WER, SIM, and UTMOS evaluators. Those three external metrics were not freshly rerun for the packaged weights.

Scope and license

These weights are for speech reconstruction and token extraction, with mono 16 kHz input. The paper's predictability probes and downstream TTS models are not included. Code is MIT licensed; third-party notices cover embedded W2v-BERT parameters and vendored components.

@misc{wan2026relmcodec,
  title         = {ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure},
  author        = {Zixiang Wan and Xusheng Yang and Zheng Wang and Peiji Yang},
  year          = {2026},
  eprint        = {2608.08286},
  archivePrefix = {arXiv},
  primaryClass  = {eess.AS},
  url           = {https://arxiv.org/abs/2608.08286}
}