justsumguy/Realtime-Venus-Omni-ablated,
quantised with NVIDIA ModelOpt 0.46.1. It is 9.1 GB on disk. The abliteration details and numbers are on that page. In
short: heretic ARA branch, ARA-LoRA rank 128, trial 83, refusals 98 → 4 of 100, KL 0.249 on
recheck, no healing applied.
Quantisation
| Component |
Format |
Notes |
llm (252 linears) |
NVFP4 W4A4 |
block 16, FP8 E4M3 block scales, FP32 global scale, static activation scale; lm_head and embed_tokens BF16; KV cache not quantised |
vpm (162 linears) |
FP8 E4M3 per tensor, W8A8 |
patch_embedding BF16 |
apm, resampler, audio_projection_layer, tts |
BF16 |
unchanged |
- Recipes: ModelOpt
NVFP4_DEFAULT_CFG for llm; FP8_DEFAULT_CFG for vpm, with patch_embedding excluded.
llm calibration: 256 samples × 512 tokens of tatsu-lab/alpaca (shuffled, seed 0), run through the extracted text tower. Mean NLL on 8 calibration chats: BF16 3.941, NVFP4 fake-quant 4.065 (+0.123).vpm calibration: 80 images from johnowhitaker/imagenette2-320. Three quarters were unsliced; one quarter used max_slice_nums=4.vpm held-out check (16 images):
| Output |
Mean cosine |
Min cosine |
Mean rel. L2 |
| vision tower, last hidden state |
0.972 |
0.957 |
0.234 |
| resampler |
0.989 |
0.977 |
0.145 |
- Hardware:
- One RTX 5060 Ti 16 GB (SM120) and one RTX 3050 Ti Laptop 4 GB (SM86), with
lm_head on CPU during calibration. The host had 15 GB of RAM.
- The BF16 tower was streamed from the shards across both GPUs. It was never materialised on one card.
Loading
transformers 4.51 (the version upstream's remote code pins) has no ModelOpt quantiser, so
from_pretrained cannot load this export. nvfp4_loader/ is a small loader that does it. It:
- builds the model on the meta device;
- replaces the quantised projections with
NVFP4Linear / FP8Linear;
- streams the shards directly onto the GPU.
The layers run on torch._scaled_mm FP4/FP8 kernels, with a fused Triton kernel for NVFP4
activation quantisation. Tested on torch 2.8.0+cu128, on an RTX 5060 Ti (SM120) only.
import os, sys, torch
os.environ["HF_HUB_OFFLINE"] = "1"
sys.path.insert(0, "/path/to/Realtime-Venus-Omni-ablated-NVFP4")
from nvfp4_loader import load_nvfp4_model
path = "/path/to/Realtime-Venus-Omni-ablated-NVFP4"
model = load_nvfp4_model(path, device="cuda:0")
model.init_tts(model_dir=os.path.join(path, "assets/token2wav"), n_timesteps=8)
duplex = model.as_duplex(generate_audio=True, n_timesteps=8)
OMNI_NVFP4_ACT=dynamic switches activation scaling from the checkpoint's static
input_scale to a per-call amax.
Realtime measurements
These were measured on one RTX 5060 Ti 16 GB. The serving sidecar is custom and not included here. On top of the loader
above, it adds:
- a static KV cache (8192 slots) with CUDA graphs over the language model's and the talker's single-token steps;
- one
lm_head application per step;
- a batched system-prompt prefill;
- a CUDA graph over the token2wav flow DiT chunk step, with the vocoder in FP16 and the solver cache sized to the step count.
|
|
| Peak VRAM, duplex session |
12.4 GB reserved by PyTorch (11.6 GB allocated), 12.8 GB for the process in nvidia-smi |
| Weights alone |
8.2 GB |
| Cold load |
106 s |
| Speaking unit (1 s of audio in, 1 s of speech out), 8 vocoder steps |
0.38–0.53 s compute: prefill 0.09–0.17, LLM 0.05–0.18 (1–5 text tokens), talker 0.11–0.15 (25 speech tokens), token2wav 0.07–0.09 |
| Listening unit |
0.12–0.16 s |
On an RTX 5060 Ti, with these optimisations, it should run in realtime with 8 vocoder
(token2wav flow-matching) steps.
License
- Model weights, config and remote code: Apache-2.0, as upstream (see
LICENSE and NOTICE). The modifications are the abliteration of the language tower and the quantisation described above.
nvfp4_loader/: MIT, Copyright (c) 2026 Schneewolf Labs.
