vitoom-nunchaku

Vitoom Nunchaku (Extended)

English | 中文

Community-maintained fork of Nunchaku. Nunchaku is an SVDQuant inference engine for 4-bit diffusion models (paper: SVDQuant). The upstream project has not been actively maintained for a long time. This fork keeps the original capabilities while extending model and quantization support, and distributes prebuilt wheels on Hugging Face.

Official docs remain useful as a baseline: Nunchaku Docs · DeepCompressor

Changelog

1.4.0

Quantized inference for three additional models. Published wheels cover Linux x86_64, Linux aarch64, and Windows, all built for CUDA 13.0:

• Linux x86_64: Python 3.11 + PyTorch 2.11; Python 3.11 / 3.12 / 3.13 + PyTorch 2.14
• Linux aarch64: Python 3.11 + PyTorch 2.11
• Windows: Python 3.11 / 3.12 / 3.13 + PyTorch 2.14

File names are listed under Prebuilt Wheels.

Anima checkpoints store plain INT8. from_pretrained(..., device="cuda") packs them for the kernel during load. Qwen-Image-2.1-Turbo and Krea2 use the existing W4A4 SVDQuant path, and those weights are already packed at export.

1.3.0

1.3.0.dev20260622 through dev20260629. Adds Chroma, FLUX.2 Klein, Qwen-Image W8A16, and quantized Qwen2.5-VL / Qwen3 text encoders. Those platform wheels are still installable; see below.

Extensions Over Upstream

Prebuilt Wheels

Current release: 1.4.0. 1.4 publishes Linux x86_64, Linux aarch64, and Windows builds. The 1.3 wheels for other platforms remain available.

Wheels are published on Hugging Face:

https://huggingface.co/tonera/vitoom-nunchaku

1.4.0 · x86_64 (Linux)

For typical x86 servers and workstations (consumer RTX 30 / 40 / 50 GPUs and compatible datacenter cards). Requires CUDA 13.0.

1.4.0 · aarch64 (Linux ARM64)

For ARM64 + CUDA 13.0 environments (e.g. GB10 / DGX Spark clusters). Cannot be installed on x86 machines.

1.4.0 · Windows (win_amd64)

For Windows workstations (consumer RTX 30 / 40 / 50 GPUs). Requires CUDA 13.0.

1.3.0 · x86_64 (Linux)

For typical x86 servers and workstations (consumer RTX 30 / 40 / 50 GPUs and compatible datacenter cards).

cu128 wheels: sm_75 / 80 / 86 / 89 / 120a — suitable for RTX 30 / 40 / 50 series.
cu130 wheels: all of the above plus sm_121a (GB10 / DGX Spark, etc.), while remaining compatible with RTX 50 series.

1.3.0 · aarch64 (Linux ARM64)

For ARM64 + CUDA 13.0 environments (e.g. GB10 / DGX Spark clusters). Cannot be installed on x86 machines.

1.3.0 · Windows (win_amd64)

For Windows workstations (consumer RTX 30 / 40 / 50 GPUs). Requires CUDA 13.0.

Use via Vitoom (Recommended)

If you only need image generation acceleration in production and prefer not to manage Python/CUDA/wheel versions yourself, use Vitoom instead of installing these wheels manually.

Vitoom is a Docker-based, locally deployable AIGC platform. Through a browser you can run text, image, audio, and video inference on your own hardware (RTX 30 / 40 / 50, DGX Spark, GB10, etc.). Its visual inference module ships with vitoom-nunchaku pre-integrated—Chroma, FLUX.2 Klein, Qwen-Image quantization, and related features are available out of the box in the Web UI after deployment.

Quick start (see the Vitoom README for full details):

git clone https://github.com/tonera/vitoom.git
cd vitoom
python scripts/setup_vitoom.py
python scripts/load_vitoom_images.py
docker compose up -d backend
docker compose -f docker-compose.inference.release.yml --profile visual up -d

Then open http://<LAN-IP>:8888 in your browser. Download models from the Web UI or via python scripts/download_initial_models.py.

Deployment guides: docker-usage-en.md · docker-usage-cn.md

Installation (Manual Wheel)

After downloading the wheel that matches your platform and Python version from Hugging Face:

pip install nunchaku-1.4.0+cu13.0torch2.11-cp311-cp311-linux_x86_64.whl

Or use the Hugging Face CLI:

hf download tonera/vitoom-nunchaku \
  nunchaku-1.4.0+cu13.0torch2.11-cp311-cp311-linux_x86_64.whl \
  --local-dir ./wheels

pip install ./wheels/nunchaku-1.4.0+cu13.0torch2.11-cp311-cp311-linux_x86_64.whl

Running also requires the latest diffusers source. The PyPI stable release does not yet include the models added in 1.4.0:

pip install git+https://github.com/huggingface/diffusers.git

Verify

python -c "import nunchaku; print(nunchaku.__version__)"

Which Wheel to Choose

Notes:

• Python tag (cp310 / cp311 / cp312 / cp313) must match your runtime.
• PyTorch version (torch2.11 / torch2.12 / torch2.14) must match your installed torch version.
• CUDA variant (cu128 / cu130) must match torch.version.cuda.
• ARM wheels and x86 / Windows wheels are not interchangeable.

Citation

If the inference acceleration in this project helps your research, please cite the original SVDQuant paper:

@inproceedings{
  li2024svdquant,
  title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models},
  author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun-Yan and Han, Song},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}

Disclaimer

This is a community-maintained extension of upstream Nunchaku. Wheels are distributed independently by the Vitoom team and are not an official release from MIT Han Lab / nunchaku-ai. For support, please open an issue on the corresponding Hugging Face repository or project tracker.