Skip to content
4ALL API Docs
Main Site News Console

# Bringing Nunchaku 4

· Hugging Face Translated
教程模型卡

Large Diffusion Transformers Can Now Be Loaded Natively in Diffusers via Nunchaku Lite

Large diffusion Transformers can generate stunning images (and even videos, audio clips, and now text), but loading modern text-to-image models in BF16 precision typically requires 20-30 GB of VRAM, which puts them out of reach for most consumer GPUs. Quantization is a powerful solution to this problem, and Diffusers already integrates multiple quantization backends, such as bitsandbytes, GGUF, torchao, and Quanto, which we covered in Exploring Quantization Backends in Diffusers.

Most of these backends are weight-only quantization. This means they store weights in low precision and dequantize them back to high precision during computation. While this significantly reduces VRAM usage, it usually does not speed up inference and may even introduce a slight latency overhead.

SVDQuant, the quantization method behind the popular inference engine Nunchaku, takes a different approach. It runs the main Transformer layers using 4-bit weights and activations (W4A4), reducing memory usage while also speeding up the denoising loop. We will detail how it works below, but until now, using these checkpoints has required a separate inference library.

In the current version of Diffusers, loading Nunchaku checkpoints is as simple as calling from_pretrained(), with no local CUDA compilation required, thanks to the kernels package. Additionally, the companion diffuse-compressor toolkit allows you to quantize new architectures yourself and publish them just like regular Diffusers repositories.

Nunchaku Lite image quality and performance comparison

Table of Contents

Getting Started with Nunchaku Lite

First, install the dependencies. You will need a recent version of Diffusers and Hugging Face’s kernels package:

pip install -U diffusers transformers accelerate kernels bitsandbytes

Then, load a pre-quantized pipeline just like you would load any other Diffusers model:

import torch
from diffusers import ErnieImagePipeline


pipe = ErnieImagePipeline.from_pretrained(
    "lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder",
    torch_dtype=torch.bfloat16,
).to("cuda")


image = pipe(
    prompt="A cinematic portrait of a red fox in a misty forest at sunrise, "
           "detailed fur, volumetric light",
    height=1024,
    width=1024,
    num_inference_steps=8,
    guidance_scale=1.0,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("output.png")

Comparison of BF16 and Nunchaku Lite outputs for the red fox prompt

No custom pipeline class is required, nor is a separate inference engine; nothing needs to be compiled locally. The NVFP4 kernels will be downloaded from the Hub via the Nunchaku Lite kernels page on first use. This checkpoint pairs the Nunchaku NVFP4 Transformer with a bitsandbytes NF4 text encoder, generating a 1024x1024 image on an RTX 5090 in about 1.7 seconds with a peak VRAM usage of around 12 GB, compared to approximately 24 GB for the BF16 pipeline. For more details on the Nunchaku Lite checkpoint format, you can check the official Diffusers documentation.

NVFP4 checkpoints require an NVIDIA Blackwell GPU (RTX 50 series, RTX PRO 6000, B200). For older generation GPUs, please use the INT4 variants. See the Hardware Support table below for details.

Background: SVDQuant and Nunchaku

SVDQuant is the quantization method behind Nunchaku, which is its reference CUDA inference engine. For diffusion Transformers, standard 4-bit quantization is challenging because of large outliers in both weights and activations. SVDQuant works by shifting the outliers in activations to the weights, using a small 16-bit low-rank branch to represent the most difficult parts of each weight matrix, and quantizing the remaining residuals to 4-bit. Nunchaku speeds up this process using fused kernels tailored for the 4-bit path and the low-rank branch.

Nunchaku kernel fusion: low-rank down-projection fused with input quantization, low-rank up-projection fused with 4-bit matrix multiplication
Nunchaku fuses the low-rank down-projection with the quantization kernel, and the low-rank up-projection with the 4-bit computation kernel, thereby eliminating the memory access overhead of the 16-bit branch. Image source: SVDQuant paper.

Introducing Nunchaku Lite

A significant portion of the speed advantage of the original Nunchaku engine comes from model-specific fused execution paths, such as fused QKV projections and fused GELU/MLP kernels. These optimizations are tightly coupled with the module layout and checkpoint format of each architecture, meaning that supporting new model families typically requires dedicated integration work for that model.

Nunchaku Lite is the new integration method in Diffusers. With it, Diffusers can load Nunchaku-style checkpoints