Skip to content
Main Site News Console

Towards Light-Speed Text Generation: Nemotron-Labs Diffusion Language Models

· Hugging Face Translated
教程模型卡

1-headline-final

Large language models (LLMs) have now become the default interface for code generation, math problem solving, summarization, document understanding, and many other developer workflows. Under the hood, however, many LLMs still generate text the same way: one token at a time, with each token depending on the ones that came before it. That’s why these models are called autoregressive (AR): they consume the output they generate.

This autoregressive approach has been remarkably successful. It is stable to train, simple to serve, and has driven much of modern language modeling forward. But it also comes with a hard limitation: every new token requires a full model pass, and all weights must be loaded from memory before computation can begin. For developers building low-latency applications, working with smaller batches, or trying to make better use of modern GPUs, token-by-token generation leaves performance on the table because GPUs spend most of their time moving data rather than computing.

In addition, once an autoregressive model has generated a token, that token is final; the model itself cannot revise earlier tokens. As a result, errors can propagate throughout generation.

Nemotron-Labs Diffusion offers a new path: diffusion language models (DLMs), which generate multiple tokens in parallel and then iteratively refine them over multiple steps. Even better, these models can take more advantage of modern GPU compute patterns and deliver significant runtime performance gains. They can also revise already generated tokens, making them better suited for rewriting existing text and for fill-in-the-middle objectives. This generate-and-refine behavior also provides a built-in way to control the inference budget: by reducing the number of refinement steps, you can lower the runtime compute requirements of these models.

The Nemotron-Labs Diffusion family includes 3B, 8B, and 14B text models, all released under the commercially friendly NVIDIA Nemotron open model license; it also includes an 8B vision-language model (VLM), released under the NVIDIA Source Code License, which offers broader research flexibility. Across the lineup, NVIDIA releases both base models and instruction-tuned chat variants. NVIDIA also released the code used to train these models through the NVIDIA Megatron Bridge framework.

Three generation modes in a single model

2-tri-mode-final

The design of Nemotron-Labs Diffusion is built on a simple idea: autoregressive generation and diffusion generation should not be two separate model families, but capabilities of the same model. The model supports three generation modes:

Autoregressive mode runs in the standard left-to-right LLM style. This preserves compatibility with the generation workflows developers already know.

Diffusion mode generates in blocks and gradually produces tokens over multiple steps.

Self-speculation mode uses diffusion to draft multiple candidate tokens, then validates them with autoregressive decoding. This combines the speed potential of diffusion-style drafting with the reliability of AR verification.

This flexible design is a key developer-facing feature: it balances speed and accuracy even for workloads with unpredictable batch sizes or single-query scenarios (batch size = 1). Choosing the desired inference mode requires almost no application-side changes, because it is simply a deployment-time setting. As a result, developers can seamlessly switch to the model they are already using, or use Nemotron-Labs Diffusion across multiple inference modes for extremely fast generation.

Performance highlights

Screenshot from 2026-05-22 15-49-43

Compared with Qwen3 8B, Nemotron-Labs Diffusion 8B delivers a 1.2% improvement in average accuracy. When inference speed is measured in tokens per forward pass (TPF, a hardware-agnostic measure of token decoding efficiency), diffusion mode achieves 2.6x the TPF of the AR model, while self-speculation pushes that to 6x for linear self-speculation and 6.4x for quadratic self-speculation, with comparable accuracy on the evaluated tasks.

How we trained Nemotron-Labs Diffusion

Diffusion language models have been promising for years, but historically they have faced practical obstacles: lower accuracy than strong AR models, harder training, and limited compatibility with KV caches.

Recent work has shifted that direction. Efficient-DLM showed that a pretrained AR model can be converted into a diffusion language model through continued pretraining and a blockwise attention mechanism. This design helps preserve the capabilities of AR models while enabling KV-cache-friendly parallel decoding.

Nemotron-Labs Diffusion builds on the same practical insight: add diffusion capabilities to an existing AR model. The model is trained with a joint AR and diffusion objective, allowing it to retain knowledge learned during the initial AR training while adding parallel drafting capabilities through diffusion. It was first pretrained on 1.3T tokens from the NVIDIA Nemotron pretraining datasets, and then further supervised fine-tuned on 45B tokens from the NVIDIA Nemotron post-training datasets.