Skip to content
Main Site News Console

# DiScoFormer: A Unified Transformer for Density and Score Across Distributions

· Hugging Face Translated
教程模型卡

Training a Transformer for Density and Score Estimation

📄 Technical report: arxiv.org/abs/2511.05924

DiScoFormer: A single Transformer for density and score estimation across distributions - Google -image-1

Many problems in machine learning and science can be reduced to the same kind of task: given a set of data points, you want to recover the distribution they came from — which values are more common, and which are rarer. Characterizing that distribution requires estimating two quantities: the density, and, in high dimensions, the more useful score. You can think of density as a smoothed version of a histogram — where data points cluster, density is high; where they are sparse, density is low. The score is the gradient of the log density, pointing in the direction where the density increases fastest: if you move a point along the score, it travels toward more probable regions.

Diffusion-based generative models, the technology behind AI image generators such as Stable Diffusion and DALL·E, start from random noise and repeatedly move in the score direction, turning noise into realistic images. The same score also drives Bayesian sampling and particle simulations used to model systems such as plasmas.

Extracting density and score from finite samples is challenging, and existing tools force us to trade off generalization and accuracy. One classic method — kernel density estimation (KDE) — computes the density at a location based on nearby data points: the closer and more numerous they are, the higher the density. It requires no training and works for arbitrary distributions, but its accuracy degrades rapidly as dimensionality grows. Another approach is neural network score matching models, which are trained to predict the score and remain highly accurate even in high-dimensional spaces, but each model must learn a specific distribution and has to be retrained from scratch for a different one.

We propose a new method called DiScoFormer (Density and Score Transformer) — a single model that, given a set of data points, can estimate both the density and the score of the underlying distribution in one forward pass, with no retraining required.

[
](#training-a-transformer-for-density-and-score-estimation)
Training a Transformer for Density and Score Estimation

DiScoFormer: A single Transformer for density and score estimation across distributions - Google -image-2

DiScoFormer uses stacked Transformer blocks to map an entire sample to the density and score of the distribution behind it. The model uses cross-attention, which lets it evaluate density and score not only on existing data points, but also at arbitrary locations. Score and density are mathematically linked: the score is the gradient of the log density. We take advantage of this by sharing a backbone network and using two output heads: one for density and one for score.

This coupling does more than save parameters. The score head must agree with the gradient of the log-density head at every query point, so any discrepancy between the two can be used as a label-free consistency loss. We use this at inference time — with the context fixed, taking a few gradient steps on this consistency loss lets DiScoFormer adapt on the fly to out-of-distribution inputs, without requiring ground-truth density or score supervision.

There is a mathematical reason the Transformer architecture is well suited to this task. KDE has only one bandwidth — the influence radius of each point — which is set in advance and applied uniformly everywhere. Attention is a strict generalization of this: we analytically prove that the weights of a single attention head are essentially a Gaussian kernel over the data, so a cross-attention block can already reproduce the density and score of KDE. On top of that, the model can learn multiple scales and adapt them to the data. DiScoFormer does not replace classical methods as a black box; it incorporates KDE as a special case and improves on it.

What do we train DiScoFormer on? For two main reasons, we use Gaussian Mixture Models (GMMs). First, GMMs are universal density approximators — with enough components, they can approximate almost any smooth distribution to arbitrary precision. Second, both the density and score of a GMM have closed-form expressions, so we always have exact supervision targets. We exploit both properties: for each batch, we sample a new GMM, giving the model effectively infinite target distributions, and supervise it against the exact density and score of each GMM.

[
](#performance)
Performance

Overall, DiScoFormer outperforms KDE on both density estimation and score estimation, and the gap is especially pronounced in the regimes where KDE struggles most. In 100 dimensions, the two are not even in the same league — compared with a heavily hand-tuned KDE, it reduces score error by about 6.5× and density error by more than 37×; and as the number of samples increases, it keeps improving while KDE has already run out of memory. It also generalizes well beyond the training data: it remains accurate on mixtures with more modes than seen during training, as well as on non-Gaussian shapes such as Laplace and Student-t distributions. KDE’s main advantage is still speed, especially on smaller datasets.

We believe the most exciting thing about DiScoFormer is that score estimation is a foundational capability shared across many fields, including generative modeling, Bayesian inference, and scientific computing. A pretrained, plug-and-play estimator that stays accurate in high-dimensional settings and removes the cost of retraining for every new problem could reduce overhead across all of them — one model, reusable wherever scores and densities are needed.

We encourage you to read our technical report for more details.