On July 30, 2026, Transformers enabled PyTorch’s scaled dot-product attention (SDPA) for T5. As a side effect, UMT5’s decoder was able to attend to future tokens. During training, it could even peek at the very answers it was supposed to predict. With no crashes or error messages, the bug lurked in the main branch for nearly two months.
When we later replayed those commits, a small UMT5 model running on a CPU flagged the issue on every single commit. This post introduces that model and the testing framework built around it.
Why bugs reach main
Transformers supports over 750 model types, and its test suite is massive: over 88,000 CPU tests on the main branch, plus more than 200,000 daily GPU tests. Model testing is only part of the story: we also test pipelines, training, distributed execution, quantization, and attention kernels.
Integration tests load real checkpoints and call the model’s API on real hardware. Running the full suite on every pull request (PR) would mean hogging large numbers of GPUs all day, which is impractical. By the time scheduled runs catch a regression, it may have already been merged into main.
Our approach is to scale models down. We built a fixture: a small model built from a real checkpoint configuration—with fewer and narrower layers, but retaining all of its layer types, along with identical tokenizer and processor files. It only needs to be published once and can be reused indefinitely. We run “fast integration tests” against it on inexpensive CPU runners. In our CPU study, the experimental UMT5 fixture weighed in at just 66 MB, compared to 1,227 MB for the original checkpoint.
This also covers extremely large models: some cannot fit on our standard GPU runners, but their fixtures can easily run on CPUs.
Model integrators already build small test models (such as the tiny-random-* checkpoints in Transformers). We wanted a standardized approach, along with checks that go beyond merely “loading the model and running a forward pass.” After all, that UMT5 bug sailed right through the forward pass without an issue.
Fixtures don’t replace full integration tests, but at the PR stage, they give us a chance to catch bugs before they ever reach main.
Introducing Model Integration Contracts
transformers-mic is a proposed standard for packaging these tests. A Model Integration Contract (MIC) consists of three parts:
- A fixture: The scaled-down model described above, published once and pinned to a revision.
- A usage path: A runnable Markdown example demonstrating what people typically do with the model, such as generating text or running a training step.
- Baselines: Recorded, human-reviewed outputs from running that example.
If running the usage path on the fixture still reproduces those baselines, the contract holds.
A contract covers an architecture and a usage path rather than a single checkpoint: compatible checkpoints can share a fixture while retaining their own expected outputs. We publish them under huggingface-mic.
For example, the SmolLM contract generates text from a prompt like “The capital of France is” and compares its token IDs against recorded answers. After loading the model and tokenizer, the generation code (simplified to a single prompt) looks like this:
inputs = tokenizer("The capital of France is", return_tensors="pt")outputs = model.generate(**inputs, max_new_tokens=4, do_sample=False)new_tokens = outputs[:, inputs["input_ids"].shape[1]:]print(tokenizer.batch_decode(new_tokens, skip_special_tokens=True))It also checks training loss, verifies that gradients are finite, and confirms that an optimizer step updates the weights. For the full details, see usage.md.
The fleet dataset on the Hub pins each MIC to a frozen revision and records the dependency versions used when its baselines were recorded. Each CI run logs the fleet revision it used so it can be reproduced later.
In PRs, we filter which MICs to run based on the files touched by the patch.
The UMT5 regression
We replayed frozen MICs across five months of Transformers commits. This small UMT5 fixture failed consistently starting from July 30, 2026, when the regression was introduced, until