Skip to content
Main Site News Console

Can Your Model Run on AWS Trainium? Find Out Now

· Hugging Face Translated
教程模型卡

In our previous post, we introduced TorchNeuron—the native PyTorch backend for AWS Trainium (the torch-neuronx package). With it, Transformers can run its modeling code directly on Trainium just like on any other device, making it possible to run any Transformers model on the Hub. But which models actually work?

The torch-neuronx model support Space answers this question for the most popular models on the Hub. It tells you whether a model runs out-of-the-box on Trainium and provides a script so you can run it yourself.

Image 9: torch-neuronx model support

Look up a model

The Space opens by default to the Selected view: a curated list of popular models based on download counts and trending rankings on the Hub. Each row shows the run result, the number of Trainium cores the model is sharded across, the wall time (including compilation time), and the peak device memory.

To find a specific model, switch to the Trending or Downloads view and type part of the model name. These views follow the Hub’s own rankings (divided by task) and list models whether they have been tested or not. When a model is intentionally excluded, the Space explains why (mostly because it is not prioritized, and occasionally due to actual incompatibility). You can also choose to inspect specific versions of the Neuron compiler (neuronx-cc) and Transformers.

At the time of writing, only Transformers models have been tested, with the curated selection limited to text-generation and image-text-to-text models. We plan to expand to diffusers, sentence transformers, and other transformers tasks in the future.

Run it yourself

Select a model to see how to run it. The Space displays the Neuron stack the model was tested with, any extra dependencies to install, and the actual run.py script used to run it. The script follows the recipe from the TorchNeuron announcement: load the model with from_pretrained, shard it when necessary, and generate using a static cache combined with torch.compile(backend="neuron").

On a Trainium2 (trn2) instance equipped with this stack (for example, inside an AWS Neuron Deep Learning Container), install the required dependencies and run:

Terminal window
python3 run.py

There is no need to use torchrun: when a model is sharded, the script automatically spawns its own torch.distributed ranks.

The first run compiles the model’s graph. To skip this step, you can directly copy the graph built during testing with the same compiler from the public NEFF cache bucket. The Space provides the command with the matching compiler version:

Terminal window
hf buckets sync hf://buckets/aws-neuron/torch-neuronx-neff-cache/neuronxcc-<version> /tmp/neff_cache

What “runs” means

Each test loads the model on a trn2 instance and shards it across the minimum number of Trainium cores needed to fit its weights. Most models are sharded using tensor parallelism. Mixture-of-Experts (MoE) models without a tensor parallelism recipe use expert parallelism alone. The test then generates 32 tokens and validates them. On a single core, the generated tokens are compared against the checkpoint’s output on CPU. When the model is sharded, a small judge model checks whether the text is coherent English.

At the time of writing, only models that fit within 4 Trainium2 devices (up to 16 cores) have been tested. A PASS simply means it runs—it does not reflect throughput performance or behavior at longer generation lengths.

The results are a dataset

Everything shown in the Space comes from the public aws-neuron/torch-neuronx-model-support dataset, with one Parquet file per neuronx-cc and library version combination.

Each row records the model’s run result, the error message on failure, the versions across the entire stack, and the exact run.py script that was executed. You can query it directly:

import pandas as pd
df = pd.read_parquet("hf://datasets/aws-neuron/torch-neuronx-model-support/data")
df[df.model_id == "Qwen/Qwen3-0.6B"][["neuronx_cc", "library", "library_version", "neuron_cores", "result"]]

The dataset card explains what each result means in detail.

Next time you want to know whether a model runs on Trainium, feel free to look it up.