Skip to content
Main Site News Console

TorchNeuron: Running HF Models on AWS Trainium Has Never Been Easier

· Hugging Face Translated
教程模型卡

TorchNeuron brings a brand-new native backend to PyTorch on AWS Trainium. With native support already available across Transformers, Diffusers, and Kernels, running models on Trainium and sharing them to the Hugging Face Hub is easier than ever.

TorchNeuron

TorchNeuron is a PyTorch plugin that delivers a new backend for Trainium, supporting eager mode, torch.compile, and native distributed APIs such as FSDP (Fully Sharded Data Parallel) and TP (Tensor Parallelism). Check out the official PyTorch blog post to learn more about TorchNeuron.

Starting October 20, 2026, TorchNeuron is available in Public Beta. Researchers and developers can install it via pip install torch-neuronx.

A New Backend and Native APIs in HF Libraries

Until now, the Hugging Face experience on AWS Trainium and Inferentia—AI chips purpose-built for cost-effective, large-scale training and inference—was primarily delivered through optimum-neuron. This library relied on the torch-neuronx package built on top of PyTorch/XLA. While a natural choice at the time, it required developers to adopt a compiler-first mindset: relying on lazy tensors instead of eager execution, strictly requiring static shapes, and triggering recompilations on every graph break. Optimum Neuron existed precisely to abstract away these complexities, maintaining custom code (model classes, Trainers, etc.) along the way.

TorchNeuron changes this paradigm. It integrates directly into PyTorch, exposing neuron as a first-class, native device with support for eager execution, PyTorch’s built-in distributed primitives (DTensor, FSDP2), and torch.compile.

For Hugging Face libraries, Trainium becomes just another standard device: Transformers, Diffusers, Accelerate, PEFT, and TRL run directly on it, using the exact same classes and scripts you would use on any other hardware accelerator. Going forward, we recommend Neuron users use these official libraries directly instead of going through Optimum Neuron.

Transformers

Inference

Transformers now offers native support for numerous text generation models. With TorchNeuron, 89% of the top 100 most popular text generation architectures work out of the box, compared to just 46% with Optimum Neuron v0.4.5. Simply pass device_map="neuron" in your code, and the model will automatically load onto Neuron devices.

Below is an example of text generation using Tensor Parallelism, helping you get the most out of your Neuron instances. Note that static cache is used during generation, which is strongly recommended on Neuron to prevent frequent recompilations.

import os
import torch
import torch.distributed as dist
import torch_neuronx # noqa: F401
from torch.distributed import DeviceMesh
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
from transformers.distributed import DistributedConfig
from transformers.generation.configuration_utils import CompileConfig
MODEL_ID = "Qwen/Qwen3.8-27B"
MAX_SEQ_LEN = 1024
PREFILL_CHUNK_SIZE = 512
dist.init_process_group(backend="neuron")
world_size = dist.get_world_size()
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
dtype=torch.bfloat16,
distributed_config=DistributedConfig(tp_size=world_size),
device_mesh=DeviceMesh("neuron", list(range(world_size)))