
- Wait, what’s a TPU? 🤔
- The old way
- Enter TorchTPU 🚀
- Your favorite libraries, now on TPU
- We test it so you don’t have to 🧪
- Get started
If you’ve ever tried running PyTorch on TPUs, you likely had to port your code to torch_xla or ended up having to learn JAX. With TorchTPU, a native PyTorch backend, a TPU is now just another regular device. It unlocks native TPU support across the Hugging Face ecosystem. Transformers, Diffusers, and Kernels can now run natively on TPUs. Just grab a model from the Hub, send it to "tpu", and you’re good to go.
Wait, what’s a TPU? 🤔
TPU stands for Tensor Processing Unit. TPUs are Google’s custom AI accelerator chips. They were designed for one job: executing the matrix math that neural networks spend most of their time doing. Google has been building them for about a decade and uses them to train and serve its own models, including Gemini.
TPUs are available on Google Cloud, scaling from single chips up to Pod clusters with thousands of chips. They shine when you need serious compute for training or large-scale inference serving.
The catch has always been the software. The TPU software stack was originally built around TensorFlow and JAX, making things tricky for PyTorch users.
The old way
Before this, running PyTorch on TPUs meant going through PyTorch/XLA (torch_xla). It worked, but it had its share of pain points. Tensors were lazily evaluated (the tensors, not you), so your code didn’t always execute when you expected it to. Minor changes could trigger graph recompilations, and debugging could be elusive.
To make things easier, we built Optimum TPU to support inference and training of Hugging Face models on TPUs. Now that TorchTPU brings native support, Optimum TPU is no longer needed to run Hugging Face models with Transformers and Diffusers.
Enter TorchTPU 🚀
TorchTPU adds a brand-new TPU backend to PyTorch. You can write standard PyTorch code directly. It ships with TPU-tuned Pallas kernels, supports distributed training, and integrates torch.compile for extra speedups. Installation on a TPU VM takes just a single command:
pip install torch-tpu --extra-index-url https://download.pytorch.org/whl/cpuThe extra index installs the CPU build of torch. Without it, pip would pull the much larger CUDA build, which you don’t need on a TPU at all.
Your favorite libraries, now on TPU
Transformers
Transformers v5 is built around a single idea: to be the single source of truth for model definitions, enabling them to run anywhere. “Anywhere” now includes TPUs. To load a model on a TPU, simply pass device_map="tpu"—that’s all it takes.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Qwen/Qwen3.5-9B"model = AutoModelForCausalLM.from_pretrained( model_id, device_map="tpu", # 将模型直接加载到 TPU 上)tokenizer = AutoTokenizer.from_pretrained(model_id)inputs = tokenizer("The future of AI is", return_tensors="pt").to("tpu")out = model.generate(**inputs, max_new_tokens=64)print(tokenizer.decode(out[0], skip_special_tokens=True))Models run in eager mode by default. You can also use torch.compile for better performance, or run across distributed setups.