Skip to content
Main Site News Console

TorchNeuron Performance Profiling (Part 1): AWS Trainium System-Level Performance Analysis

· Hugging Face Translated
教程模型卡

Many machine learning practitioners develop intuitions for making models run fast, but fewer know how to profile why they run slow. In part one of this series, you will learn how to systematically analyze inference performance and pinpoint exactly where time is spent.

This tutorial starts from scratch and assumes no prior profiling experience. It uses AWS Trainium—a custom AI chip designed for deep learning training and inference workloads—and its open-source native PyTorch backend, TorchNeuron. TorchNeuron enables running standard PyTorch workflows on Trainium with minimal code changes, supporting eager execution, torch.compile, and standard PyTorch APIs. It also integrates with the Neuron Kernel Interface (NKI) for developing custom operators and optimizing performance for Trainium hardware. By the end of this post, you will understand:

  • How to configure torch.profiler with the Neuron backend and capture clean traces.
  • How to read the key_averages() table to identify which operations consume Neuron device time and compare them against their Self CPU time.
  • How to follow a single operation across its full lifecycle in Neuron Explorer: from framework dispatch, through runtime scheduling, to hardware execution on a NeuronCore.
  • How three different implementations of the same operation differ across these signals.

Neuron as a PyTorch Backend

Before profiling a workload, it helps to understand how Neuron integrates with PyTorch.

The Neuron backend leverages PyTorch’s PrivateUse1 extension mechanism. This mechanism allows hardware accelerators to integrate into the framework without introducing a dedicated dispatch key (such as cuda). The torch_neuronx package works alongside upstream PyTorch to register Neuron as a native backend for PyTorch code. For more details on this extension mechanism, see Facilitating New Backend Integration via PrivateUse1.


Set Up the Profiler

Profiling a Neuron workload requires two core objects:

  • NeuronConfig: Controls which Neuron streams and NeuronCores are captured, and where the output is written. You pass this as the experimental_config to torch.profiler.profile.
  • NeuronProfiler: Takes a NeuronConfig and provides an export_trace callback function, which writes the framework-level trace.json alongside the Neuron system traces.
nc = NeuronConfig(
modes=[ProfileMode.DEVICE, ProfileMode.RUNTIME],
profile_output_dir=OUTPUT_DIR,
capture_enabled_for_nc='0',
correlate_device_time=True,
)
exporter = NeuronProfiler(nc)

The modes argument selects which streams to capture, such as device instructions and runtime traces; capture_enabled_for_nc specifies which NeuronCore to trace; and setting correlate_device_time=True brings device time from the trace into the key_averages() table, allowing you to directly compare CPU and Neuron device execution times. For complete parameter descriptions and all available ProfileMode enum values, refer to the NeuronConfig Profile Modes documentation.


Tooling: Neuron Explorer

Neuron Explorer is a browser-based visualization tool for viewing system traces, device timelines, and per-engine breakdowns. You can access it via an SSH tunnel or through the Visual Studio Code extension. In this post, you will use Explorer to verify the figures reported by key_averages() and inspect…