Many machine learning practitioners develop intuitions for making models run fast, but fewer know how to profile why they run slow. In part one of this series, you will learn how to systematically analyze inference performance and pinpoint exactly where time is spent.
This tutorial starts from scratch and assumes no prior profiling experience. It uses AWS Trainium—a custom AI chip designed for deep learning training and inference workloads—and its open-source native PyTorch backend, TorchNeuron. TorchNeuron enables running standard PyTorch workflows on Trainium with minimal code changes, supporting eager execution, torch.compile, and standard PyTorch APIs. It also integrates with the Neuron Kernel Interface (NKI) for developing custom operators and optimizing performance for Trainium hardware. By the end of this post, you will understand:
- How to configure
torch.profilerwith the Neuron backend and capture clean traces. - How to read the
key_averages()table to identify which operations consume Neuron device time and compare them against their Self CPU time. - How to follow a single operation across its full lifecycle in Neuron Explorer: from framework dispatch, through runtime scheduling, to hardware execution on a NeuronCore.
- How three different implementations of the same operation differ across these signals.
Neuron as a PyTorch Backend
Before profiling a workload, it helps to understand how Neuron integrates with PyTorch.
The Neuron backend leverages PyTorch’s PrivateUse1 extension mechanism. This mechanism allows hardware accelerators to integrate into the framework without introducing a dedicated dispatch key (such as cuda). The torch_neuronx package works alongside upstream PyTorch to register Neuron as a native backend for PyTorch code. For more details on this extension mechanism, see Facilitating New Backend Integration via PrivateUse1.
Set Up the Profiler
Profiling a Neuron workload requires two core objects:
NeuronConfig: Controls which Neuron streams and NeuronCores are captured, and where the output is written. You pass this as theexperimental_configtotorch.profiler.profile.NeuronProfiler: Takes aNeuronConfigand provides anexport_tracecallback function, which writes the framework-leveltrace.jsonalongside the Neuron system traces.
nc = NeuronConfig( modes=[ProfileMode.DEVICE, ProfileMode.RUNTIME], profile_output_dir=OUTPUT_DIR, capture_enabled_for_nc='0', correlate_device_time=True,)exporter = NeuronProfiler(nc)The modes argument selects which streams to capture, such as device instructions and runtime traces; capture_enabled_for_nc specifies which NeuronCore to trace; and setting correlate_device_time=True brings device time from the trace into the key_averages() table, allowing you to directly compare CPU and Neuron device execution times. For complete parameter descriptions and all available ProfileMode enum values, refer to the NeuronConfig Profile Modes documentation.
Tooling: Neuron Explorer
Neuron Explorer is a browser-based visualization tool for viewing system traces, device timelines, and per-engine breakdowns. You can access it via an SSH tunnel or through the Visual Studio Code extension. In this post, you will use Explorer to verify the figures reported by key_averages() and inspect…