Higher-Order Pruning of Experts in Mixture-of-Experts Language Models
Introduction
Mixture-of-Experts (MoE) models have enabled state-of-the-art Large Language Models (LLMs) to scale to hundreds of billions of parameters, but they come with a fundamental and costly drawback: at inference time, every single expert MLP must reside in GPU memory (VRAM), even though the vast majority of experts sit idle for any given token. For example, Qwen3.5-122B features 256 experts across each of its 48 MoE layers, and 95% of its roughly 250 GB parameter footprint in VRAM is accounted for solely by these expert MLPs alone. Yet, for any given token, only 8 experts per layer are actually activated.
Expert pruning is one of the most direct approaches to tackle this VRAM bottleneck—permanently removing the least useful experts. Pruning bypasses the latency penalties that make dynamic loading/unloading of experts impractical; it requires zero training; and it can be combined with quantization techniques. Given these compelling benefits and the conceptual simplicity of the approach, it is easy to see why expert pruning has become the method of choice for compressing MoE model sizes. The challenge lies in determining which experts are safest to discard.
Existing pruning methods evaluate each expert independently, assigning a scalar importance score to each one (based on a calibration dataset) and pruning away the least important experts independently within each layer. These are first-order approaches, as they operate under the assumption that each expert’s contribution is independent of the others. In reality, experts form collaborative groups. We have observed consistent clustering of co-activated experts in our own experiments, and independent studies have similarly shown that expert collaboration is a stable property of MoEs and vital to model performance. Disregarding these interactions yields suboptimal pruning outcomes, particularly at high pruning ratios.
Our latest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions. At a 50% pruning ratio, HOPE achieves the best average rank across all methods, outperforming the next-best pruning method by up to +6.1% on agentic coding tasks.
In this post, we explain how HOPE works, highlight where it outperforms existing methods, and demonstrate how to apply it to your own models. Readers unfamiliar with the MoE architecture can refer to Mixture of Experts Explained.
Experts Do Not Work in Isolation
In a standard MoE layer containing $E$ experts, each input token $x$ is independently routed to $K$ of those experts (typically choosing $K = 8$). Routing is handled by a single layer that computes softmax-normalized gating weights $g_{k}(x)$ for each expert $k$; the $K$ experts with the highest gating weights are selected to form the active expert set $T(x) \subset {1, \dots, E}$. The layer output is computed as $\sum_{k \in T(x)} g_{k}(x) f_{k}(x)$, where $f_{k}(x)$ is the output of expert $k$ for token $x$.

Left: A typical MoE layer where each token is routed to top-K experts. Right: First-order methods rank and eliminate experts according to scalar importance scores, whereas optimal pruning decisions also factor in the interaction strength between experts.
Crucially, the experts within the selected set $T(x)$ are not independent of one another. Experts organize into stable collaborative groups that are consistently co-activated. This phenomenon has also been documented in prior literature…