Amazon researchers introduce HOPE expert-pruning method

Amazon researchers introduced HOPE, a method for pruning mixture-of-experts language models that measures how pairs of experts contribute together instead of scoring each expert only in isolation. The team also released code to calibrate a model, select experts for removal and create a smaller standard checkpoint.

The method addresses a deployment burden in mixture-of-experts, or MoE, models: each token activates only a small subset of experts, but every expert still occupies memory at inference time. The researchers say HOPE made the strongest difference when 40% to 50% of experts were removed, the range in which deleting experts can break more of the cooperative groups that ordinary importance scores overlook.

In an MoE layer, a router sends each token to several experts and combines their outputs. First-order pruning methods give every expert a single importance score, then remove the lowest-ranked experts. That can miss dependencies: an expert that appears expendable on its own may be repeatedly activated with another important expert, so deleting it can disrupt a useful group.

HOPE records those relationships in an interaction matrix during a calibration pass. Diagonal entries represent individual expert importance, while off-diagonal entries represent pairwise contributions from experts activated together. For each layer, the method uses a continuous relaxation of a binary quadratic program, followed by rounding, to choose a set for removal. The paper describes REAP as the first-order special case in which the interaction terms are ignored.

The authors evaluated HOPE on Qwen3.5-122B-A10B, Qwen3.5-35B-A3B and GLM-4.5-Air across 54 conditions, using two calibration datasets, four first-order baselines, pruning rates from 10% to 50%, the Tulu3 Dev suite and SWE-Bench Pro. They report that HOPE had the best average rank across all conditions at 2.07 among five methods, finished first in 39% of conditions and in the top two in 67%, and recorded a 73% average head-to-head win rate.

Across the aggressive 40% to 50% pruning conditions, the authors report an average rank of 1.58 for HOPE. Their blog separately lists a 2.42 average rank for REAP in its 50% comparison. For Qwen3.5-35B-A3B with half of its experts removed, the blog gives HOPE a SWE-Bench Pro score of 42.3, versus 41.8 for the unpruned model and 36.2 for REAP. All of these performance figures come from the method’s authors and have not been independently replicated. The work is available as an arXiv preprint, and the opened sources do not identify acceptance at a peer-reviewed venue.

The released implementation still prunes every layer independently. The paper outlines a framework for modeling interactions across layers, but the researchers call those results preliminary. HOPE also captures pairs rather than triplets or larger expert groups, which the authors say would be combinatorially difficult to estimate.

The AWS Labs repository provides a three-step workflow: calibration writes expert-usage statistics to an HDF5 file, optimization writes the selected prune set to JSON, and pruning produces a smaller Hugging Face checkpoint. The Apache-2.0-licensed code has been tested on Qwen3 MoE, Qwen3-Next, Qwen3.5 MoE and GLM-4.5-Air, and requires a documented stacked-experts model layout.