Apple adds MLX distributed inference across Mac Studio systems

Apple released a distributed-inference setup for MLX that divides large language model workloads across two or four Mac Studio computers connected through RDMA over Thunderbolt. The configuration can run models whose weights and intermediate data exceed one Mac Studio’s unified memory. In Apple’s tests, the largest speed gains came from a communication-intensive tensor-parallel design.

The release adds systems work to Apple’s recent machine-learning research. Apple’s MLX research release describes two ways to split a model. In a two-system pipeline, the first Mac stores and computes the opening group of layers, then sends intermediate results to a second Mac that processes the remaining layers. In tensor parallelism, every Mac stores part of every layer and computes its portion simultaneously, requiring repeated broadcasts and aggregation as tokens move through the model.

Those designs require different network layouts. Apple used two Thunderbolt cables between the machines for its pipeline tests. Its tensor-parallel tests used a full peer-to-peer mesh across two or four systems, with two cables joining every pair. MLX’s distributed communication documentation identifies JACCL as the low-latency RDMA-over-Thunderbolt backend intended for communication-heavy patterns such as tensor parallelism. The documented JACCL path requires MLX 0.30.1 or later.

The host requirements are specific. Apple’s RDMA technote says RDMA over Thunderbolt is available on Apple-silicon Macs with Thunderbolt 5 beginning with macOS 26.2. An administrator must enable RDMA from macOS Recovery with rdma_ctl enable, reboot, and use ibv_devices to verify that the devices are visible.

MLX provides a configuration helper to check SSH access, map Thunderbolt links, confirm that a tensor-parallel cluster has a fully connected mesh, test RDMA and create a JACCL hostfile. The launch documentation says automatic setup requires passwordless sudo. Apple’s examples then run MLX LM through mlx.launch with the JACCL hostfile, --backend jaccl and MLX_METAL_FAST_SYNCH=1; the release labels the faster CPU-GPU synchronization path experimental. Current MLX documentation warns that this setting is unreliable, can deadlock and leave the GPU wedged, is off by default and is best left unset.

For its largest workload, Apple used Muse Spark Open, which it describes as an 816-billion-parameter mixture-of-experts model with 42 billion active parameters. Apple says the NVFP4 version occupies about 493GB before inference intermediates, leaving it unable to fit on one Mac Studio configured with the maximum 512GB of unified memory.

Apple reported that, at an 8,192-token context and batch size 1, Muse Spark Open decoded at 15.74 tokens per second with pipeline parallelism on two M5 Ultra systems, 24.99 tokens per second with two-system tensor parallelism and 36.73 tokens per second across four systems. Its reported time to first token was 8.4 seconds, 8.2 seconds and 4.8 seconds, respectively. Apple measured decoding across 128 generated tokens; the results have not been independently reproduced.

The company also said pipeline parallelism left decode speed close to a single-system result because the layers still execute sequentially, although splitting a 65,536-token prefill into 8,192-token chunks yielded up to roughly a 1.8-times prefill speedup by overlapping work. On four M5 Ultra systems at the longer context, Apple said two cables per machine pair reduced time to first token from about 50.5 seconds to 40.6 seconds compared with one cable per pair.

Apple reported peak draw of about 200 watts for its M3 Ultra system and 300 watts for its M5 Ultra system in the tested long-context workloads, but did not publish enough instrumentation detail to reproduce its efficiency comparison. The release also omits cluster pricing, cable models and lengths, run counts, variance and full raw benchmark logs.