News

EUROPA adapts a 952B-parameter MoE training setup to JUPITER

The EUROPA consortium says it raised throughput for a 951.9-billion-parameter mixture-of-experts training configuration on JUPITER after adapting parallelism, memory use and optimization to the shared supercomputer. The result demonstrates an operating training system, not a completed or evaluated model.

D
Nov 19, 2026 · 3 min read

The EUROPA consortium has run and optimized a 951.9-billion-parameter mixture-of-experts training configuration on JUPITER, demonstrating how it adapted a shared public supercomputer for the workload. The experimental configuration had 32.8 billion active parameters per token, 74 layers, 384 routed experts with top-eight routing and a sequence length of 65,536 tokens.

The consortium said cumulative software and systems changes increased steady-state throughput from 90 to 1,293 tokens per second per GPU in single-replica tests using 384 GPUs across 96 nodes. Domyn defined the measurement as the median of steps four through 10 on rank zero and noted that some runs in its chart stopped early. The result shows the training setup operated at that scale; it does not establish that EUROPA completed the planned pretraining run or produced a model of any particular quality.

JUPITER’s Booster has 5,884 accelerated compute nodes, each with four NVIDIA GH200 Grace Hopper superchips. Each superchip pairs 96 GB of HBM3 with 120 GB of Grace memory, and the CPU and GPU connect through a 900 GB/s NVLink-C2C link. Four InfiniBand NDR200 adapters connect each node to the wider system. The consortium said the bandwidth gap between intra-node and inter-node communication made network traffic a central constraint.

The team used 384-way fully sharded data parallelism across the job and four-way expert parallelism within each node. It did not use pipeline parallelism, keeping the mixture-of-experts token exchange on the faster intra-node NVLink fabric. A batch of 384 sequences at 65,536 tokens amounted to about 25 million tokens per step. Domyn also reported a four-replica test across 1,536 GPUs at 89.4% of the per-GPU throughput of one replica, a scaling result that does not demonstrate linear scaling.

Several changes targeted memory capacity and communication. The team moved layer-boundary activations from Hopper HBM to Grace memory over NVLink-C2C, reporting an increase from 370 to 508 tokens per second per GPU and support for longer sequences. It used the Muon optimizer for most weight matrices and AdamW for the rest. TorchTitan’s FlexShard assigned whole experts to individual ranks to avoid cross-rank communication during Muon’s Newton-Schulz operations. Running those operations on GPUs instead of Grace CPUs cut the reported optimizer step from 41.7 seconds to 2.2 seconds.

The consortium also changed fully sharded data-parallel memory handling so all-gather setup and related buffers used reusable compute-stream memory pools. It reported that the change eliminated allocator retries across 384 ranks, lowered peak reserved memory from 88.1 GiB to 74.6 GiB and raised throughput from 996 to 1,210 tokens per second per GPU. Its technical account lists contributions to PyTorch and TorchTitan covering activation-offload hooks, optimizer support, random-number state handling and distributed-tensor behavior.

The work is part of the European Commission’s Frontier AI Grand Challenge, which offered one project access to up to 2.5% of EuroHPC computing capacity for one year. The Commission selected the Domyn-led EUROPA consortium in June to develop an open-source model covering all 24 official EU languages.

The reviewed sources do not document completion of a full EUROPA pretraining run, release of model weights, benchmark results or an independent quality evaluation. Domyn said the experimental configuration was still changing and that some values could change before the training run began. It also said the long-run quality effects of group-limited routing and possible FP8 expert calculations remained unresolved. The consortium said it would release the model and training recipes when the EUROPA model is published.

More news