Apple study finds English data mixing outpaced extra tuning in constrained Arabic pre-training

Apple-affiliated researchers found that adding fresh English data to constrained Arabic language-model pre-training improved Arabic validation loss more than searching for new learning-rate and weight-decay settings at each of the three larger model sizes tested. In this setup, data diversity mattered more than additional tuning. The finding comes from one Arabic–English experiment across four model scales and roughly 1,000 pre-training runs.

The study compared LLaMA-style decoder-only transformers with 150 million, 380 million, 600 million and 1.43 billion parameters. For the main comparison, the researchers limited the target-language pool to 200 million unique Arabic tokens from FineWeb2. They then spent the remaining training budget either repeating that Arabic text or sampling fresh English text from FineWeb.

The researchers paired those two data strategies with two tuning regimes. One used Maximal Update Parametrization, or μP, to carry settings chosen on the 150 million-parameter proxy to larger models as width changed. The other ran a separate search at each scale across five learning rates and five weight-decay values. For the bilingual runs, the researchers also tried seven Arabic repetition limits, producing a 5-by-5-by-7 grid at each scale.

At 380 million parameters, the authors report that English mixing cut Arabic validation loss by 5.7% from the monolingual baseline, compared with 1.2% from tuning alone. At 600 million parameters, the figures were 8.7% and 0.9%; at 1.43 billion, they were 12.9% and 3.1%. Tuning after mixing added less than 1% at every scale on that metric.

The authors point to two effects that tuning cannot fully reproduce. Fresh English examples diversified the training signal and reduced overfitting to repeatedly seen Arabic text. They also supplied knowledge missing from the constrained Arabic corpus. Adjusting learning rate and weight decay could regularize how a model learned from repeated data, but it could not add new information.

The same distinction appeared on the study’s only downstream benchmark, five-shot ARC Easy. With proxy-transferred settings, bilingual training improved Arabic accuracy over comparable monolingual training by 0.4 percentage points at 380 million parameters and 6.3 points at 1.43 billion. The gains in English were 9.8 and 19.5 points. Using a fitted monolingual baseline, the researchers estimated that mixing was equivalent to roughly two to three times as much unique Arabic data on validation loss and two to 13 times as much on downstream accuracy.

That limited return from more search did not mean hyperparameters were irrelevant. The loss landscape grew more sensitive at 1.43 billion parameters, yet μP transfer kept bilingual Arabic validation loss within 0.6% of the separately tuned optimum through the largest model tested. In the authors’ analysis of practical operating regions, the Arabic repetition or mixing setting accounted for most of the variation in Arabic validation loss.

For context on a different modality and language pair, Apple-affiliated research on language-aware English–French speech pre-training examined a separate controlled multilingual setup. This study covered only Arabic as the target language and English as the auxiliary language, with static mixing ratios and a shared XGLM tokenizer.

The downstream evidence came only from ARC Easy, whose Arabic version was machine-translated from English. Translation artifacts could therefore have contributed to the measured bilingual advantage. Each grid cell was trained once, without across-seed error bars. Some tuning optima also fell on a grid boundary, leaving open the possibility that better settings existed beyond the tested range.