Apple study finds LLM translation grading unreliable in most settings

Large language models sometimes agreed with human translation graders as much as—or more than—the graders agreed with one another. Even so, Apple researchers found that neither pattern was reliable in most settings they tested.

The authors said setting-specific agreement tests and quality controls are needed before using LLMs to grade translations. They cautioned that beating a weak human-human baseline does not establish that an LLM can replace human evaluators.

The study compared four models—Gemini 2.5 Flash, GPT-OSS-120B, DeepSeek-V3.1 and Qwen3.5-397B—under two translation-quality schemes. It covered an internal long-context set spanning 70 language pairs, along with public WMT23 Multidimensional Quality Metrics (MQM) and WMT25 Error Span Annotation (ESA) data.

MQM asks annotators to identify and classify translation errors, while ESA uses a simpler scheme focused on error spans and severity. The researchers compared agreement on scores, error counts and spans. For scores and counts, they used Krippendorff’s alpha, a statistic that measures agreement while accounting for chance, and replaced each human annotation round in turn with an LLM annotation.

The researchers report that agreement varied sharply by annotation scheme, language pair, domain and model. Results were generally below a commonly used alpha threshold of 0.667 for tentative conclusions. The authors also said that threshold is not well established for subjective, fine-grained translation annotation, limiting how strongly it can be interpreted.

On WMT23 MQM scores, human-human agreement was significantly higher than LLM-human agreement for English-to-German and Chinese-to-English. The exception was Gemini 2.5 Flash on Chinese-to-English, where its agreement with humans was statistically on par with the human baseline. Across the MQM and ESA analyses, both people and models agreed less on minor errors than on major ones, with the models especially challenged by subtle mistakes.

WMT25 error-span labeling showed the clearest gap. Human-human agreement significantly exceeded every tested LLM-human pairing on all listed span measures. Character-overlap F1 was 0.43 for pairs of human annotations, compared with 0.16 to 0.18 for the four model-human pairings.

One test exposed a language-variant failure. The authors estimated that 60% of English-to-Egyptian-Arabic samples used Modern Standard Arabic instead of Egyptian Arabic. Human annotators consistently assigned those translations a score of zero, while the models gave them high scores despite instructions to penalize the wrong variant.

Human judgments were inconsistent, too. The researchers reported that fine-grained MQM labels and low-resource language pairs were particularly difficult for people. For English-to-Hindi and English-to-Thai in the long-context set, human-human agreement was near zero or negative, and one human round agreed much less with Gemini than the other. Related DataPhoenix coverage has examined Apple research on multilingual model gaps.

The model outputs also needed extra processing. At least 80% of spans reconstructed from model-provided character indices did not match the accompanying span text across every tested model and dataset. The researchers therefore searched for the supplied text and used its first match to realign spans. They also excluded any segment where at least one model returned no parseable annotation, removing 1.1% of WMT23 MQM, 5.1% of the long-context set and 7.3% of WMT25 ESA segments.

The evidence measures consistency with human annotators, not whether either side’s labels were objectively correct against an external ground truth. The study tested four models with one zero-shot prompt and a simple span-correction heuristic.

The authors could not release the internal 70-language-pair dataset because of licensing restrictions, limiting independent replication of those results, and said WMT23 data may have appeared in model training sets.