Apple study finds MMLU weakly tracks LLM judge quality

Apple researchers found only a weak relationship between MMLU scores and how closely large language models matched human preferences when judging personalized writing, according to a study accepted at a NeurIPS 2026 workshop. Within these experiments, MMLU was therefore a weak proxy for evaluator quality; the result does not extend to every benchmark or evaluation setting.

Evaluator quality shifted across three long-form writing tasks and according to whether a judge assessed general helpfulness or adherence to a user’s preferred style. The judges were also more consistent and performed better when comparing two responses than when scoring responses independently.

The team generated responses with four LLMs. Human annotators chose the better of two responses, judge models made the same pairwise choice, and the researchers compared those model labels with the human labels. The experiments examined the writing task, the generator-and-judge combination, the evaluation objective, and general commonsense and reasoning ability.

Apple’s summary of the current workshop version reports Pearson correlations no greater than 0.2 between MMLU and judge quality. It also says the models Apple described as strongest did not match human evaluations most reliably. The authors hypothesized that stronger models may respond to subtle details that people and weaker models overlook, but the accessible summary does not establish that mechanism.

The work is workshop research, not a main-conference or journal paper. A university publication list independently identifies it as a NeurIPS 2026 workshop paper. The current public summary does not name the models, define the three tasks, report annotation counts, or provide uncertainty intervals.