MBZUAI introduces MedRoundsQA benchmark for diagnostic conversations

Mohamed bin Zayed University of Artificial Intelligence has introduced MedRoundsQA, a benchmark that tests how language-model doctor agents gather information and diagnose cases in simulated patient conversations. According to MBZUAI and the research team, its controlled design spans 1,387 cases and 17 medical specialties.

Unlike a single-prompt evaluation that provides the complete case at once, MedRoundsQA requires a doctor agent to question a patient agent over multiple turns. This tests whether the model can collect the details it needs to reach a diagnosis. The paper appears in the October 2026 EMNLP proceedings, where the Association for Computational Linguistics lists it on pages 2433–2458.

The researchers derived the cases from board-exam questions, then converted each one into a structured clinical record with 24 information slots. Doctor and patient agents use those records to generate consultations. Because the underlying record remains fixed, researchers can compare a single-turn diagnosis with a multi-turn consultation built from the same normalized case.

MedRoundsQA also varies how the simulated patient communicates along four axes: education or health literacy, emotional tone, language proficiency and occupational framing. Cases are labeled easy, medium or hard using a model-based uncertainty score. These controls test whether the way information is expressed affects diagnostic performance without changing the underlying clinical content.

In the authors’ evaluation of 15 language-model doctor agents, diagnostic accuracy fell by about 13 to 39 percentage points when the models moved from complete, single-turn records to simulated multi-turn consultations. The authors also reported that accuracy generally plateaued after six to 12 turns and that changes in the education persona shifted accuracy by about seven to eight points. The results have not been independently reproduced.

The study also compared model agents with three resident physicians on 60 difficult cases using the same simulated-patient interface. The authors reported average diagnostic accuracy of 48% for the residents and 27% for the model agents. This limited comparison is part of the benchmark evaluation, not a clinical trial or evidence about patient outcomes.

The paper describes text-only, single-encounter simulations based on publicly available, de-identified cases and synthetic conversations. It says the study collected no new patient data and involved no real patients. The benchmark does not cover longitudinal records or medical imaging, and the research does not establish prospective clinical validity, safety in a care setting, regulatory status or deployment readiness. The authors say it is not a substitute for professional diagnosis or medical decision-making.