MBZUAI details CT-SpatialVQA; eight tested models score below 50%

Researchers at Mohamed bin Zayed University of Artificial Intelligence developed CT-SpatialVQA, a benchmark for spatial reasoning in chest CT scans. It contains 9,077 question-and-answer pairs drawn from 1,601 CT volumes and associated radiology reports. All eight 3D medical vision-language models tested by the authors scored below 50% accuracy under their LLM-as-Jury evaluation.

The benchmark provides a dedicated test of whether models preserve three-dimensional anatomical relationships when answering questions about a scan. The paper reports a mean accuracy of 34.08%. CT-Chat produced the highest score at 43.69%, followed by MedEvalKit at 40.16% and MedGemma-1.5 at 38.24%. The other five systems—RadFM, Med3DVLM, M3D, MERLIN and VILA-M3—scored between 28.20% and 31.49%.

Those results were reported by the research team. They were not independently reproduced and do not measure diagnostic performance.

CT-SpatialVQA covers six kinds of spatial information: laterality; superior-inferior position; anterior-posterior relationships; central-peripheral location; adjacency or containment; and spatial extent or boundaries. During testing, each model received a 3D CT volume and a spatial question, but not the radiology report used to derive the reference answer. The researchers reported weaker performance on vertical and depth-oriented relationships than on central-peripheral localization.

The team built the dataset from the test split of CT-RATE. GPT-4o generated candidate questions and answers from radiology reports, while Gemini 2.5 Flash checked whether each pair was spatial, concise and supported by the report. Items judged unsupported, ambiguous, non-spatial or overly verbose were discarded.

The researchers manually audited a random 12% of the validated pairs and reported 95% agreement with the automated validator, with no systematic discrepancies. The full dataset was not manually checked item by item, and its reference answers came from report text rather than new pixel-level annotations for every volume.

For the main accuracy measure, GPT-4o, Gemini 2.5 Flash and Qwen3 independently judged model answers as correct or incorrect, and their decisions were combined into an LLM jury. The authors also reported separate similarity metrics: an average SBERT cosine similarity of 0.512, BLEU of 4.816, ROUGE-L of 0.225 and METEOR of 0.193.

Because the headline accuracy depends on an LLM-based judging process, the result does not by itself establish diagnostic accuracy, patient outcomes, regulatory clearance or readiness for clinical use.

The MICCAI Society lists the paper in its 2026 proceedings. The authors have also published the question-answer dataset and evaluation materials, including generation and validation code and benchmarking scripts. The repository does not redistribute the CT scans; it instead points users to the CT-RATE download process.