Large language models in radiology exams: A cross-sectional comparative analysis of performance in Turkish and English
Medicine (United States), vol.105, no.26, 2026 (SCI-Expanded, Scopus)
- Publication Type: Article / Article
- Volume: 105 Issue: 26
- Publication Date: 2026
- Doi Number: 10.1097/md.0000000000049517
- Journal Name: Medicine (United States)
- Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus, BIOSIS, CINAHL, EMBASE, MEDLINE, Directory of Open Access Journals
- Keywords: artificial intelligence, ChatGPT, Claude Sonnet, Google Gemini, Grok, large language models, radiology
- Kütahya Health Sciences University Affiliated: Yes
Abstract
This study evaluated the success of large language models on radiology questions, analyzing language variations, temporal consistency, and performance against residents. We evaluated ChatGPT-5, Grok-4, Claude 4.5 Sonnet, and Gemini 2.5 Pro using 100 multiple-choice questions across 5 subspecialties. Linguistic impact (Turkish vs English) and 1-week temporal reliability were assessed. Performance was benchmarked against a control group of 18 radiology residents (years 1–3). Gemini 2.5 Pro achieved the highest accuracy (90%), followed by Claude 4.5 Sonnet (86%). All models and 3rd-year residents significantly outperformed junior residents. While no significant performance gap existed between languages (P = 1.000), Claude 4.5 Sonnet demonstrated superior temporal reliability (κ = 0.872) compared with the moderate consistency of Grok-4 and ChatGPT-5. High-performance large language models provide accurate radiology knowledge comparable with senior residents, showing significant potential for education. Future research must incorporate image-based datasets to determine clinical efficacy.