Toward Foundation Models in Radiology? Quantitative Assessment of GPT-4V’s Multimodal and Multianatomic Region Capabilities

URN to cite this document:: urn:nbn:de:bvb:355-epub-767898
DOI to cite this document:: 10.5283/epub.76789

Strotzer, Quirin D.

; Nieberle, Felix

; Kupke, Laura S. ; Napodano, Gerardo ; Muertz, Anna Katharina ; Meiler, Stefanie

; Einspieler, Ingo ; Rennert, Janine ; Strotzer, Michael ; Wiesinger, Isabel ; Wendl, Christina ; Stroszczynski, Christian

; Hamer, Okka W.

; Schicho, Andreas

License: Creative Commons Attribution 4.0
PDF - Published Version
(1MB)

Date of publication of this fulltext: 30 May 2025 14:05

Alternative links to fulltext:DOI Publisher

Details

Preview

Export bibliographical data

Abstract

Abstract

Background
Large language models have already demonstrated potential in medical text processing. GPT-4V, a large vision-language model from OpenAI, has shown potential for medical imaging, yet a quantitative analysis is lacking.
Purpose
To quantitatively assess the performance of GPT-4V in interpreting radiologic images using unseen data.
Materials and Methods
This retrospective study included single representative abnormal and healthy control images from neuroradiology, cardiothoracic radiology, and musculoskeletal radiology (CT, MRI, radiography) to generate reports using GPT-4V via the application programming interface from February to March 2024. The factual correctness of free-text reports and the performance in detecting abnormalities in binary classification tasks were assessed using accuracy, sensitivity, and specificity. The binary classification performance was compared with that of a first-year nonradiologist in training and four board-certified radiologists.
Results
A total of 515 images in 470 patients (median age, 61 years [IQR, 44–71 years]; 267 male) were included, of which 345 images were abnormal. GPT-4V correctly identified the imaging modality and anatomic region in 100% (515 of 515) and 99.2% (511 of 515) of images, respectively. Diagnostic accuracy in free-text reports was between 0% (0 of 33 images) for pneumothorax (CT and radiography) and 90% (45 of 50 images) for brain tumor (MRI). In binary classification tasks, GPT-4V showed sensitivities between 56% (14 of 25 images) for ischemic stroke and 100% (25 of 25 images) for brain hemorrhage and specificities between 8% (two of 25 images) for brain hemorrhage and 52% (13 of 25 images) for pneumothorax, compared with a pooled sensitivity of 97.2% (1103 of 1135 images) and pooled specificity of 97.2% (1084 of 1115 images) for the human readers across all tasks. The model exhibited a clear tendency to overdiagnose abnormalities, with 86.5% (147 of 170 images) and 67.7% (151 of 223 images) false-positive rates for the free-text and binary classification tasks, respectively.
Conclusion
GPT-4V, in its earliest version, recognized medical image content and reliably determined the modality and anatomic region from single images. However, GPT-4V failed to detect, classify, or rule out abnormalities in image interpretation.

Owner only: item control page

Download Statistics

Altmetric

Alternative Statistics (altmetrics)

More literature (via CORE)

Details

Preview

Export bibliographical data

Abstract

Abstract

Download Statistics

Downloads

Alternative Statistics (altmetrics)

More literature (via CORE)

University Library