Background: Large Language Models (LLMs) such as ChatGPT are gaining attention for their potential applications
in healthcare. This study aimed to evaluate the diagnostic sensitivity of various LLMs in detecting hip or
knee osteoarthritis (OA) using only patient-reported data collected via a structured questionnaire, without prior
medical consultation.
Methods: A prospective observational study was conducted at an orthopaedic outpatient clinic specialized in hip
and knee OA treatment. A total of 115 patients completed a paper-based questionnaire covering symptoms,
medical history, and demographic information. The diagnostic performance of five different LLMs—including
four versions of ChatGPT, two of Gemini, Llama, Gemma 2, and Mistral-Nemo—was analysed. Model-generated
diagnoses were compared against those provided by experienced orthopaedic clinicians, which served as the
reference standard.
Results: GPT-4o achieved the highest diagnostic sensitivity at 92.3 %, significantly outperforming other LLMs.
The completeness of patient responses to symptom-related questions was the strongest predictor of accuracy for
GPT-4o (p < 0.001). Inter-model agreement was moderate among GPT-4 versions, whereas models such as
Llama-3.1 demonstrated notably lower accuracy and concordance.
Conclusions: GPT-4o demonstrated high accuracy and consistency in diagnosing OA based solely on patientreported
questionnaires, underscoring its potential as a supplementary diagnostic tool in clinical settings.
Nevertheless, the reliance on patient-reported data without direct physician involvement highlights the critical
need for medical oversight to ensure diagnostic accuracy. Further research is needed to refine LLM capabilities
and expand their utility in broader diagnostic applications.