ChatGPT, Bard, and Bing Chat Are Large Language Processing Models That Answered Orthopaedic In-Training Examination Questions With Similar Accuracy to First-Year Orthopaedic Surgery Residents

Gage A Guerra; Hayden L Hofmann; Jonathan L Le; Alexander M Wong; Amir Fathi; Cory K Mayfield; Frank A Petrigliano; Joseph N Liu

doi:10.1016/j.arthro.2024.08.023

ChatGPT, Bard, and Bing Chat Are Large Language Processing Models That Answered Orthopaedic In-Training Examination Questions With Similar Accuracy to First-Year Orthopaedic Surgery Residents

Arthroscopy. 2024 Aug 28:S0749-8063(24)00621-2. doi: 10.1016/j.arthro.2024.08.023. Online ahead of print.

Authors

Gage A Guerra¹, Hayden L Hofmann², Jonathan L Le², Alexander M Wong², Amir Fathi², Cory K Mayfield², Frank A Petrigliano², Joseph N Liu²

Affiliations

¹ USC Epstein Family Center for Sports Medicine, Keck Medicine of USC, Los Angeles, California, U.S.A.. Electronic address: gageguer@usc.edu.
² USC Epstein Family Center for Sports Medicine, Keck Medicine of USC, Los Angeles, California, U.S.A.

PMID: 39209078
DOI: 10.1016/j.arthro.2024.08.023

Abstract

Purpose: To assess ChatGPT's, Bard's, and Bing Chat's ability to generate accurate orthopaedic diagnoses or corresponding treatments by comparing their performance on the Orthopaedic In-Training Examination (OITE) with that of orthopaedic trainees.

Methods: OITE question sets from 2021 and 2022 were compiled to form a large set of 420 questions. ChatGPT (GPT-3.5), Bard, and Bing Chat were instructed to select one of the provided responses to each question. The accuracy of composite questions was recorded and comparatively analyzed to human cohorts including medical students and orthopaedic residents, stratified by postgraduate year (PGY).

Results: ChatGPT correctly answered 46.3% of composite questions whereas Bing Chat correctly answered 52.4% of questions and Bard correctly answered 51.4% of questions on the OITE. When image-associated questions were excluded, ChatGPT's, Bing Chat's, and Bard's overall accuracies improved to 49.1%, 53.5%, and 56.8%, respectively. Medical students correctly answered 30.8%, and PGY-1, -2, -3, -4, and -5 orthopaedic residents correctly answered 53.1%, 60.4%, 66.6%, 70.0%, and 71.9%, respectively.

Conclusions: ChatGPT, Bard, and Bing Chat are artificial intelligence (AI) models that answered OITE questions with accuracy similar to that of first-year orthopaedic surgery residents. ChatGPT, Bard, and Bing Chat achieved this result without using images or other supplementary media that human test takers are provided.

Clinical relevance: Our comparative performance analysis of AI models on orthopaedic board-style questions highlights ChatGPT's, Bing Chat's, and Bard's clinical knowledge and proficiency. Our analysis establishes a baseline of AI model proficiency in the field of orthopaedics and provides a comparative marker for future, more advanced deep learning models. Although in its elementary phase, future AI models' orthopaedic knowledge may provide clinical support and serve as an educational tool.