Rise of the Machines: The Prevalence and Disclosure of Artificial Intelligence-Generated Text in High-Impact Orthopaedic Journals

Benjamin D Pesante; Cyril Mauffrey; Joshua A Parry

doi:10.5435/JAAOS-D-24-00318

Rise of the Machines: The Prevalence and Disclosure of Artificial Intelligence-Generated Text in High-Impact Orthopaedic Journals

J Am Acad Orthop Surg. 2024 Jun 27. doi: 10.5435/JAAOS-D-24-00318. Online ahead of print.

Authors

Benjamin D Pesante¹, Cyril Mauffrey, Joshua A Parry

Affiliation

¹ From the Department of Orthopaedics, Denver Health Medical Center, University of Colorado School of Medicine, Denver, CO.

PMID: 38941622
DOI: 10.5435/JAAOS-D-24-00318

Abstract

Introduction: While most orthopaedic journals permit the use of artificial intelligence (AI) in article development, they require that AI not be listed as an author, that authors take full responsibility for its accuracy, and that AI use be disclosed. This study aimed to assess the prevalence and disclosure of AI-generated text in abstracts published in high-impact orthopaedic journals.

Methods: Abstracts published from January 1, 2024, to February 19, 2024, in five orthopaedic journals were analyzed: the American Journal of Sports Medicine; the Journal of Arthroplasty; the Journal of Bone and Joint Surgery; the Knee Surgery, Sports, Traumatology, and Arthroscopy (KSSTA) journal; and the BMC Musculoskeletal Disorders (BMC MD) journal. Artificial intelligence detection software was used to evaluate each abstract for AI-generated text. Disclosure of AI use, country of origin, and article type (clinical, preclinical, review, or AI/machine learning) were documented. To evaluate the accuracy of AI detection software, 60 consecutive articles published in the Journal of Bone and Joint Surgery in 2014, before AI writing software was available, were also evaluated. These abstracts were evaluated again after being rewritten with AI writing software. The sensitivity and specificity of the software program for AI-generated text were calculated.

Results: A total of 577 abstracts were included in the analysis. AI-generated text was detected in 4.8% of abstracts, ranging from 0% to 12% by journal. Only one (3.6%) of the 28 abstracts with AI-generated text disclosed its use. Abstracts with AI-generated text were more likely to be from the Asian continent (57.1% vs. 28.0%, P = 0.001) and to involve topics of AI or machine learning (21.4% vs. 0.6%, P < 0.0001). The sensitivity and specificity of the AI detection software program were determined to be 91.7% (55/60) and 100% (60/60).

Discussion: A small percentage of abstracts published in high-impact orthopaedic journals contained AI-generated text, and most did not report the use of AI despite journal requirements.

Level of evidence: Diagnostic Level III.