Chat Generative Pretrained Transformer Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test

Kelly Suchman; Shashank Garg; Arvind J Trindade

doi:10.14309/ajg.0000000000002320

Chat Generative Pretrained Transformer Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test

Am J Gastroenterol. 2023 Dec 1;118(12):2280-2282. doi: 10.14309/ajg.0000000000002320. Epub 2023 May 22.

Authors

Kelly Suchman¹, Shashank Garg², Arvind J Trindade^{1

3}

Affiliations

¹ Zucker School of Medicine at Hofstra/Northwell, Long Island Jewish Medical Center, New Hyde Park, New York, USA.
² Arkansas Gastroenterology, North Little Rock, Arkansas, USA.
³ Institute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, New York, USA.

PMID: 37212584
DOI: 10.14309/ajg.0000000000002320

Abstract

Introduction: Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing model that generates human-like text.

Methods: ChatGPT-3 and ChatGPT-4 were used to answer the 2022 and 2021 American College of Gastroenterology self-assessment tests. The exact questions were inputted in both versions of ChatGPT. A score of 70% or higher was required to pass the assessment.

Results: Overall, ChatGPT-3 scored 65.1% on 455 included questions and GPT-4 scored 62.4%.

Discussion: ChatGPT did not pass the American College of Gastroenterology self-assessment test. We do not recommend its use for medical education in gastroenterology in its current form.

MeSH terms

Gastroenterology*
Humans
Self-Assessment
Universities