Journal article
Performance of ChatGPT on American Board of Surgery In-Training Examination Preparation Questions
The Journal of surgical research, Vol.299, pp.329-335
07/2024
DOI: 10.1016/j.jss.2024.04.060
PMID: 38788470
Abstract
Chat Generative Pretrained Transformer (ChatGPT) is a large language model capable of generating human-like text. This study sought to evaluate ChatGPT's performance on Surgical Council on Resident Education (SCORE) self-assessment questions.
General surgery multiple choice questions were randomly selected from the SCORE question bank. ChatGPT (GPT-3.5, April-May 2023) evaluated questions and responses were recorded.
ChatGPT correctly answered 123 of 200 questions (62%). ChatGPT scored lowest on biliary (2/8 questions correct, 25%), surgical critical care (3/10, 30%), general abdomen (1/3, 33%), and pancreas (1/3, 33%) topics. ChatGPT scored higher on biostatistics (4/4 correct, 100%), fluid/electrolytes/acid-base (4/4, 100%), and small intestine (8/9, 89%) questions. ChatGPT answered questions with thorough and structured support for its answers. It scored 56% on ethics questions and provided coherent explanations regarding end-of-life discussions, communication with coworkers and patients, and informed consent. For many questions answered incorrectly, ChatGPT provided cogent, yet factually incorrect descriptions, including anatomy and steps of operations. In two instances, it gave a correct explanation but chose the wrong answer. It did not answer two questions, stating it needed additional information to determine the next best step in treatment.
ChatGPT answered 62% of SCORE questions correctly. It performed better at questions requiring standard recall but struggled with higher-level questions that required complex clinical decision making, despite providing detailed responses behind its rationale. Due to its mediocre performance on this question set and sometimes confidently-worded, yet factually inaccurate responses, caution should be used when interpreting ChatGPT's answers to general surgery questions.
Details
- Title: Subtitle
- Performance of ChatGPT on American Board of Surgery In-Training Examination Preparation Questions
- Creators
- Catherine G Tran - University of IowaJeremy Chang - Department of Surgery, University of Iowa Hospitals & Clinics, Iowa City, IowaScott K Sherman - University of IowaJames P De Andrade - Department of Surgery, University of Iowa Hospitals & Clinics, Iowa City, Iowa. Electronic address: james-deandrade@uiowa.edu
- Resource Type
- Journal article
- Publication Details
- The Journal of surgical research, Vol.299, pp.329-335
- DOI
- 10.1016/j.jss.2024.04.060
- PMID
- 38788470
- eISSN
- 1095-8673
- Language
- English
- Electronic publication date
- 05/23/2024
- Date published
- 07/2024
- Academic Unit
- Surgery
- Record Identifier
- 9984630596802771
Metrics
6 Record Views