ChatGPT succeeds in a difficult, years-long examination for US medical licensure.
According to research, Microsoft's AI chatbot ChatGPT may achieve scores of at or around the roughly 60% passing mark for the United States Medical Licensing Exam (USMLE), with comments that make sense internally and frequently contain insights.
ChatGPT uses word prediction to create writing that resembles that of a human. ChatGPT is unable to conduct online searches, unlike most chatbots. Instead, it produces text based on word relationships that are predicted by internal processes.
ChatGPT's performance on the USMLE was evaluated in the study, which Tiffany Kung, Victor Tseng, and colleagues at AnsibleHealth published in the open-access journal PLOS Digital Health.
The USMLE is a test that medical students and doctors-in-training take to gauge their understanding of the majority of medical specialties, from biochemistry to diagnostic reasoning to bioethics.
The authors evaluated the software on 350 of the 376 public questions available from the June 2022 USMLE release after screening to eliminate image-based questions.
After removing undetermined responses, ChatGPT's scores on the three USMLE tests ranged from 52.4% to 75%.
Additionally, ChatGPT showed a concordance of 94.6% across all of its responses and generated at least one substantial insight (something novel, original, and clinically valid) for 88.9% of its responses.
Notably, ChatGPT performed better than PubMedGPT, a rival model that was only trained on literature within the biomedical domain, which obtained a score of 51.8% on an earlier dataset of USMLE-style questions.
"Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation," said the authors.
"ChatGPT contributed substantially to the writing of our manuscript. We interacted with ChatGPT much like a colleague, asking it to synthesize, simplify, and offer counterpoints to drafts in progress. All of the co-authors valued ChatGPT's input," said Kung.
