ChatGPT (Generative Pre-trained Transformer), artificial intelligence (AI) software that operates like a chatbot, has received prominent media attention since its launch in November 2022. With applications for job recruitment and hiring, stock trading, and even writing essays, ChatGPT has the potential to change the landscape of various industries. Because ChatGPT interacts with users as if they’re having a conversation, the program can answer questions and provide follow-up information to users.
Since its arrival, questions have arisen regarding potential applications to support doctors or patients dealing with illness or chronic disease. Indeed, ChatGPT is intelligent enough for use in the medical field. One preprint study found that ChatGPT performed quite well on medical licensing exams, where the program scored comparably to a third-year medical student. Another preprint demonstrated how ChatGPT could write medical reports with correct and complete data. But, how deep is ChatGPT’s medical knowledge?
A recent study published in Clinical and Molecular Hepatology assessed how ChatGPT interacted with patients with questions related to two diseases: cirrhosis, an advanced form of liver disease, and hepatocellular carcinoma (HCC), the most common subtype of liver cancer. In the United States, about 1 in 400 adults develop cirrhosis, while experts project HCC rates to increase by 55% in the next two decades. The high incidence of these diseases, paired with the complexity of symptom management and treatment, creates a patient population who could benefit from more accessible and efficient ways of obtaining information and knowledge about their health. Further, because cirrhosis is a significant risk factor for HCC, disease management is essential for cancer prevention.
The study evaluated how accurately ChatGPT answered patients’ questions related to HCC. Two transplant hepatologists reviewed ChatGPT’s responses to assess the accuracy. The researchers found that ChatGCT provided accurate information about cirrhosis 79.1% of the time and about HCC 74% of the time. However, the doctors grading the quality of the knowledge ChatGPT relayed determined it to be comprehensive far less frequently (47.3% of the time for cirrhosis and 41.1% of the time for HCC).
The researchers classified ChatGPT as more appropriate for delivering “basic knowledge” related to lifestyle and treatment and less accurate for providing information on diagnostics and preventative measures. Further, the researchers found ChatpGPT’s report on the specifics of treatment recommendations and durations limited.
The researchers concluded that ChatGPT may serve a supportive role as an informational tool for patients and caregivers. ChatGPT can provide accurate and informative answers to patients’ questions related to cirrhosis and HCC. While the study is promising, some experts warn that we have a ways to go to optimize and maximize the benefits of ChatGPT for healthcare.
Sources: MedrXiv, arXiv, Clin Molec Hepatol, J Hepatol, Nature Med