A group of researchers have tested artificial intelligence chatbot technology to respond to patient questions posted to a public social media forum, comparing them with responses from health and care professionals.
“Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum” was published 28 April in JAMA Intern Med, where the study evaluated the quality and empathy of responses to patient questions.
Using questions posted to an online forum, r/AskDocs (a subreddit with approximately 474 000 members), where verified healthcare professional volunteers submit answers, the study aimed to test and compare the responses to these questions. The answers that responded on the forum were used, as well as asking the same questions to the AI chatbot.
The study involved 195 randomly drawn patient questions from the forum. A team of three healthcare professionals compared the responses, and rated the chatbot responses “significantly higher for both quality and empathy”.
The paper describes how the “evaluators chose ‘which response was better’ and judged both the quality of information provided (very poor, poor, acceptable, good, or very good) and the empathy provided (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians.”
On the results, “evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations.” It also noted the number of words used in the responses “were significantly different, with the average physician response range 17-62 words, and the chatbot 168-245 words”.
The authors highlighted a number of limitations of the study, particularly in the use of the online forum question and answer exchanges, that “such messages may not reflect typical patient-physician questions”. They add that “we do not know how chatbots will perform responding to patient questions in a clinical setting, yet the present study should motivate research into the adoption of AI assistants for messaging.”
However, the researches conclude that further exploration is warranted, trialing the tech could improve the responses, and that the results suggest that artificial intelligence assistants may be able to aid in drafting responses to patient questions.
Citation: Ayers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. Published online April 28, 2023. doi:10.1001/jamainternmed.2023.1838
To view the study, please click here.
