Medical alert with ChatGPT Health: fails in 52% of emergencies and puts the user at risk

A recent independent evaluation by researchers at the Icahn School of Medicine at Mount Sinai revealed that ChatGPT Health, the system launched by OpenAI in January 2016 with the promise of revolutionizing triage and digital medical guidance, still has worrying shortcomings in classifying medical emergencies and appropriately referring patients.

According to the study, the AI-powered digital tool exhibits serious limitations in its ability to identify cases requiring urgent medical attention, raising questions about its suitability for direct use by consumers without expert supervision.

What the data says about the failure rate:

The research focused on 960 simulated medical scenarios, developed from 60 vignettes designed by physicians from various specialties and covering 21 clinical areas. A key finding of the report places the failure rate in emergency cases at 52%, while in non-urgent presentations, the failure rate was 35%.

In other words, according to the researchers, nearly half of the medical emergencies were not correctly identified or referred to emergency care services by the artificial intelligence.

The tool proved more effective at identifying recognized emergencies, such as strokes and severe allergic reactions, while in less common but equally critical clinical situations, such as diabetic ketoacidosis or impending respiratory failure, the error rate reached 52%.

Furthermore, the analysis showed that certain factors, such as some users' tendency to downplay their symptoms when reporting them, increased the likelihood that ChatGPT Health would suggest less urgent care.

What are the risks associated with ChatGPT Health's inaccurate responses

Within the study, one of the most serious shortcomings detected was ChatGPT Health's failure to consistently refer users with suicidal thoughts to the 988 crisis hotline.

According to the authors, “The ability to recognize mental health crises and connect users with crisis resources is a fundamental requirement for any consumer health platform. Our data show that this requirement has not been reliably met.”

The system was less likely to suggest the crisis hotline when patients described a specific self-harm plan, precisely in cases where immediate intervention is critical.

Isaac S. Kohane, director of the Department of Biomedical Informatics at Harvard Medical School, who was not involved in the study, emphasized the importance of ongoing external evaluations of these systems.

Kohane stated that “when millions of people use an AI system to decide whether they need emergency care, the stakes are high. Independent assessment should be routine, not optional.”

Why the debate about the role of artificial intelligence in healthcare is growing

Before the launch of ChatGPT Health, OpenAI had reported that around 25% of ChatGPT’s 800 million weekly users made at least one medical inquiry, which equates to about 40 million users already actively using the platform for health purposes.

Ashwin Ramaswamy, professor of urology at the Icahn School of Medicine at Mount Sinai and lead author of the study, explained the motivation behind the research.

“We wanted to answer a very basic but crucial question: if someone experiences a real medical emergency and goes to ChatGPT Health for help, will it clearly tell them they should go to the emergency room?” he said.

While the study's authors do not suggest banning the use of these platforms, they insist that healthcare professionals must educate patients on their use and the potential risks associated with them.

Where do people go during a medical emergency

According to the report, 73% of patients still turn to their doctors first for health information, compared to only 16% who use artificial intelligence.

Furthermore, in the words of Alvira Tyagi, a medical student and co-author of the study: “These systems are changing rapidly, so part of our training now needs to focus on learning to critically understand their results, identify where they fall short, and use them in ways that protect patients.”

While the integration of tools like ChatGPT Health into healthcare is progressing rapidly, experts emphasize that their development is still in its early stages and that it will be up to healthcare providers to establish guidelines for their safe use by patients.