Humans Make Better Cancer Treatment Decisions Than AI, Study Finds

Treating cancer is becoming increasingly complex, but also offers more and more possibilities. After all, the better a tumor’s biology and genetic features are understood, the more treatment approaches there are. To be able to offer patients personalized therapies tailored to their disease, laborious and time-consuming analysis and interpretation of various data is required. Researchers at Charité – Universitätsmedizin Berlin and Humboldt-Universität zu Berlin have now studied whether generative artificial intelligence (AI) tools such as ChatGPT can help with this step. This is one of many projects at Charité analyzing the opportunities unlocked by AI in patient care.

If the body can no longer repair certain genetic mutations itself, cells begin to grow unchecked, producing a tumor. The crucial factor in this phenomenon is an imbalance of growth-inducing and growth-inhibiting factors, which can result from changes in oncogenes – genes with the potential to cause cancer – for example. Precision oncology, a specialized field of personalized medicine, leverages this knowledge by using specific treatments such as low-molecular weight inhibitors and antibodies to target and disable hyperactive oncogenes.

The first step in identifying which genetic mutations are potential targets for treatment is to analyze the genetic makeup of the tumor tissue. The molecular variants of the tumor DNA that are necessary for precision diagnosis and treatment are determined. Then the doctors use this information to craft individual treatment recommendations. In especially complex cases, this requires knowledge from various fields of medicine. At Charité, this is when the “molecular tumor board” (MTB) meets: Experts from the fields of pathology, molecular pathology, oncology, human genetics, and bioinformatics work together to analyze which treatments seem most promising based on the latest studies. It is a very involved process, ultimately culminating in a personalized treatment recommendation.

Can artificial intelligence help with treatment decisions?

Dr. Damian Rieke, a doctor at Charité, Prof. Ulf Leser and Xing David Wang of Humboldt-Universität zu Berlin, and Dr. Manuela Benary, a bioinformatics specialist at Charité, wondered whether artificial intelligence might be able to help at this juncture. In a study just recently published in the journal JAMA Network Open*, they worked with other researchers to examine the possibilities and limitations of large language models such as ChatGPT in automatically scanning scientific literature with an eye to selecting personalized treatments.

“We prompted the models to identify personalized treatment options for fictitious cancer patients and then compared the results with the recommendations made by experts,” Rieke explains. His conclusion: “AI models were able to identify personalized treatment options in principle – but they weren’t even close to the abilities of human experts.”

The team created ten molecular tumor profiles of fictitious patients for the experiment. A human physician specialist and four large language models were then tasked with identifying a personalized treatment option. These results were presented to the members of the MTB for assessment, without them knowing where which recommendation came from.

Improved AI models hold promise for future uses

“There were some surprisingly good treatment options identified by AI in isolated cases,” Benary reports. “But large language models perform much worse than human experts.” Beyond that, data protection, privacy, and reproducibility pose particular challenges in relation to the use of artificial intelligence with real-world patients, she notes.

Still, Rieke is fundamentally optimistic about the potential uses of AI in medicine: “In the study, we also showed that the performance of AI models is continuing to improve as the models advance. This could mean that AI can provide more support for even complex diagnostic and treatment processes in the future – as long as humans are the ones to check the results generated by AI and have the final say about treatment.”

AI projects at Charité aim to improve patient care

Prof. Felix Balzer, Director of the Institute of Medical Informatics, is also certain medicine will benefit from AI. In his role as Chief Medical Information Officer (CMIO) within IT, he is responsible for the digital transformation of patient care at Charité. “One special area of focus when it comes to greater efficiency in patient care is digitalization, which also means the use of automation and artificial intelligence,” Balzer explains.

His institute is working on AI models to help with fall prevention in long-term care, for example. Other areas at Charité are also conducting extensive research on AI: The Charité Lab for Artificial Intelligence in Medicine is working to develop tools for AI-based prognosis following strokes, and the TEF-Health project, led by Prof. Petra Ritter of the Berlin Institute of Health at Charité (BIH), is working to facilitate the validation and certification of AI and robotics in medical devices.

Reference: Benary M, Wang XD, Schmidt M, et al. Leveraging large language models for decision support in personalized Oncology. JAMA Netw Open. 2023;6(11):e2343689. doi: 10.1001/jamanetworkopen.2023.43689

This article has been republished from the following materials. Note: material may have been edited for length and content. For further information, please contact the cited source.