My LinkedIn feed mentioned this recent paper by Italian researchers on the unreliability “AI” being used in medical decision making. Many of the points should be familiar to Data Methods participants, but it is nice to see others sounding the alarm.
Cabitza, F., Jurman, G., Molinari, F., & Bellazzi, R. (2026). Why almost all ML models for medicine are wrong-and what we need for evidence-based medical AI. International Journal of Medical Informatics, 106538. https://www.sciencedirect.com/science/article/pii/S1386505626002789?via%3Dihub
5 Likes
This paper really captures the key challenges of (blindly) applying modern ML methods. Good read. One aspect I’d like to underscore is that these challenges do not cease to exist in different fields: they are largely universal across fields so long as they operate in a noisy data/measurement error world.
At this time, there is barely any critique as soon as we move beyond the medical/health space, and I hope this changes.
3 Likes
I may be diverging slightly from the main topic, but due to my limited knowledge, I would like to seek the perspectives of others.
AI has recently become a widespread topic of discussion, and I have attended several lectures regarding its application. But the content primarily consisted of impractical generalizations. The core message was that AI improves efficiency, yet no specific practical applications were demonstrated (The only practical examples provided were limited to instructions on how to input keywords into ChatGPT 
). I believe AI and related technologies are highly valuable in areas such as diagnostic imaging and quantitative measurement. However, in most other contexts, particularly within prediction models, I struggle to identify practical applications. I would be interested in hearing examples of actual AI implementation from others.
1 Like
Jiaqi, in your specifically Japanese context you might be interested to appreciate the cultural legacy of the Fifth Generation Computer Systems (FGCS) project, as related by Alan Kay in this quora answer and also the answer by Markus Triska which he refers to.
3 Likes
A patient advocate’s perspective: I have no disagreement with the issues raised in the paper and its proposals. It’s important for the public to recognize that the quality of an AI response depends on the quality of the questions we ask and the related details. Similarly, critics of AI in medical decision-making must recognize that trained doctors remain the gatekeepers of medical care, as their oversight is required for any medical intervention. So far, I have found AI responses to be exceedingly helpful for summarizing the current standard of care for “Disease X” and for exploring differential diagnoses for specific sets of symptoms. As such, it is an invaluable resource to help patients ask better-informed questions. It is also enormously superior to previous search methods, which frequently returned “snake oil” promotions in response to medical questions from patients who were already frightened by sources promoting fear of the FDA and for-profit medical providers.
2 Likes