Important Paper: Why almost all ML models for medicine are wrong-and what we need for evidence-based medical AI

data analysis
1

My LinkedIn feed mentioned this recent paper by Italian researchers on the unreliability “AI” being used in medical decision making. Many of the points should be familiar to Data Methods participants, but it is nice to see others sounding the alarm.

Cabitza, F., Jurman, G., Molinari, F., & Bellazzi, R. (2026). Why almost all ML models for medicine are wrong-and what we need for evidence-based medical AI. International Journal of Medical Informatics, 106538. https://www.sciencedirect.com/science/article/pii/S1386505626002789?via%3Dihub

6 Likes
2

This paper really captures the key challenges of (blindly) applying modern ML methods. Good read. One aspect I’d like to underscore is that these challenges do not cease to exist in different fields: they are largely universal across fields so long as they operate in a noisy data/measurement error world.

At this time, there is barely any critique as soon as we move beyond the medical/health space, and I hope this changes.

3 Likes
3

I may be diverging slightly from the main topic, but due to my limited knowledge, I would like to seek the perspectives of others.
AI has recently become a widespread topic of discussion, and I have attended several lectures regarding its application. But the content primarily consisted of impractical generalizations. The core message was that AI improves efficiency, yet no specific practical applications were demonstrated (The only practical examples provided were limited to instructions on how to input keywords into ChatGPT :person_shrugging: ). I believe AI and related technologies are highly valuable in areas such as diagnostic imaging and quantitative measurement. However, in most other contexts, particularly within prediction models, I struggle to identify practical applications. I would be interested in hearing examples of actual AI implementation from others.

1 Like
4

Jiaqi, in your specifically Japanese context you might be interested to appreciate the cultural legacy of the Fifth Generation Computer Systems (FGCS) project, as related by Alan Kay in this quora answer and also the answer by Markus Triska which he refers to.

3 Likes
6

A patient advocate’s perspective: I have no disagreement with the issues raised in the paper and its proposals. It’s important for the public to recognize that the quality of an AI response depends on the quality of the questions we ask and the related details. Similarly, critics of AI in medical decision-making must recognize that trained doctors remain the gatekeepers of medical care, as their oversight is required for any medical intervention. So far, I have found AI responses to be exceedingly helpful for summarizing the current standard of care for “Disease X” and for exploring differential diagnoses for specific sets of symptoms. As such, it is an invaluable resource to help patients ask better-informed questions. It is also enormously superior to previous search methods, which frequently returned “snake oil” promotions in response to medical questions from patients who were already frightened by sources promoting fear of the FDA and for-profit medical providers.

2 Likes
7

A note of caution from a family physician. My colleagues and I are now, routinely, encountering patients who have sought medical advice from “AI” before they come to see us. Often, the patient has experienced a new somatic symptom, wonders if it could be “caused” by one of his medications or a certain underlying disease he might fear that he has, and poses the following question to an AI search engine: “Can (insert symptom of concern) be caused by (insert feared disease or medication)”? Invariably, AI will dredge up an affirmative answer, often identifying anecdotal reports or opinions from very dubious sources to support the patient’s “hypothesis,” no matter how far-fetched or implausible it might be, medically-speaking.

Interactions with patients who have spent a lot of time with AI are exhausting for physicians. Some might argue: “If the physician can’t help the patient understand why he shouldn’t trust the information he’s getting from AI, then maybe the physician’s opinion shouldn’t be viewed as credible…” But this stance ignores the vagaries of human psychology and reasoning. A patient who is already predisposed to believe a particular explanation and seeks “independent” confirmation of his hypothesis is taking a very different approach to diagnosis than a physician who takes a careful history, uses training/context/experience to arrive at a differential diagnosis, and then ranks the plausibility/importance of each potential diagnosis. Once patients have gone down the first path, it becomes nearly impossible to sway them, no matter how solid the argument.

Frankly, many of us are starting to throw in the towel trying to rebut the B.S. that AI is feeding our patients. And some of our patients are making bad decisions about their health as a result.
Physicians are far from perfect. We make mistakes all the time. Over the course of a clinical day, there are an awful lot of physical symptoms that we can’t explain with any degree of certainty. Maybe some of those "unexplained’ symptoms are a function of the physician’t ignorance and someone smarter could have made a confident diagnosis. But good physicians know what they don’t know, respect the unfathomable complexity of human biology, will readily admit uncertainty, and will, over time, become expert at decision-making in the face of that uncertainty. AI never admits ignorance- and this is exactly why it’s so dangerous.

3 Likes
8

I have little doubt that an AI chatbot can provide misleading information - leading to frustration for the patient and their healthcare providers.

Its validity depends on how the patient frames the question and what background we provide it. As an advocate with high standing in clinical research (FDA, NCI, CIRB … experience), I’ve found that its plain language summaries of clinical trials results to be very well done – for any that I’ve asked it to review: an application that clinical researchers should make use of! As always, it’s an advisor, and the user must vet it for accuracy – look at the provided citations, consider it as a basis to aid a conversation with a trained doctor.

I use Gemini Notebook to organize my clinical picture. I find it extremely useful for helping me to organize my clinical background, current meds, supplements, and to understand my labs and other test results, … for the purpose of asking informed questions of my doctors.

I wrote the following for a newsletter I publish in my community:

Chatbot Responses have Two Levels of Reliability

Obviously, knowing whether an AI Chatbot is using a fact-checked source is critical.
See Weighing AI’s Response (below).

Standard AI chatbots can confidently make up false information (called hallucinations).

RAG-regulated AI chatbots provide evidence-based information.

With this enabled (turned on by the nature of your question it seems) the Chatbot must look up the facts in a verified database before it answers you.

So, which is it? Is it Evidence-based or a Hallucination?

Look for the links: Real evidence-based RAG AI will provide specific, clickable source links or footnotes next to its facts. These are called references or citations. Look for live prompts like “Searching [Database Name]” or “Reviewing source documents” before the AI answers.

Be skeptical of “naked” text: If an AI gives you a highly specific medical dose or a complex climate statistic without a single link or citation, assume it is a hallucination until proven otherwise. In fact, no matter the source, providing information without a citation is a red flag warning!

Make sure you have provided accurate background information: Is your diagnosis correct? Have you provided your age, other medical conditions, and a complete list of your medicines and supplements, for example? And how we ask the question can contribute to the reliability of the response, such as by asking for the current standard of care, or best practice, or when considering a treatment decision: What are the risks and benefits for TREATMENT X to treat CONDITION Y, and what is the level of evidence for each?

Probably the best course of action for concerned physicians (and thank you for that!) is to use AI for your own medical circumstance so you can provide informed guidance to patients about its strengths and dangers.

2 Likes
9

More on this important topic:

Here’s what I found on physician guidance for helping patients use AI responsibly for medical information:

I found that major medical organizations (such as the American Medical Association) and leading health systems advise physicians to actively guide patients toward viewing AI as an “augmented intelligence” tool for health literacy and visit preparation, rather than an autonomous diagnostician or substitute for clinical care.

Key themes I noticed:

  1. Framing as “Augmented Intelligence”: Clinicians encourage patients to use AI tools to translate complex medical jargon, organize symptoms, or draft questions for their doctor, while reminding them that AI lacks clinical reasoning, physical examination capabilities, and personal context.
  2. Strict Privacy & Security Guardrails: Physicians should explicitly warn patients never to upload unredacted medical records, lab PDFs, or personal health identifiers (PHI) into consumer AI chatbots, as consumer tools are not HIPAA-compliant and may use entries for model training.
  3. Guarding Against “Hallucinations” & Worst-Case Anxiety: Because general AI search tools synthesize broad internet data—including unverified forums—they can generate convincing errors or surface extreme “worst-case scenarios,” which doctors can help reframe and contextualize.
  4. Prompt Specificity & Source Verification: Doctors can teach patients to ask specific, structured prompts (e.g., requesting plain-language summaries, lists of reputable medical sources, or emergency red-flag warnings) to get higher-quality, safer responses.
  5. Shared Decision-Making: Bringing AI-assisted search summaries or question lists to appointments is welcomed as a way to enhance patient engagement and streamline the clinical visit.
1 Like
10

Thanks, Karl, for your thoughtful response.

Don’t get me wrong- even a Luddite like me can see some valuable niches for AI in medicine. Unfortunately, however, the circumspect, thoughtful approach to AI that you describe in post #7 above (e.g., asking for a summary of a clinical trial) is NOT how most of my patients are using AI. Rather, they are using it to self-diagnose and are coming to me with already-entrenched ideas about the cause(s) of their symptoms.

I don’t believe that it is physicians’ responsibility to launch a mass public education campaign for patients on the responsible use of AI for medical diagnosis and decision-making. We simply have way too many other priorities and can’t afford to spend an hour with every patient, rebutting every anecdote that AI has dredged up for him. And, even if such a campaign could be implemented, I doubt that it would be effective. The attraction of a technology that promises a confident answer to any question we might ask is simply too powerful.

Like most physicians, most patients will not be savvy enough to be put safeguards in place to ensure they are only seeing high quality AI-generated evidence. If a patient presents me with a high quality study and asks for my opinion, I’m more than happy to read it after the appointment is done and provide my opinion- I consider this to be part of my job. But when I’m presented, repeatedly, with huge volumes of poor quality evidence and asked, effectively, to provide a substantive rebuttal to each claim, the clinical interaction becomes exasperating very quickly. Believe me, I’ve tried every possible way to interact with patients who have spent countless hours in various Youtube/AI rabbit holes, who then ask me to refute one crappy/outrageous claim after another. The exchanges are utterly exhausting and eventually I just give up (Brandolini’s Law). The patient will do what he wants to do. All I can do is present what I consider to be high quality evidence and my considered opinion.

The “evidence” that AI provides to patients about potential underlying causes for their symptoms is often (but not always) woefully inadequate for diagnosis, in the hands of a non-medically-trained person. Are there people who will successfully self-diagnose using AI? Undoubtedly, yes. Can AI sometimes do a better job than a bad doctor? Undoubtedly, yes. Than a good doctor? Occasionally, perhaps. But there is an underlying criticism of AI that cuts across disciplines- it’s often only people with extensive training in the discipline who can identify AI’s (sometimes) severe errors and limitations.

Patients are now going to their physician with AI-reinforced ideas about their diagnosis and expecting us to be able to rebut everything ever written on the Internet by anyone on that topic in the span of a 15-minute appointment, evidence quality-be-damned. This would be like a statistically-naive clinician slapping an AI-generated clinical trial SAP down on the table in front of a human applied statistician the night before the SAP needs to be submitted, asking for its “approval” and then arguing about the reason for every revision the human statistician advises. Explaining the “reason” to someone without training would effectively require the statistician to distill 5 years of PhD-level training and 20 years of experience into a 2 minute response- i.e., it’s an impossible request. Now imagine that the statistician had to address multiple such requests each day. This is how physicians are picturing the non-too-distant future, with ever-expanding use of AI by patients. At present, these types of interactions are happening a couple of times per day. But things are only bound to get worse…

4 Likes
11

Time saving is a strength of AI… You might ask an AI chatbot to write plain language recommendations for using AI as an aid in preparing for doctor visits. You can even specify the grade level: AI: please write a draft for me (that I will review) that I can give to my patients about the risks and benefits of using AI for self diagnosis and other matters related to their health and well-being. Please provide a draft that grandma could understand or a person with an 8th grade reading level. Include how to ask questions and verify the answers provided by the AI – such as the level of evidence, the known and unknown risks and benefits. …(If you like, I can ask this for you and post it’s response here … if you think it of general interest to the group?)

1 Like
12

Alright, so I’ll risk it. I asked and here was its response:
Here is a warm, easy-to-read draft designed at an 8th-grade reading level that doctors and clinics can hand out to patients or post on a patient portal.

A Patient’s Guide to Using AI for Your Health

Dear Patient,

Many people today are using computer search tools called Artificial Intelligence (AI)—like ChatGPT, Copilot, or Gemini—to look up health questions and symptoms.

AI can be a helpful assistant, but it is not a doctor. We want you to be safe and well-informed, so here is a quick guide on how to use AI wisely, where it can get confused, and how to double-check what you find.

The Good: How AI Can Help You

  • Explaining Tricky Terms: If you receive lab results filled with complex medical words, AI can rephrase them in simple, plain English.
  • Getting Ready for Your Visit: AI can help you organize your symptoms and draft a clear list of questions to bring to your next doctor appointment.
  • Learning General Facts: It is great for looking up basic information about wellness, diets, or common health conditions.

The Risks: What AI Cannot Do

  • AI Cannot Examine You: AI does not know your personal health history, your family background, or how you actually look and feel today.
  • AI Can Guess Wrong (“Hallucinations”): Sometimes AI sounds completely sure of itself even when it is making a mistake or giving wrong advice.
  • It Might Cause Unnecessary Worry: AI often lists rare or extreme conditions first. A simple muscle pull on your screen can end up looking like a scary emergency!
  • It Cannot Give a Diagnosis: Only a real medical professional can safely diagnose and treat a health condition.

How to Ask AI Good Questions

To get the safest, most helpful answers from AI, try these tips:

  1. Ask for Plain Language:
    • Example: “Please explain what ‘hypertension’ means in simple 8th-grade English.”
  2. Ask About Pros, Cons, and Evidence:
    • Example: “What are the known benefits and risks of this idea? How strong is the medical evidence behind it?”
  3. Ask for Reputable Sources:
    • Example: “What major medical organizations or websites (like Mayo Clinic or the CDC) support this statement?”
  4. Protect Your Privacy:
    • Never type your full name, address, birth date, or medical record numbers into public AI tools.

How to Check (Verify) What the AI Tells You

  • Watch for “Magic Cures”: If the AI says something like “This will definitely cure you” or “You don’t need to see a doctor,” be very careful. Real medicine rarely relies on absolute promises.
  • Check the Evidence: Ask the AI: “Is this based on large medical studies or just general internet forums?” Strong medical advice comes from verified clinical research, not random blogs.
  • Bring Your Notes to Your Doctor: Write down or print out what the AI told you and bring it to your visit. We welcome your questions! Together, we can figure out what actually applies to your body and what you can ignore.

Bottom Line: Think of AI like a digital library catalog—it’s great for gathering information and asking questions, but always count on your healthcare team for real medical care!

:light_bulb: Would you like me to generate a clean, printable PDF handout version of this guide in Studio so you can easily save or print it for clinic use?

3 Likes
13

This is great, Karl! Thanks! Clear communication is vital for effective patient care and I can get on board with everything written here. Based on personal experience, I’m still skeptical about the proportion of AI “misuse” that even these clear, common sense guidelines will prevent. But they certainly can’t hurt and might be valuable for at least some subset of patients.

1 Like
14

Right, misuse of online medical information by the public will not go away. Before AI it was worse, in my view – but in a different way: patients often intentionally fed bad information – the cult of natural alternative medicine. (JRK Jr, and company.) At least AI responses are RAG-regulated - all of it judging from my use of it.

1 Like
15

Erin, here’s the latest draft (image of opening page) providing the bones of the guidance paper for patients. Feel free to make additional suggestions.

Link to shared PDF file online: https://drive.google.com/file/d/1mckgAiVjLFUhHM3Uuz_O-E6fffJxXssu/view?usp=sharing

image
image675×721 106 KB

2 Likes
16

“How AI Can Help (or just worry) Patients” … seems ready for sharing / downloading.

https://drive.google.com/file/d/1mckgAiVjLFUhHM3Uuz_O-E6fffJxXssu/view?usp=sharing

I’ve shared it with my doctor, and a few MDs online. So far, it’s been well received. Copying comments from nurses who meet to discuss advocacy issues where I reside now in “retirement.”:

  • Funny to attribute “hallucinate” to AI since I have only known that word used in reference to humans. But I see it being used now related to AI!

  • True AI does not have the whole picture like we hope our doctors have, however, doctors are spread thin and can miss important symptoms.

  • Doctors have limited time to keep up on the latest research or even the meta-analysis, although we expect that of them. So this is where we can help be our best advocates, if we know how to use AI effectively!

  • I think the statement “ensuring your AI tool is responding in an evidence based model” requires very clear instructions/clarification. It sounds like there is a special appt or turn on click and it will automatically work that way. But then that is what the rest of the presentation goes onto clearly explain!

2 Likes
17

this isn’t true for clinical studies: “Modern science requires researchers to publish raw data online for open verification”, although the regulator has access to the data if it’s relevant for approval. It would be good if they left a brochure like this in the waiting room of the doctors’ surgery. In north america they make you sit in the waiting room a long time

18

YES, the waiting room time is a missed opportunity to provide answers to questions frequently asked by patients. For indolent lymphoma, for example, explaining the rationale for watchful waiting would aid in making the consultation time more productive. Regarding clinical research reporting, it is my understanding that study teams are required to report raw results without interpretation to ClinicalTrials.gov. Although there might not be 100% adherence, and reporting is sometimes not as timely as it should be, the data remains a vital resource.

1 Like
19

“raw data” is normally understood to be patient level data, ie source data. clinicaltrials.gov gets some data summaries. i think your point in the document was that if we have the data we can check the results for ourselves, but only the regulator can do that

20

I see what you mean. I agree that raw data is even more challenging for patients to interpret. I might revise it to: "Modern science requires researchers to publish raw data online for open verification by other scientists in the field (aka peer review).

An aside: I asked AI to summarize the results of a study on ClinicalTrials.gov … to gauge how good/useful it is at that.

Summarize the result of this study for patient understanding in plain language and include level of evidence: ClinicalTrials.gov ..

The link you provided corresponds to the ROSEWOOD clinical trial (NCT03332017). This study evaluated a new treatment combination for adult patients with follicular lymphoma (a type of blood cancer) whose disease had come back (relapsed) or stopped responding to previous treatments (refractory).

STUDY DESIGN & LEVEL OF EVIDENCE

  • Level of Evidence: Level 2 (High-Quality Evidence).
  • Why it ranks here: The ROSEWOOD study is a Phase II randomized controlled trial. In this study, 217 patients were randomly assigned (like a flip of a coin) to two different treatment groups. Because it is randomized and includes a direct comparison group, it provides strong, highly reliable evidence—though large Phase III trials are traditionally considered the absolute highest tier (Level 1).

WHAT THE STUDY COMPARED … (I can provide more if interested)

2 Likes
21

Regulators EMA and HC provide clinical data and study reports proactively publicly, sometimes including individual patient listings in Annexes 16.2 and following of ICH E3 template-using reports of clinical studies:

  • Health Canada (HC) has study NCT03332017 (BGB-3111-212, ROSEWOOD, EudraCT 2017-001552-54) here: https://clinical-information.canada.ca/ci-rc/item/273215, see response indivdual-level data listings from p 2732 onwards in section ‘Summary and Imaging Reports’
  • The European Medicines Agency publishes clinical data including for the medicinal product Brukinsa, but data for study NCT03332017 do not yet seem published; published is however the assessment report of the study, found using my search
1 Like