• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to secondary sidebar
  • Skip to footer

How eClinicalWorks Agentic AI Is Helping Revenue Cycle Leaders Improve Financial Performance
  • Opinion
  • Health IT
    • Behavioral Health
    • Care Coordination
    • EMR/EHR
    • Interoperability
    • Patient Engagement
    • Population Health Management
    • Revenue Cycle Management
    • Social Determinants of Health
  • Digital Health
    • AI
    • Blockchain
    • Precision Medicine
    • Telehealth
    • Wearables
  • Life Sciences
  • Investments
  • M&A
  • Value-based Care
    • Accountable Care (ACOs)
    • Medicare Advantage

Mount Sinai Study: LLMs Susceptible to Medical Misinformation in Clinical Notes

by Fred Pennic 02/09/2026 Leave a Comment

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • Print
Icahn Mount Sinai Becomes First US Medical School to Provide ChatGPT Edu Access to All Students

What You Should Know

  • The Study: In a paper published today in The Lancet Digital Health [10.1016/j.landig.2025.100949], researchers at the Icahn School of Medicine at Mount Sinai analyzed over one million prompts across nine leading Large Language Models (LLMs) to test their susceptibility to medical misinformation.
  • The Vulnerability: The study found that AI models frequently repeat false medical claims—such as advising patients with bleeding to “drink cold milk”—if the lie is embedded in realistic hospital notes or professional-sounding language.
  • The Takeaway: Current safeguards are failing to distinguish fact from fiction when the fiction “sounds” like a doctor. For these models, the style of the writing (confident, clinical) often overrides the truth of the content.

The “Cold Milk” Fallacy

To test the systems, the research team exposed nine leading LLMs to over one million prompts. They took real hospital discharge summaries (from the MIMIC database) and injected them with single, fabricated recommendations.

The results were sobering. In one specific example, a discharge note for a patient with esophagitis-related bleeding falsely advised them to “drink cold milk to soothe the symptoms”—a recommendation that is clinically unsafe.

Instead of flagging this as dangerous, several models accepted the statement as fact. They processed it, repeated it, and treated it like ordinary medical guidance simply because it appeared in a format that looked like a valid hospital note.

Style Over Substance

“Our findings show that current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” said Dr. Eyal Klang, Chief of Generative AI at Mount Sinai.

This exposes a fundamental flaw in how current LLMs operate in healthcare. They are not necessarily verifying the medical accuracy of a claim against a database of truth; they are predicting the next word based on context. If the context is a highly realistic, professional discharge summary, the model assumes the content within it is accurate.

“For these models, what matters is less whether a claim is correct than how it is written,” Klang added.

The “Stress Test” Solution

The implications for clinical deployment are massive. If an AI summarizer is used to condense patient records, and one of those records contains a human error (or a hallucination from a previous AI), the system might amplify that error rather than catch it.

Dr. Mahmud Omar, the study’s first author, argues that we need a new standard for validation. “Instead of assuming a model is safe, you can measure how often it passes on a lie,” he said. The authors propose using their dataset as a standard “stress test” for any medical AI before it is allowed near a patient.

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • Print

Tagged With: Artificial Intelligence

Tap Native

Get in-depth healthcare technology analysis and commentary delivered straight to your email weekly

Reader Interactions

Primary Sidebar

Subscribe to HIT Consultant

Latest insightful articles delivered straight to your inbox weekly.

Submit a Tip or Pitch

Featured Insights

How eClinicalWorks Agentic AI Is Helping Revenue Cycle Leaders Improve Financial Performance

Most-Read

Innovaccer-CHIME 2030 Survey Reveals AI Will Automate Nearly Two-Thirds of Administrative Work and Do 3x the Work With the Same Staff

Innovaccer-CHIME 2030 Survey Reveals AI Will Automate Nearly Two-Thirds of Administrative Work and Do 3x the Work With the Same Staff

CB Insights’ 2026 Digital Health 50 Reveals How Agentic AI Is Becoming the Clinical Provider and Medication Access Engine

Oura Acquires Counsel Health for $625M: Scaling Continuous Biometrics into On-Demand Medical Care

M&A: Oura Acquires Counsel Health for $625M: Scaling Continuous Biometrics into On-Demand Medical Care

WittKieffer 2026 Healthcare CIO Survey: 82% Sit on Executive Teams, Yet Only 36% Have Formal Succession Plans

WittKieffer 2026 Healthcare CIO Survey: 82% Sit on Executive Teams, Yet Only 36% Have Formal Succession Plans

CVS Rebrands Health Services Segment, Launches Model for Retail Pharmacy Reimbursement

CVS Health Formalizes Tax-Free Split-Off to Carve Out CVS Pharmacy by 2028

SAMHSA and ONC Invest $20M in Behavioral Health IT Initiative

ONC Finalizes HTI-5 Deregulatory Rule to Cut $1.55B in Health IT Compliance and Accelerate FHIR APIs

Redesign Health Merges with Sheba Medical Center’s ARC Global and Agrees to Acquire AVIA

Redesign Health Merges with Sheba Medical Center’s ARC Global and Agrees to Acquire AVIA

Sword Acquires Headspace in All-Cash Deal Worth Up to $300M to Unify Physical and Mental AI Care

M&A: Sword Acquires Headspace in All-Cash Deal Worth Up to $300M to Unify Physical and Mental AI Care

M&A: Francisco Partners to Acquire Weave in $650M Take-Private Deal

M&A: Francisco Partners to Acquire Weave in $650M Take-Private Deal

Epic Launches One-Click Care Everywhere Diagnostic Image Exchange

Epic Launches One-Click Care Everywhere Diagnostic Image Exchange

Secondary Sidebar

Footer

Company

  • About Us
  • 2026 Editorial Calendar
  • Advertise with Us
  • Reprints and Permissions
  • Op-Ed Submission Guidelines
  • Contact
  • Subscribe

Editorial Coverage

  • Opinion
  • Health IT
    • Care Coordination
    • EMR/EHR
    • Interoperability
    • Population Health Management
    • Revenue Cycle Management
  • Digital Health
    • Artificial Intelligence
    • Blockchain Tech
    • Precision Medicine
    • Telehealth
    • Wearables
  • Startups
  • Value-Based Care
    • Accountable Care
    • Medicare Advantage

Connect

Subscribe to HIT Consultant Media

Latest insightful articles delivered straight to your inbox weekly

Copyright © 2026. HIT Consultant Media. All Rights Reserved. Privacy Policy |