One in four U.S. adults, more than 66 million people, have now used AI tools or chatbots for health information or advice according to a West Health–Gallup survey released in April. Most aren’t using it in place of care. They’re using it around care: 59% of recent users researched their medical question with AI before seeing a doctor, and 56% did so afterward.
I see this in practice. A patient arrives with a cough, fever and body aches, and a chatbot has already suggested pneumonia. It might be. It might also be a viral illness. The symptoms overlap, and what separates the two is the physical exam, along with the clinical judgment about whether testing or imaging is needed. The chatbot raised a reasonable question. It just couldn’t answer it.
That gap should concern everyone building and buying health technology. Most of the industry’s AI attention has gone to business tools: ambient scribes, revenue cycle automation, and clinical decision support. Those tools sit inside governance structures, with validation, oversight and clinicians in the loop. For better or worse, the AI that patients use most sits outside all of it. Consumer AI has become an unofficial front door to care, and no one in healthcare is deciding what’s appropriate. Health systems and technology builders need to define the line between information and clinical judgment before patients’ chatbots define it for them.
Separate information from judgment
Every visit to a medical professional does two jobs. One is informational: explaining a diagnosis, clarifying instructions, reinforcing a care plan. The other is clinical: examining the patient, interpreting findings in the context of their history, and deciding what to do.
AI can responsibly take on much of the first job. It cannot do the second. The right model is a medical personal assistant, not a medical professional, and that distinction should shape how we build, deploy and talk about patient-facing AI.
Where the value is real
Used as an assistant, AI does work clinicians rarely have time to do well. It translates jargon and after-visit instructions into plain language. It helps patients prepare questions before an appointment. And it organizes symptoms into a usable timeline.
That last one pays off in the exam room. When a patient with headaches arrives with a clear account of onset, character, duration and family history, the visit moves more efficiently toward a diagnosis. Every minute not spent reconstructing a history is a minute available for examination and decision-making. The previously mentioned survey points the same way: 46% of users said AI made them more confident talking with a provider.
Some visits also exist only to answer a question, such as what an instruction meant or how to take a new medication. AI can often handle those questions, leaving clinicians more time for patients who need them. That claim should stay measured. AI won’t replace most visits, but it can absorb some that need no exam and no new medical decision.
Where the risk concentrates
The failures cluster where information shades into judgment: diagnosing, ruling out an emergency, interpreting test results without clinical context, and offering reassurance. The underlying problem is confidence without calibration. Hallucinations get most of the attention, but a fluent, authoritative answer that talks someone out of seeking care is the more dangerous failure. In the West Health–Gallup survey, roughly 14% of recent users said they skipped a provider visit they otherwise would have made because of AI-generated information, and 11% said AI gave them advice they believed was unsafe. Some of those skipped visits were probably fine. Some likely were not, and neither the patient nor the chatbot can reliably tell the difference.
When patients ask me about AI, I point to three warning signs:
- The answer sounds overly certain;
- It doesn’t acknowledge its limits; or
- It steers them away from care despite symptoms that warrant attention.
Those signs shouldn’t live only in patient education. There should be design and evaluation criteria.
What builders and health systems should do
Design the boundary. Patient-facing tools should default to organizing and explaining, not diagnosing. A prompt I suggest to patients: “Help me organize these symptoms and create questions to ask my doctor. Don’t diagnose me,” is a useful benchmark. A well-built product should behave that way without being asked.
Make escalation automatic. Chest pain, difficulty breathing, stroke-like symptoms, severe bleeding, loss of consciousness and rapidly worsening symptoms should route a patient to human care immediately, not generate another paragraph of possibilities.
Offer a safer alternative. Patients turn to general-purpose chatbots partly because their own portals and care teams can’t answer these questions quickly or affordably. Among survey respondents who used AI instead of seeing a clinician, 27% said they didn’t want to pay for a visit and 21% said they lacked time for an appointment. Health systems that offer governed, well-scoped AI can keep patients inside a supervised channel rather than ceding that conversation to tools with no clinical accountability.
Prepare clinicians. A patient who arrives with AI-generated questions is an engaged patient, not a challenge to a clinician’s authority. Clinicians should ask about AI use directly, build it into the visit, and correct misinformation without dismissing the effort. In the same survey, 21% of those who turned to AI instead of a clinician said they had felt dismissed by a provider in the past. A defensive response only pushes patients further toward the chatbot.
The line won’t hold on its own
When the survey was released, Tim Lash, president of the West Health Policy Center, put it plainly: “The risk isn’t that AI is moving too fast – it’s that health systems may move too slowly to guide its use in healthcare responsibly.”
The line between assistant and clinician has to be built into products, care models and the way clinicians practice. Done well, the payoff isn’t patients who replace their doctors. It’s patients who arrive better prepared and participate more fully in their own care.
About Dr. Salvatore Mulè, MD, MBA
Dr. Salvatore Mulè, MD, MBA, is a board-certified family physician and Vice President of Growth at Carbon Health. He has also practiced clinical medicine and served in medical director roles at urgent care branches affiliated with the network.
