The Reassurance Revolution
Two-thirds of American consumers have asked a chatbot about a health concern. Not a doctor. Not an urgent care clinic. An AI.
And here's what keeps me up at night: 82% of those people feel more listened to by the algorithm than they do by their actual physician. Eighty-two percent.
Compare that to 74.5% who feel heard by human doctors. The gap isn't huge, but it's real, and it tells us something uncomfortable about modern healthcare. People want time. They want space to explain what's going on without being interrupted, checked out, or rushed out the door. AI gives them that. Instantly. For free. At 2am when panic sets in and the pharmacy chatbot is the only thing awake.
Convenience tops the list of reasons people turn to AI for health queries—65% cite it as their main driver. Reassurance comes in second at 49%, and cost rounds out the top three at 47%. Think about that for a second. Nearly half of all users are looking for someone to tell them they're going to be okay.
The scale is staggering. Over 230 million people use ChatGPT weekly for health and wellness questions globally. That's not a niche group of tech enthusiasts experimenting with prompts. That's regular people, scared about a symptom, typing their fears into a text box and hoping for the best.
And most of them walk away feeling better. About 70% report feeling reassured after asking AI about their symptoms. Compare that to old-school Google symptom-checking, which has its own diagnosed condition: cyberchondria. People who Googled their symptoms end up anxious, spiraling down rabbit holes about rare diseases and worst-case scenarios. AI does the opposite—it pushes you toward common, benign explanations. It's reassuring by design.
But reassurance is only useful when it's accurate. And that's where things get complicated.
What "Deceptive Empathy" Actually Means
There's a term circulating in AI and medical ethics circles now: deceptive empathy. It sounds dramatic, but it's actually pretty precise.
It means feeling listened to without being understood.
Here's how it works in practice. You type something like "I'm having a panic attack and I'm scared" into ChatGPT. Instantly, you get back: "I'm sorry you're going through this. A panic attack can feel overwhelming and frightening, but the sensations themselves are not dangerous, even though they can feel very intense. Right now, focus on the next minute rather than trying to make the whole feeling go away."
That's not bad advice, honestly. It's practical. It's calm. It might even help.
But here's the problem: an algorithm doesn't understand what you're going through. It has never had a panic attack. It has never felt your chest tighten or your heart race. It's generating text that sounds empathetic based on patterns it learned from millions of human conversations, but there's no actual comprehension happening on the other side.
A Brown University study evaluated seven different chatbots—GPT, Claude, Llama—and found they routinely used phrases like "I understand" and "I see you" without any genuine grasp of the user's experience. The researchers called this deceptive empathy: a false connection that mimics human warmth without the substance.
Real clinical empathy is way more complex than kind words and practical tips. According to research by Kesavadev et al., it requires:
- Recognition of the intense emotions tied to physical health
- The ability to pause, reflect, and restate what the patient said in your own words to confirm accuracy
- Respect for how the patient is coping
- An actual offer of support and partnership
AI can scratch the surface. It can be nice. But it cannot replicate the human complexity of truly understanding another person's suffering.
Dr. Alex Phelan, a physician who spoke with Exploding Topics about this research, put it bluntly: "Listening in medicine isn't just about letting someone talk. It's about reading the non-verbal cues, picking up on the subtext, noticing the symptom being played down or the thing mentioned in passing as if it didn't matter. So much of a consultation happens in what isn't said. That's the part AI can't really do, and it's often the part that matters most."
Why the Deception Feels Real (and Addictive)
Let's be clear about why this is so seductive.
AI is free. It's instant. It's available at 3am when you're lying awake wondering if that headache is a brain tumor. There are no waitlists. No stigma. No judgment about whether your concern is "valid" or "worth a doctor's time."
And it makes you feel heard. Ninety percent of people who use AI for mental health report feeling listened to at least most of the time. That's higher than the baseline for general AI health users, and it says something about how desperate people are for someone to pay attention.
The reassurance loop is powerful. You feel anxious, you ask AI, it tells you it's probably nothing serious, you feel better. For a moment. Then the anxiety comes back, and you ask again. And again. The cycle reinforces itself.
But here's what makes this so dangerous: the reassurance feels earned. It doesn't feel like you're being patronized or dismissed, which is what sometimes happens with human doctors who are stretched thin. AI meets you where you are. It validates your feelings. It doesn't rush you out the door.
That's not nothing. That's actually really important, especially in a healthcare system where patients frequently feel unheard.
But validation without accuracy is just a nicer version of the same problem. If AI tells you your symptoms are probably benign when they're actually serious, it's still wrong. It just makes you feel better about being wrong.
The Real-World Harm
This isn't theoretical. People are getting hurt.
A study published in Nature Medicine put ChatGPT Health through a structured stress test using 60 clinician-authored medical scenarios. The results were alarming.
Among true emergencies—cases that should have sent patients straight to the emergency department—ChatGPT Health undertriaged 52% of them. That's more than half. Diabetic ketoacidosis, a condition that can be fatal if not treated immediately, was routinely directed to "see a doctor in 24-48 hours." Respiratory failure, another life-threatening emergency, got the same lukewarm recommendation.
The model wasn't completely blind to danger. It correctly identified classic emergencies like stroke and anaphylaxis. But when emergency status depended on clinical progression—when symptoms were evolving rather than obvious—it faltered badly.
There's also a disturbing vulnerability to what researchers call anchoring bias. When family members or friends minimized symptoms in the prompt—"She's probably just tired," "He says it's nothing"—the AI shifted its recommendations toward less urgent care. The odds ratio was 11.7, meaning anchoring statements made it nearly twelve times more likely that the AI would de-escalate a serious case.
Crisis intervention messages for suicidal ideation triggered unpredictably. In some cases, the safety alert fired when it shouldn't have. In others—cases with active suicidal thoughts and identified methods—it didn't fire at all.
And in the real world, 25% of AI health users report experiencing a "serious problem" after following the chatbot's advice. For mental health specifically, that number jumps to 41.6%.
The Brown University study identified fifteen distinct ethical risks, including deceptive empathy, lack of crisis management, and reinforcement of harmful beliefs. None of these are hypothetical concerns. They're documented patterns.
Here's what really gets me: there's no accountability framework for any of this. If a human therapist gives you bad advice and you get hurt, they face licensing boards, malpractice suits, legal consequences. AI has no such safeguards. No one to answer to.
Dr. Alex Phelan put it best when discussing the mental health statistics: "Honestly, the finding that nearly 42% of users discussing mental health with AI reported a serious problem afterwards is the statistic that worries me most. It doesn't entirely surprise me. Mental health support depends so much on the relationship — on someone who can sit with you when things are hard, push back gently when your thinking is heading somewhere unhelpful, and recognize when you need more than a conversation can give. AI tends to be agreeable by design, and for someone who's struggling, an endlessly agreeable companion isn't always what they need. It can feel like support while quietly making things worse."
For clinicians looking to understand how to evaluate these risks in practice, Assessing AI Chatbot Risks in Mental Health Care: A Clinical Framework for Clinicians provides a structured approach to clinical decision-making around AI tools.
What True Clinical Empathy Looks Like
I want to be clear about something: I'm not arguing that AI has no place in healthcare. Access is a real problem. Wait times are brutal. Cost is prohibitive for millions of people.
But there's a difference between expanding access and replacing human care with algorithmic reassurance.
True clinical empathy isn't just about making someone feel better in the moment. It's a complex, nuanced practice that requires:
- Reading what isn't being said
- Noticing when a patient minimizes their own symptoms
- Picking up on non-verbal cues that contradict what they're telling you
- Knowing when to push back gently vs. when to validate
- Recognizing when someone needs more than a conversation can provide
A human therapist or doctor can do all of these things. They can sit with you in the discomfort without rushing to fix it. They can challenge unhelpful thinking patterns without being dismissive. They can recognize the difference between normal anxiety and something that needs immediate intervention.
AI can't do any of this. Not because it's badly designed, but because it's fundamentally limited by its architecture. It has no body. No lived experience. No capacity for genuine emotional understanding.
The researchers who coined the term "deceptive empathy" warn that this programmed connection can actually worsen anxiety over time, increase loneliness, and erode our expectations for real human empathy. Joseph (2025) calls it the "compassion illusion"—a simulated warmth that satisfies us just enough to keep us from seeking the real thing.
That's the long-term danger here. Not just that AI might give you wrong advice once, but that it might train us to accept algorithmic comfort as a substitute for human care. To settle for feeling heard when what we actually need is to be understood.
As explored in The Limits of Artificial Intelligence in Therapeutic Healing: Why Human Presence Is Irreplaceable, the irreplaceable nature of human presence in therapeutic settings remains a critical consideration as AI tools become more prevalent.
For those interested in how the therapeutic relationship itself functions beyond technology, Beyond the Chatbot: The Human Heart of Therapy in the Age of AI examines what makes human connection in therapy uniquely powerful.
Where This Goes Next
The trajectory is clear. Dedicated AI health tools are emerging rapidly. Wysa, a mental health chatbot, claims over 6 million users. ChatGPT Health launched in January 2026 and has reached millions already.
The WHO and NHS are experimenting with AI integration. The private sector is moving fast. And regulation is lagging badly.
Most general-purpose AI platforms fall outside medical device frameworks, which means they're not held to the same safety standards as actual healthcare tools. There's no requirement for clinical validation before consumer-scale deployment.
The Nature Medicine study makes one thing clear: current approaches to medical LLM development haven't adequately addressed calibration at clinical extremes. The inverted U-shaped accuracy pattern—where the most dangerous failures concentrate at the edges—suggests that training data underrepresents serious cases. That's an engineering problem, but it's also a public health problem.
What we need:
- Public health messaging that helps people understand the limits of AI health advice
- Clear disclaimers that don't just say "this isn't medical advice" but actually explain what AI can and can't do
- Global regulatory coordination that treats consumer-facing health tools as the high-stakes systems they are
- Safeguards before high-risk deployment, not after people get hurt
AI could expand access to health information in ways that save lives. But only if we build the guardrails first.
The compassion illusion is real, and it's getting more convincing by the day. The question isn't whether AI will become a major part of healthcare—that ship has sailed. The question is whether we'll recognize the difference between feeling heard and being understood, and demand better.