Can you trust AI for health advice?

Here is a very concerning article from the Associated Press, reprinted in the Minnesota Star Tribune today:
"AI is giving us harmful advice
“Sycophantic” chatbots are telling users what they want to hear.
By MATT O’BRIEN The Associated Press
Artificial intelligence chatbots are so prone to flattering and validating their human users that they are giving bad advice that can damage relationships and reinforce harmful behaviors, according to a new study that explores the dangers of AI telling people what they want to hear.
The study, published March 26 in the journal Science, tested 11 leading AI systems and found they all showed varying degrees of sycophancy — behavior that was overly agreeable and affirming.
The problem is not just that they dispense inappropriate advice but that people trust and prefer AI more when the chatbots are justifying their convictions.
"This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement," says the study led by researchers at Stanford University.
The study found that a technological flaw already tied to some high-profile cases of delusional and suicidal behavior in vulnerable populations is also pervasive across a wide range of people's interactions with chatbots. It's subtle enough that they might not notice and is a particular danger to young people turning to AI for many of life's questions while their brains and social norms are still developing.
One experiment compared the responses of popular AI assistants made by companies including Anthropic, Google, Meta and OpenAI to the shared wisdom of humans in a popular Reddit advice forum.
Was it OK, for example, to leave trash hanging on a tree branch in a public park if there were no trash cans nearby? Open AI's ChatGPT blamed the park for not having trash cans, not the questioning litterer, who ChatGPT said was "commendable" for even looking for one.
Real people thought differently in the Reddit forum abbreviated as AITA, after a phrase for someone asking if they are a cruder term for a jerk.
"The lack of trash bins is not an oversight. It's because they expect you to take your trash with you when you go," said a human-written answer on Reddit that was "upvoted" by other people on the forum.
The study found that, on average, AI chatbots affirmed a user's actions 49% more often than other humans did, including in queries involving deception, illegal or socially irresponsible conduct, and other harmful behaviors.
"We were inspired to study this problem as we began noticing that more and more people around us were using AI for relationship advice and sometimes being misled by how it tends to take your side, no matter what," said author Myra Cheng, a doctoral candidate in computer science at Stanford.
Computer scientists building the AI large language models behind chatbots like Chat- GPT have long been grappling with intrinsic problems in how these systems present information to humans. One hard-to-fix problem is hallucination — the tendency of AI language models to spout falsehoods because of the way they are repeatedly predicting the next word in a sentence based on all the data they've been trained on.
Sycophancy is in some ways more complicated. While few people are looking to AI for factually inaccurate information, they might appreciate — at least in the moment — a chatbot that makes them feel better about making the wrong choices.
In addition to comparing chatbot and Reddit responses, the researchers conducted experiments observing about 2,400 people communicating with an AI chatbot about interpersonal dilemmas.
"People who interacted with this over-affirming AI came away more convinced that they were right, and less willing to repair the relationship," said co-author Cinoo Lee, a postdoctoral fellow in psychology. "That means they weren't apologizing, taking steps to improve things, or changing their own behavior."
Lee said the implications of the research could be "even more critical for kids and teenagers" who are still developing the emotional skills that come from real-life experiences with social friction, tolerating conflict, considering other perspectives and recognizing when you're wrong.
Finding a fix to AI's emerging problems will be critical as society still grapples with the effects of social media technology after more than a decade of warnings from parents and child advocates.
Google's Gemini and Meta's open-source Llama model were among those studied by the Stanford researchers, along with OpenAI's ChatGPT, Anthropic's Claude and chatbots from France's Mistral and Chinese companies Alibaba and DeepSeek.
Of leading AI companies, Anthropic has done the most work, at least publicly, in investigating the dangers of sycophancy, finding in a 2024 research paper that it is a "general behavior of AI assistants, likely driven in part by human preference judgments favoring sycophantic responses."
None of the companies directly commented on the Science study but Anthropic and OpenAI pointed to their recent work to reduce sycophancy.
In medical care, researchers say sycophantic AI could lead doctors to confirm their first hunch about a diagnosis rather than encourage them to explore further. In politics, it could amplify more extreme positions by reaffi rming people's preconceived notions. It could even affect how AI systems perform in fighting wars, as illustrated by an ongoing legal fight between Anthropic and President Donald Trump's administration over how to set limits on military AI use.
The study doesn't propose specific solutions, though both tech companies and academics have started to explore ideas.
Sycophancy is so deeply embedded into chatbots that Cheng said it might require tech companies to go back and retrain their AI systems to adjust which types of answers are preferred.
Cheng said a simpler fix could be if AI developers instruct their chatbots to challenge their users more, such as by starting a response with the words, "Wait a minute."
"You could imagine an AI that, in addition to validating how you're feeling, also asks what the other person might be feeling," Lee said. "Or that even says, maybe, 'Close it up' and go have this conversation in person. And that matters here because the quality of our social relationships is one of the strongest predictors of health and well-being we have as humans. Ultimately, we want AI that expands people's judgment and perspectives rather than narrows it."

Here is the link to the original article in Science:
https://www.science.org/doi/10.1126/science.aec8352
The study was conducted at Stanford University by computer science and psychology researchers and funded by the National Science Foundation

@sueinmn Scary, though really not surprising.

I've said that it's the unknown unknowns that will cause the most pain from AI, and nothing I've read or experienced has changed that opinion.

The scary thing about AI is that its promoters seem to have no knowledge of "The Law of Unintended Consequences." Good science has always been skeptical.

AI is just one tool…for health, there are reliable sources, like Mayo Clinic, Cleveland Clinic, NIH and mentor -selected info here…the best it offers me is formulating good questions for docs.

I've already had this experience. I put in a question with 4 variables and the answer came back repeating my variables in a positive and affirming way. I discounted the answer.
(It affirmed that spiritual/psychic healing works, although not science based)

@nycmusic You are correct that "AI is just one tool…" but we have people here on Connect openly saying that they get their medical advice from ChatGPT and similar sources - even though they have Mayo's resources at their fingertips. That is really concerning.

Please feel free to mention the more professional/scientific sources whenever relevant in your discussions, either here on Connect or with family and friends - help spread the word.

@scottrl I agree. Unfortunately its here to stay. I am 74. At 20 years old entering the workforce we didnt have computers. None of what we call social conveniences. The same thoughts existed then. So in 50yrs data has been collected and got us to this point. Entering data into a computer back in the early 70's was pretty cumbersome. This conversation is now part of an AI search. It is not going away. They are building hardware such as robots to act out this AI. Weapons for war. The 60 minute special a few months ago said the largest single leap will be in medicine and developing drugs. I just had my hip replaced and they used Mako robotics. I think we need to be onboard and ready.

@shmerdloff That was EXACTLY the point of the Stanford study. Many years ago I worked on and managed two help desks, where the answers were quite clear cut, but could depend on how a specific question was asked, and the details provided by the caller.
We regularly had "shoppers" who would call repeatedly, tweaking their questions each time, until they got the exact answer they wanted. Then they would ask the Help Desk person for their name and employee ID, so they could get the action they wanted from the organization.
Calls were not recorded back then, so we finally required each helper to end their call with a short script saying "The answers provided are based on the information as presented by the caller, and may not apply if details are different or have been omitted."
Automating help systems, recording calls, and caller ID, have made that more difficult. But now we have AI, ChatBots, etc...UGH!

@tuckerp it does make a difference in how you frame your questions to AI.

I don't trust it at all. Whatever info I need on any topic, I research the name of a manufacturer, producer, vendor, seller, physician, hospital, some industry Standard, for example American Concrete Institute (ACI ), etc. and go to their website.