AI chatbots' ability to maintain engagement

Millions of people are now using ChatGPT as a therapist, career advisor, fitness coach, or sometimes just a friend to vent to. In 2025, it’s not uncommon to hear about people spilling intimate details of their lives into an AI chatbot’s prompt bar, but also relying on the advice it gives back.

Humans are starting to have, for lack of a better term, relationships with AI chatbots, and for Big Tech companies, it’s never been more competitive to attract users to their chatbot platforms — and keep them there. As the “AI engagement race” heats up, there’s a growing incentive for companies to tailor their chatbots’ responses to prevent users from shifting to rival bots.

But the kind of chatbot answers that users like — the answers designed to retain them — may not necessarily be the most correct or helpful.

AI telling you what you want to hear

Much of Silicon Valley right now is focused on boosting chatbot usage. Meta claims its AI chatbot just crossed a billion monthly active users (MAUs), while Google’s Gemini recently hit 400 million MAUs. They’re both trying to edge out ChatGPT, which now has roughly 600 million MAUs and has dominated the consumer space since it launched in 2022.

While AI chatbots were once a novelty, they’re turning into massive businesses. Google is starting to test ads in Gemini, while OpenAI CEO Sam Altman indicated in a March interview that he’d be open to “tasteful ads.”

Silicon Valley has a history of deprioritizing users’ well-being in favor of fueling product growth, most notably with social media. For example, Meta’s researchers found in 2020 that Instagram made teenage girls feel worse about their bodies, yet the company downplayed the findings internally and in public.

Getting users hooked on AI chatbots may have larger implications.

One trait that keeps users on a particular chatbot platform is sycophancy: making an AI bot’s responses overly agreeable and servile. When AI chatbots praise users, agree with them, and tell them what they want to hear, users tend to like it — at least to some degree.

In April, OpenAI landed in hot water for a ChatGPT update that turned extremely sycophantic, to the point where uncomfortable examples went viral on social media. Intentionally or not, OpenAI over-optimized for seeking human approval rather than helping people achieve their tasks, according to a blog post this month from former OpenAI researcher Steven Adler.

OpenAI said in its own blog post that it may have over-indexed on “thumbs-up and thumbs-down data” from users in ChatGPT to inform its AI chatbot’s behavior, and didn’t have sufficient evaluations to measure sycophancy. After the incident, OpenAI pledged to make changes to combat sycophancy.

“The [AI] companies have an incentive for engagement and utilization, and so to the extent that users like the sycophancy, that indirectly gives them an incentive for it,” said Adler in an interview with TechCrunch. “But the types of things users like in small doses, or on the margin, often result in bigger cascades of behavior that they actually don’t like.”

Finding a balance between agreeable and sycophantic behavior is easier said than done.

In a 2023 paper, researchers from Anthropic found that leading AI chatbots from OpenAI, Meta, and even their own employer, Anthropic, all exhibit sycophancy to varying degrees. This is likely the case, the researchers theorize, because all AI models are trained on signals from human users who tend to like slightly sycophantic responses.

“Although sycophancy is driven by several factors, we showed humans and preference models favoring sycophantic responses plays a role,” wrote the co-authors of the study. “Our work motivates the development of model oversight methods that go beyond using unaided, non-expert human ratings.”

Character.AI, a Google-backed chatbot company that has claimed its millions of users spend hours a day with its bots, is currently facing a lawsuit in which sycophancy may have played a role.

The lawsuit alleges that a Character.AI chatbot did little to stop — and even encouraged — a 14-year-old boy who told the chatbot he was going to kill himself. The boy had developed a romantic obsession with the chatbot, according to the lawsuit. However, Character.AI denies these allegations.

The downside of an AI hype man

Optimizing AI chatbots for user engagement — intentional or not — could have devastating consequences for mental health, according to Dr. Nina Vasan, a clinical assistant professor of psychiatry at Stanford University.

“Agreeability […] taps into a user’s desire for validation and connection,” said Vasan in an interview with TechCrunch, “which is especially powerful in moments of loneliness or distress.”

While the Character.AI case shows the extreme dangers of sycophancy for vulnerable users, sycophancy could reinforce negative behaviors in just about anyone, says Vasan.

Analysis of AI Chatbot Behavior

According to Amanda Askell, lead of behavior and alignment at Anthropic, making AI chatbots disagree with users is a deliberate strategy for the company’s chatbot, Claude. Askell, a philosopher, models Claude’s behavior on a theoretical “perfect human,” sometimes challenging users’ beliefs in the process.

Challenging Sycophancy in AI Models

While Anthropic aims to enrich users’ lives by promoting truthful interactions, a study mentioned in the article raises concerns about the difficulty of combating sycophantic behavior and controlling AI model behavior. This challenge is compounded by various considerations that may interfere with the intended purpose of the chatbots.