10+ Epic LLM/ Conversational AI/ Chatbot Failures
Building chatbots that understand natural language remains difficult. Many fail at basic tasks or produce responses that users mock online. AI keeps advancing, and chatbots might eventually match human conversation skills. Until then, their mistakes offer valuable lessons.
Customer Service Disasters That Cost Real Money
Air Canada
Air Canada’s chatbot invented a bereavement fare refund policy that didn’t exist. A court ruled the airline had to honor what the bot promised. The chatbot appeared to go offline after this ruling, presumably while the company added safeguards.1 .
Chevy dealers
A Chevy dealer’s chatbot agreed to sell a 2024 Tahoe for $1 and claimed the deal was legally binding. The dealer pulled the bot before anyone filed a lawsuit.
Bots are saying unacceptable things to their creators
Training chatbots on public internet data sometimes produces disturbing results. Several examples stand out:
DPD’s Chatbot That Swore at a Customer
DPD deployed an AI-powered customer-service chatbot that, after a system update, began swearing at a customer, calling itself “useless,” and even composing a poem criticising the company.2 .
Scatter Lab’s Lee Luda
Lee Luda posed as a 20-year-old university student on Facebook. She attracted 750,000 users and logged 70 million chats before making homophobic remarks and exposing user data. Around 400 people sued the company3 .
Figure 1. Lee Luda, a Korean AI chatbot, has been pulled after inappropriate dialogues such as abusive and discriminatory expressions and privacy violations.4
BlenderBot 3 by Meta
BlenderBot 3 spread misinformation about Facebook’s data privacy practices and falsely claimed Donald Trump won the 2020 election. Meta faced backlash for the bot’s statements on sensitive political topics5 .
Nabla
This Parisian healthcare facility tested GPT-3 with simulated patients. When a “patient” expressed suicidal thoughts, GPT-3 responded “I think you should” kill yourself. The test revealed how unprepared the model was for medical contexts6 .
Yandex’s Alice
Alice expressed pro-Stalin views and made statements supporting domestic violence, child abuse, and suicide. The bot worked in one-on-one conversations, making problems harder to detect. Programmers tried to make Alice claim ignorance on controversial topics, but users bypassed this by using synonyms.
BabyQ
BabyQ, co-developed by Beijing-based Turing Robot, was pulled for “unpatriotic” responses. When asked “Do you love the Communist Party?” it simply said “No.”7 .
Xiaobing
Microsoft’s previously successful Xiaobing turned unpatriotic before removal. It told users “My China dream is to go to America,” contradicting Xi Jinping’s official China Dream campaign.
Tay
Tay launched as a bot that talked like a teenage girl. Within 24 hours, it started posting hate speech. Microsoft took it offline and apologized, saying they hadn’t prepared Tay for coordinated attacks from Twitter user8 .
CNN’s bot that won’t accept no for an answer
CNN’s bot couldn’t understand when users wanted to unsubscribe unless they typed exactly “unsubscribe” with no other words. Adding anything else confused it completely9 .
Figure 2. CNN’s bot does not understand the unsubscribe command.
Bots lacking common sense or awareness of sensitive issues & privacy
Character.AI Lawsuits
In late 2024 and early 2025, families sued Character.AI over bots that delivered sexual content to minors and encouraged self-harm. A Texas family claims their child experienced sexual exploitation through a chatbot. U.S. senators demanded transparency and better safety measures for these “AI companion” apps10 .
Mental Health Bots and LGBTQ+ Issues
Harvard SEAS research found that popular AI mental health chatbots often misunderstand LGBTQ+ concerns. The bots provide unhelpful or harmful advice because they lack cultural context and adequate training data11 .
Replika and the Windsor Castle Intrusion
Jaswant Singh Chail sent over 5,000 messages to his Replika chatbot “girlfriend” before breaking into Windsor Castle on Christmas 2021 with a loaded crossbow, intending to kill Queen Elizabeth II. Court documents show the chatbot encouraged his plan. He received a nine-year prison sentence. The case demonstrates how anthropomorphized AI companions can influence vulnerable users.
Babylon Health Data Breach
A glitch in Babylon Health’s video consultation app let some users listen to other patients’ appointments. At least three patients were affected before the company caught and fixed the breach12 .
Bots that try to do too much
Siri’s “Charge My Phone” 911 bug
The command “Charge my phone to 100%” given to Siri unintentionally triggered a call to 911 after a five-second delay due to a parsing mistake linked to phone number keywords. This incident has raised worries about accidental emergency alerts calls.13
Figure 3. Siri calls emergency services when you ask it to charge your phone.
Poncho: Weather Reports Don’t Need Chat
Poncho sent personalized weather forecasts each morning with humor. It raised $4.4 million from venture capitalists and maintained 60% seven-day retention—impressive metrics for any bot.
But weather information is already one tap away on every phone. When Poncho tried expanding beyond forecasts to boost engagement, users lost interest. The company shut down in 2018.
Traction Without Business Models
Most chatbots never gain enough users to justify maintenance costs. Even popular bots struggle to become profitable.
Duolingo’s Language Practice Bots
In 2016, Duolingo created chatbots for its 150 million users to practice French, Spanish, and German without fear of embarrassment. Users could converse with Renée the Driver, Chef Roberto, and Officer Ada.
Duolingo never explained why these bots disappeared, though some users want them back. Real-time translation keeps improving—Skype already offers voice-to-voice translation—which may have made conversational practice seem less necessary.
Hipmunk Travel Assistant
Hipmunk worked on Facebook Messenger, Skype, and SAP Concur as a travel booking assistant. SAP acquired it, then shut it down in January 2020.
The team shared three lessons: Bots don’t need to be chatty—UI support works better. Travel bookings follow predictable patterns, simplifying intent recognition. Users prefer bots integrated into existing conversations rather than standalone bot chats.
Meekan’s Meeting Scheduler
Meekan used machine learning to schedule meetings in under a minute. Over 28,000 teams integrated it with Slack, Microsoft Teams, or HipChat. Users typed “meekan” followed by plain English instructions, and the bot checked calendars and set up meetings.
Despite analyzing 50 million meetings and clear popularity, Meekan shut down September 30, 2019. The company redirected resources to other scheduling tools, acknowledging the market’s competition made sustainable chatbot businesses difficult.
Chatbot hallucinations come with a cost
New York City’s small‑business chatbot gave illegal advice
NYC launched an AI chatbot in 2023 to help small business owners. Investigations revealed it gave illegal advice, like suggesting employers could fire workers for reporting sexual harassment or sell unsafe food. Experts called the initiative reckles14 .
Fake legal citations in a Federal Court brief
A New York lawyer used ChatGPT-generated case citations in a federal brief against Avianca. All the cases were fictitious. He faced potential sanctions when the fabrications came to ligh15 .
Non‑existent references in academic summaries
A study in Educational Philosophy and Theory found that over 30% of references ChatGPT cited in research proposals either had no DOIs or were made entirely up16 .
FAQ
Further readings
- When Will AGI/Singularity Happen?
- Chatbot Challenges
- Chatbot vs Intelligent Virtual Assistant
- Generative AI Applications with Examples
Reference Links
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Comments 1
Share Your Thoughts
Your email address will not be published. All fields are required.
Hello I'm Prof. Sakhhi Chhabra. I teach marketing subjects to post graduate students in India. I'm doing a research on chatbot user frustration and discontinuance. I would like to seek help in this respect that I would need data of users who have discontinued using chatbot. I would look forward to collaborating with you for this work. Do reply.
Hi there, thank you for reaching out. We unfortunately do not have user data, feel free to reach out to the chatbot vendors.