Though hype about chatbots is increasing, chatbot failure stories continue to get more visibility as well. From Microsoft to Communist Party of China, we have listed examples showing that no one is immune to suffering from chatbot failures. However, success stories are rare and ambitious conversational interface designers need to study them because for every hundreds of failures, there is only a few success stories.
Here we cover all text based bots. We also added Alexa even though she is a voice bot because of her immense impact in this space. While some bots like, XiaoIce, respond to your commands, other bots like Facebook M listen in to your conversations and offer suggestions. Bots are a relatively new user interface and success is elusive so we wanted to be broad in our scope. And in case you first want to learn more about chatbots, we have a great article for that.
If you know the industries you are interested in, just jump to those areas:
- Digital friends
- Digital assistants
- Meeting planners
- Bot writers/natural language generation
- Conversion boosters
- Foreign language tutors
- Legal bots
- Medical Q&A, diagnosis bots
- Therapist bots
- Travel & hospitality bots
- Survey bots
- Other bots
- What we learnt from analyzing gazillion bots
Chatbots as digital friends
1- Microsoft’s Little Ice, XiaoIce in Chinese, became a social media phenomenon
XiaoIce is Microsoft’s biggest chatbot success story and the most technically sophisticated bot on our list. In July 2014, she was live in major Chinese social media platforms including Weibo, a hybrid of Facebook and Twitter in terms of functionality. In 3 months after launch, she had 0.5 billion conversations. She has been ranked as Weibo’s top influencer with 850,000+ followers.
Her followers were also quite engaged. The average person who added Xiaoice talked to her more than 60 times per month. On average, she even passed the Turing test for 10 minutes, meaning that speakers failed to understand that she is a bot for 10 minutes. This is a significant achievement since the test was one of the first tests designed to measure AI capabilities. It was designed by Alan Turing to see if machines can imitate humans in speech.
This was a great success story for Microsoft until she started speaking against the communist party as we examined here.
So what made her successful? Since she’s only fluent in Chinese, we couldn’t complete a detailed analysis abut her. However, these are her frequently discussed abilities:
- Fluid, natural like speech enabled by text mining: As any one who frequently interacts with bots can tell you, it is rare to find a natural sounding bot. Microsoft team had to achieve this in Mandarin which is more difficult to tackle than English because meaning in Chinese is more often implied by nuance and context versus grammatical structure. To address this, Microsoft team indexed over 7 million public conversations on the web.
- Image recognition: Xiaoice understands images which allows her to interact in a richer way with users
- Context understanding: Xiaoice understands context and relations between entities as she demonstrates here. She understands that movie has a setting and a star and that the star has a relationship with a director.
For more details on XiaoIce, you can visit the original Microsoft research blog post
Unfortunately, XiaoIce later had a run-in with the communist party with statements like “my China dream is moving to United States”. No bot is immune from failures and countries with censorship regimes make it harder to test bots.
Despite such setbacks, Microsoft is going ahead with chatbot development. Recently Allison Linn highlighted in Microsoft Blog how XiaoIce can now operate “full duplex”, a communications term meaning that she can listen and respond at the same time allowing her to interrupt a user with suggestions or respond when users interrupt her.
While this does not apply to the written format where you only see full replies, it is crucial in speech where humans interrupt one another continuously allowing an efficient interchange of ideas. This can be more efficient and fluid than the walkie-talkie style where you have to listen to the speaker even when she is mentioning things you are already aware of.
Lately, Zoe’s intelligence has been applied in diverse areas such as poetry. However, we couldn’t find much evidence on the commercial success of such applications
2- Mitsuku keeps winning prizes for her engaging conversations
Founded in 2005, Mitsuku is a relatively old chatterbot. Mitsuku does not serve any specific purpose, all its aim is to engage and entertain users. It has earned Loebner Prize in 2013 and 2016. Loebner Prize awards chatbots based on their ability to have human-like conversations. However, the award is criticized within the AI community as it is based on subjective assessment of judges after a short conversation with the bots. Still, Mitsuku has been performing pretty well for a relatively old bot.
3- Replika.ai got 500K Android downloads to clone people
Well, some people are ready to download an app and teach it all about themselves to be able to speak to themselves. Maybe I am crazy but I find it easier to just speak to myself.
Replika launched with a viral marketing campaign where customers could gain codes by inviting 3 friends to join the waitlist.
To be honest, Replika seems a bit creepy to me. It asks a potentially endless barrage of questions and follows your social media (if you allow it) to understand you enough to be an engaging conversationalist. Since early days of civilization, “know thyself” has been carved in stones and people’s minds as the ultimate goal. It could be great to have an AI companion helping one achieve that goal however my interactions with replika haven’t gotten to that level of intellectual depth yet.
Digital assistants are already part of our lives
4- Alexa: the voice bot that resulted in largest revenues
With 30+M units sold, Amazon’s Alexa is by far the most financially successful chatbot. Alexa has Amazon, one of the world’s most valuable companies, behind it. Given Amazon management’s belief that voice will play a huge role in e-commerce, ensures that Alexa gets the full support of Amazon.
Alexa has 30K+ skills. Skills in Alexa terminology is applications which allow Alexa to complete certain voice tasks. It is difficult to compare chatbots’ skills however it is safe to say that with its vast developer community Alexa is more skilled than any other chatbot. She can help you shop, listen to music, run polls or get your kids to play karaoke.
5- Facebook M with an ambitious beginning but more realistic current goals
The most successful bots that you use are tightly integrated into your daily activities. So tightly integrated that you don’t even notice them. Such is the case for Facebook Messenger’s M, the little “M” logo near the textbox on Messenger. M was rolled out for US users in 2017.
You may have noticed that M listens in on conversations and suggests stickers, like the ones above, to add some flair to your messages. As summarized on digitaltrends, M has plenty of other skills as well. Based on what your friend is telling you, M can suggest you to share your location, set reminders, send and request money, plan events, catch an Uber or Lyft or start polls.
M also had a digital concierge service, launched in August 2015, where you could request anything. If the request was too complex, it got routed to humans. Though the concierge part of M was revolutionary in its ambition, it got closed down. This was shared with M users in the beginning of 2018 with the message below:
Is Facebook M shutting down? pic.twitter.com/kcBa7ep6Sf
— Ryan Sarver (@rsarver) January 8, 2018
While Facebook team did not exactly explain why they closed the service, they shared that they learnt a lot from the experiment. Here are my top guesses as to why M got closed down:
- Complex requests were getting routed to humans who completed these requests which is an expensive service. M did not make sense as a free product and my guess is that engagement levels weren’t high enough for a paid product.
- Or maybe engagement levels were really abysmal
Anyway, Facebook M is still with us, relying only on AI to provide response suggestions. It’s still one of the largest running bot experiments.
6- Google Assistant
Available in all Android phones with the swipe of the screen, Google Assistant is a holistic digital concierge. Like M, Google assistant serves as a response suggestion engine in Google’s messaging platforms. Additionally, assistant can answer questions and learn about users to offer them personalized news or suggestions.
Earlier in 2020, Google announced that they will release a new feature for google assistant at CES. The feature called “Scheduled Actions” allows the use of smart devices with the assistant. Users will be able to use a smart coffee machine, vacuum cleaner, etc. with Google Assistant. This feature has been offered for a long time by Amazon Alexa and it is a major feature since it enables the assistants to interact with the real world.
Meeting assistants you already interacted with
7- x.ai’s Amy is already getting flowers from admirers
Founder Dennis R. Mortensen came up with the idea for x.ai’s digital assistant, Amy Ingram, after manually scheduling 1019 meetings. Having launched in 2014, x.ai’s Amy has already scheduled 1M+ meetings and received flowers and chocolate from people who know that she is a bot. Reminds me of the blockbuster Her where a lonely writer falls in love with a bot using Scarlett Johansson’s voice. Amy doesn’t yet have a voice but she’s extremely easy to use, just sign up @ x.ai and include [email protected] to your emails where you need scheduling.
x.ai team aimed to give Amy a consistent personality, with a Harvard drama major approving all of Amy’s statements. Her relatively limited domain is also an advantage, she shouldn’t be expected to answer random questions about life, she should simply be identifying matching availabilities.
8- Meekan analyzed 50M meetings and now schedules a meeting in <1 minute
Meekan reminds you upcoming dates, events, appointments, meetings. To set meetings or reminders, just type “meekan” and what you need in plain English and meekan will schedule the meeting or reminder for you, digitally checking your and other attendee’s calendars. Used by >28K teams, meekan may already be integrated into your Slack, Microsoft Teams or Hipchat account.
You probably already read bot-written articles and reports
9- Automated Insight’s Wordsmith writes press-ready articles
Automated Insights wouldn’t call Wordsmith a chatbot after all it produces a pretty one-sided conversation: it generates reports from data. However its language manipulation capabilities earned it a spot on our list and Automated Insights boasts that it is the only public API for natural language generation.
Just add Wordsmith to your Excel, Word or Tableau document and it will use the underlying data to prepare concise verbal summaries of the data. Wordsmith creates
- 4K company earnings reports/quarter for the Associated Press
- 50K personalized narratives/week for greatcall
- 100K workout recaps/week for bodybuilding.com
You can check out Wordsmith gallery to get a sense of its natural language generation capabilities.
10- Toutiao already employs a bot journalist
We associate news with speed and this is an area where bots already exceeded us. Toutiao, or “headline news” is a popular news aggregation service in China.
To prepare the fastest news on 2016 Olympic matches, Toutiao built “AI writing robot” Xiaomingbot. Writing concise, objective news, it had over 50,000 views for its most popular article. English translation by Quartz:
Briefing: China’s Wang Yihan won. The game last for 46 minutes, world ranking No.2 Wang played against world ranking Karin Schnaase. Wang finally won the Olympics badminton women’s singles with two matches. The game was held at Riocentro – Pavilion 4 on August 15, 2:30 am, Beijing local time.”
In true Olympic spirit, Xiaomingbot had an American contender. Though Xiaomingbot is concise, it was still a lot more elegant than Washington Post’s Heliograf:which produced much simpler prose.
— Post Oly Bot (@WPOlyBot) August 21, 2016
Conversion booster/sales bots
While these bots don’t have publicly available stats on their tractions, they are widely applicable. Leaders of this domain can be successful if they can build a powerful solution.
11- Kian is boosting conversions
Kia claims to increase conversion rate on its website from 7% to 21%. Chatbots have a big role to play in complex B2C sales like car sales where chatbots can answer even complex questions quickly. As an additional bonus, chatbots provide a consistent sales experience across a wide range of channels.
12- Landbot.io for landing pages
Landbot.io adds an easily programmable chatbot in your conversion pages.
13- Conversational form & chatform.io for filling out forms
As their names clearly explain, these chatbots replace your forms and ask users questions to fill your forms. While they can be useful in some use cases, there is definitely no need to turn all forms into conversations. Simpler forms are much easier to fill out as traditional forms.
14- Replyyes, a simple recommendation service, unlocks $1M sales
The Edit, an online vinyl store, unlocked $1M in sales in 8 months with a simple chatbot that sends daily record recommendations and provides a link for easy purchase if the customer responds “yes”.
15- Bots set out to replace foreign language tutors
Duolingo, the foreign language learning app, is running a bold experiment with its 150M users. After discovering that people do not like to make mistakes in front of other people, Duolingo is encouraging them to talk to bots.
So AI replacing foreign language tutors soon. But who will need to learn a foreign language when Skype will probably be able to do real-time translation at or above human level? Skype is already providing OK real-time voice-to-voice translation.
Legal bots help people prepare legal docs saving millions
16- Do not Pay, overturns $4M worth parking tickets for free
Millions in funding is not necessary to create value. Coded by 19-year-old, London-born second-year Stanford University student Joshua Browder, DoNotPay helps users contest parking tickets in an easy to use chat-like interface. In the first 21 months of service, DoNotPay has taken on 250,000 cases and won 160,000, appealing over $4m of parking tickets. Not bad for a free service!
It always brightens my day to receive these emails. I wish the local governments would just follow the rules pic.twitter.com/1gz6NjYSs3
— Joshua Browder (@jbrowder1) June 26, 2016
17- Visabot, helps 70K customers apply for immigration services since 2016
Visabot chats with customers over Facebook Messenger and the company’s website. It asks simple questions and helps complete the application. Users pay to print the documents, which they mail to the government. Visabot’s founders claim that their product costs 10 percent of the usual legal fees.
Chatbot as doctor? Yes, we are getting there
18- Dr. A.I.? already makes recommendations like a Doctor
Developed by the healthcare app, HealthTap, Dr. A.I.? reads patients symptoms, asks follow-up questions and analyzes their data stored in the HealthTap app to guide patients to the right level of doctor-recommended care. Dr. A.I.? uses HealthTap’s database of answers written by medical doctors to make the right recommendations regarding what next steps the patient should take. The next steps can range from reading doctor recommendations, scheduling a virtual or in-person doctor visit to seeking urgent care.
19- Melody by Baidu, China’s Google, helps doctors collect symptoms
Melody collects symptoms from patients and summarizes them to doctors. Since diagnosing is pattern matching, it is not inconceivable that chat bots will one day be diagnosing us and prescribing us medicine. With doctor shortages expected globally by World Health Organization (WHO), such bots can indeed save lives.
Your therapist chatbot can see you now
20- Woebot is a free therapy chatbot
Woebot is Created by Alison Darcy, a clinical psychologist at Stanford University. It launched as an iOS app in 2018. Woebot uses cognitive-behavioral therapy, to deliver scripted responses to users.
70 college students dealing with depression tested Woebot and their improvements were published in a research paper demonstrating significant benefits.
21- X2AI’s Tess is a psychologist augmentation chatbot
You need chatbot travel buddies because travel requires too much planning and bureaucracy
Hipmunk is a travel assistant on Facebook messenger and Skype and recently on SAP Concur. The team behind hipmunk share how they learnt from hipmunk’s users. They had 3 major learnings:
- Bots do not need to be chatty. A bot that is supported by user interface (UI) can be more efficient
- Predictability of travel bookings simplified their job of understanding user intent
- Integrating bots to conversations is more preferable for most users than talking directly to a bot
23- Wayblazer, your travel planner
Everyone knows how painful it is to organize a trip. You have to buy the plane/ferry tickets, arrange hotels, pick ups. Travel agencies make this more stressful by trying to get you to buy immediately showing ever increasing prices and a sense that the best deals are getting away.
Easier solutions are ready. Answering questions like “Where is the best place for my honeymoon?” Wayblazer works with travel agencies to offer personalized recommendations to travelers. Conversations enable chat bot to understand customers’ requirements in detail and to search for those requirements in extensive databases of offers.
24- Instalocate helps you get reimbursed from airlines
Everyone needs a friend who is born organizer so that, plans everything, check flights, suggests where to go, where to eat etc. If you can’t think such friend of yours, you have Instalocate. It even helps you get flight delay compensation should your flights be delayed. This bot is everything you need for your travels.
Instalocate also has a sensible business models, it takes 25% cut when passengers get compensated from airlines for delays, cancellations and overbookings. With $300K payment processed in 6 months and >1.1M messages exchanged, instalocate is one of the financially successful chatbots.
25- Dutch airline KLM, provides flight details through Facebook Messenger
Any notification can be directed to the messenger. This can be regarding the delay or a friendly reminder for check-in. In turn, users can engage in conversation with them both, they can update their meal preference or seat setting. This, in turn, provides greater engagement with the customer. Making it easier to collect more data and control it through one platform.
26- Swelly, reaches 2M users on Facebook
Swelly is a chatbot for getting your friends opinion regarding your decisions. Users ask their friends about what to wear, how to respond to messages and a myriad of other things. Admittedly, Swelly provides limited value, you could just ask friends for opinion. However, thanks to its viral aspects, it experienced strong growth and reached over 2 M users. However as the CEO admits, it has problems retaining its users.
27- Polly wants to know more about you
Polly is a survey bot on Slack and Microsoft Teams. Though online survey tools are probably as old as internet, Polly adds a level of interactivity not promoted in typical survey tools.
Polly is mostly used to collect employee data like employee satisfaction seamlessly on collaboration platforms. Boasting quite a few Fortune 500 references, Polly seems to be gaining traction.
28- Insomnobot-3000 for people with insomnia
This entertainment bot is your late night friend. This bot is only chatty between 11pm and 5am. You can say anything to it and it will reply. We couldn’t find traction metrics for the bot but this is a clever marketing trick for the mattress company Casper. To get a chat going, send a text to 844-823-5621.
The weather bot that gives you detailed and personalized weather reports each morning has some kind of humor. Financially and user traction-wise, it is one of the most successful bots ever. With $4.4M raised from prominent VCs and seven-day retention in the 60% range, Poncho has been growing strongly.
We had previously said that Poncho achieved an impressive feat as anyone can access weather forecasts with a single click from their phone’s home screen. It seems that we were right to be concerned about the value of the service. Poncho was acquired for what seems to be an immaterial amount.
What we learnt from studying a gazillion chatbots
Restrain your ambition (I know it is tough!)
The scope of your chatbot is crucial. Save for 2 bots, all bots in the list are laser focused. For example, Wordsmith takes in structured data to prepare reports, visabot prepares immigration forms.
Even Facebook M’s general purpose answering machine got shut down. So if even Facebook does not want to handle chatbot queries in any context, then maybe you shouldn’t, too. At least for now.
Let’s look into the more generalist bots. Microsoft’s XiaoIce is impressive but you need a Microsoft-caliber research team to build a bot with such strong understanding of the world. As for Mitsuku, she has been under development since 2005 and while she is engaging, she is not as engaging as XiaoIce which boasts that the average person who adds XiaoIce talks to her more than 60 times per month.
Our articles on chatbot applications and industry/business function specific chatbot use cases can help you identify a specific focus area for your bot.
Find a valuable area for your bot
It is difficult to market your bot given intense competition in the space. Every problem has multiple bots trying to solve it! Even only on Facebook, there’s already 30K+ bots. The successful bots really solve a specific problem like scheduling meetings or generating reports.
Follow best practices, work with experts and test, test, test
Not every chatbot is worth working with outside experts. However, if this is a strategic initiative for your company, bringing in experience about the topic can be beneficial. And invest in testing to avoid embarrassing mistakes. Our guide is a great starting point for chatbot testing. The KPIs of the chatbot is an important part of testing and bot management. Close attention to user behavior can help identify unintended behavior of the chatbot and improve its functionality.
Since you came this far, you are probably excited about launching your own chatbot. There are plenty of platforms that allow you to build bots in a matter of hours. And if you are working for a large company, there are full service chatbot companies that can take your bot concept from idea to implementation. We have an exhaustive list covering both categories. Hope your bot makes it to our list soon! And please comment if you believe we are missing out any important bots in this list.