(Courtesy of Master of Code)

Web focused metrics do not calculate misunderstood requests or message delays. Chatbots require new metrics. If you are already familiar with web metrics, jump to chatbot specific metrics such as message metrics or bot metrics

We have previously written an in-depth guide on conversational bots/chatbots. We argued that the chatbots will change the world and brands should start experimenting with them. Chatbot analytics is an important tool in guiding brands’ experiments with chatbots. We cover all relevant question about bot analytics and the important metrics that a company needs to measure the progress of its chatbot/conversational AI system:

What is chatbot analytics?

Chatbot analytics, is also referred to as conversational analytics, chatbot analytics, bot analytics, chatbot intelligence. It is the process of analyzing historical bot conversations to gain insights about chatbot performance and customer experience.

Why is it important now?

A business’s work as a chatbot developer doesn’t end once their bot goes live. Due to increasing competition in every industry, customer experience became the key driver in gaining a competitive edge. After a business deploys a chatbot, it is time to track how people are using it. Using this data, organizations can make data-driven decisions to enhance the customer journey they offer.

What is a metric?

a metric is a quantifiable measure that is used to track and assess the status of a specific business process

Defining metrics is important. Since many of the capabilities of the chatbot will be measured through those metrics. For a newly created chatbot, those metrics can fluctuate dramatically. After implementing the chatbot, companies need to monitor it closely. Since the expectations are efficiency enhancement, faster response, and greater conversion, companies need to define the right metrics. This way, the performance of the bot can be monitored and improved efficiency. For data scientists, this dimension of the issue provides many use cases.

(Courtesy of Botanalytics)

User Metrics

  • 1- Total Users: This is the most basic metric. It captures the number of people using your chatbot. This matter because its trend shows the change in the number of users and therefore the amount of data your chatbot has been exposed to. Also, this would provide critical information regarding the market size calculations and potentially the effect of your chatbot.
  • 2- Active Users: Active users can be defined as the people who read a message in the chatbot in a defined time frame. These are your potential targets. Measuring the potential effects of a promotional activity can be estimated from that number. The number of people who read your message is critical. This is something like the promotional content on social media. Engagement is not guaranteed but, the content is seen by the people
  • 3- Engaged Users: Those users are the ones who communicate with the chatbot. They receive and send messages. This is important since your chatbot will be able to provide the conversation statistics based on this sub-sample. They are likely to shape your decision regarding the effectiveness of the chatbot. It doesn’t make any sense if the chatbot is not able to start the conversation with the users.
  • 4- New Users: This metric captures the overall success of your chatbot promotion campaign. New users will be necessary to keep an active user number. Customers’ preferences changes over time and the amount of interaction with the chatbot shows an exponential decay. For that reason, new users will keep your customer base strong.
  • 5- User sentiment: This metric is captured by performing sentiment analysis so that you can categorize messages as positive, neutral, or negative. You can gain insights into the user experience and where/when the conversation went wrong.
(Courtesy of Chatbots Life)

Message Metrics

First four metrics capture the overall trend in your user base, but you will be needing a greater detail regarding how an individual interacts with your chatbot. One such metric group is the message metrics.

  • 6- Conversation Starter Messages: This is the number of messages where you start the interaction by the bot. This critical for measuring the organicness of your platform. It is possible to elicit a response by sending messages to users, but as the time moves forward, companies would need a lower number for that metric. Since we will implement chatbot for customer relations management and digital marketing, after the initial greeting, we need continuing users to send messages to chatbot directly.
  • 7- Bot Messages: Bot messages are the total number of messages sent by the chatbot in each interaction. This measures the length of a conversation between a customer and the chatbot. We normally want the number of messages to be high, but there is one critical condition, our chatbot needs to respond correctly. In case of misunderstanding or failure to comprehend the input by the user, the chatbot will say similar words repeatedly.
  • 8- In Messages: This category shows the messages sent by the user. We need to see whether the user engages in with the chatbot or not. If this category is significantly low, we don’t need to use a chatbot. Using regular social media such as a Facebook Page or a Twitter account may make more sense, rather than using a Facebook Messenger chatbot or a voice-assisted technology.
  • 9- Miss Messages: Miss messages are the ones our chatbot can’t process. This metric may be hard to calculate. Requires the times the chatbot miss interprets the input. This would be a key metric if the firm starts to engage in countries where the language is used more idiomatically.
  • 10- Total Conversations: Number of conversations started and successfully completed on a given day. This is the concept engaged users
  • 11- New Conversations: Number of new conversations started. This captures both the inexperienced users and the conversations that are initiated by the returning users on a different matter, problem, or order.

Startups such as Pulse Chat focuses on measuring the number of messages and provides the details in their platform. Their innovative solution makes it possible to maintain and improve the quality of the conversation instantly.

The above-mentioned metrics are the key metrics for our metric construction. We know that the other traditional digital marketing metrics can be constructed using those basic metrics.

Bot Metrics

We will now introduce some other metrics that are critical for measuring the performance of a chatbot.

  • 12- Retention Rate: This is the percentage of users that return to using the chatbot on the given time frame. This important since we need to keep the customer engaged to extract valuable insight regarding the customers’ preferences by making them spend time on the chatbot. Higher retention rate can be achieved by promotional campaigns such as chat-to-receive-discount or a lottery like a word guessing games. The important is keeping that high through an organic process. This mostly can be achieved by providing a high-quality chatbot that meets the customers’ expectations and needs.
A simple retention report that shows retention rate per day after a visit
(A simple retention report courtesy of botanalytics.co)
  • 13- Goal Completion Rate (GCR): This captures the percentage of successful engagement through chatbot. Users will probably try to reach different information or service. For an e-commerce company’s bot relevant goals can be informing user about a product’s details or purchasing a product. This shows that the number of times our bot successfully processed the input and provided the asked information. There are other ways to utilize this concept. One way is to mine data through the questions asked by the users. This would show an overall trend in consumer preferences; hence a greater emphasis can be placed on that issue while training the chatbot.
Most common phrases visitors use while interacting with a chatbot
(Courtesy of pulsechat)

In the above graph, we can see an example of such use-case. Users mostly used the chatbot for scheduling their rides, hence greater emphasis should be placed on that issue, for keeping the consumer engaged and active.

  • 14/15/16 – Goal Completion time/messages/taps: Chatbots need to provide a seamless and efficient experience and they have plenty of substitutes such as company’s web pages or apps. Minimizing the effort to complete a goal can improve user experience.
  • 17- Fallback Rate (FBR): No robot is perfect; therefore chatbots are expected to fail sometimes, but what really matters is those failures happen regularly or in some extreme cases. Fallback Rate captures that information. This is the percentage of times our chatbot failed or experienced a near-failure situation. Our aim is to minimize this since we need a chatbot that replaces humans such as customer service representatives, salespeople, and call center people. When experience a high fallback rate, it might be right to try finding new data sources or training sets for improving the performance. Fallback rate can be examined in three categories:
    • Confusion Rate: Bot may not understand the question that the user is asking or the vice versa. Confusion triggers are a valuable indicator of how and where a chatbot needs to be improved. It also provides insights into the quality of the customer experience.
    • Reset Rate: Sometimes users may want to change the previous response in the question chain. These cases are also considered as fallbacks. You can add a reset button to the conversational interface to solve the problem. Tracking this metric may enable you to identify patterns in customer behavior for certain questions
    • Human Takeover Rate: Though sales chatbots are designed to transfer the user to the human agent so that the lead can turn into sales, there are some cases where a human agent had to take over due to repeated failure of the bot or customers prefer to have a comprehensive discussion about the product. According to customer experience index survey, 86% of customers believe that chatbots should always have an option to transfer to a live agent.
  • 18- User Satisfaction: A new metric can be defined through the exit surveys. Customers or the people engaging with the chatbot can rate their experience to achieve further product excellence. This can be implemented as a binary variable such as “did the bot perform well? – Yes, or No” or we can create more complex evaluation forms to rank and provide points for each different category. This metric would capture the overall effectiveness of the bot from the user experience point. directly provided by the user.
  • 19- Virality: Not all bots are viral but bots that motivate the user to include others in the conversation can achieve viral growth. If your solution can involve others, that will surely aid growth. You can read an analysis of how viral features helped the growth of the simple bot, Roll.
  • 20-Self-Service Rate: Frequency of completed conversations without a need for a second-tier call center. This metric would capture the overall effectiveness of the bot from your point.

Commercial Metrics

  • 21- ROI / payback period: Return on investment (ROI) is critical to know for any project. Companies need to track, at least on a high level, their spending for the bot and the benefits it generates. the benefits will depend on metrics like leads generated, fallback rate, cost per fallback so that businesses can compare bot’s benefits with other channels.
  • 22-Leads generated: For chatbots in sales, this is one of the most important metrics. They should lead to a higher number or quality of leads captured. For chatbots in customer service, this is a less relevant metric however some customer service bots can also identify users’ pain points and cross/up-sell opportunities.

These are some of the common key performance metrics for chatbots. For specific projects, companies sometimes create new, ad hoc metrics to measure the specific impact of improvement projects if they are not captured by these metrics. However, for typical use cases, these metrics will cover the most critical areas.

What are key features of chatbot analytics tools?

Sentiment analysis

Chatbots can capture and process customer sentiments from conversations, without explicitly asking. Thanks to sentiment analysis, businesses can understand whether users are responding positively or negatively and make their bot more user friendly.

Customer segmentation

Chatbot analytics tools can use natural language processing(NLP) to extract conversational data and combine it with web analytics data such as demographics to segment customers. With customer segmentation, businesses are able to personalize conversations and try out A/B testing to chatbot landing pages and query responses to see how they can increase engagement.

Intent analysis

Some tools like Dashbot provides intent mapping feature to help developers check how they match messages to intent categories.

An illustration of DashBot intend mapping feature
Source: DashBot

Transcript Search

Analytics tools enable businesses to track search full text of transcripts to track the complete user lifecycle.

Identification of task failures

With analytics tools, organizations record and categorize all instances where a bot fails to complete a task so that developers work on bot’s responses to improve bot’s performance.

What are example chatbot analytics case studies?

Talkpush

Talkpush is a recruitment platform that leverages conversations and social media to help companies hire talent. Their chatbot Stanley can screen a large volume of candidates for the right fit. Stanley provides initial screening and can pass inquiries to a human, schedule interviews, and engage candidates while they’re waiting at an onsite interview. Talkpush’s clients are large-scale enterprises that’s why they wanted to ensure they are delivering a flawless experience to their customers.

They purchased DashBot analytics tool to improve their chatbot with data-driven insights. With the conversational analytics tool, Talkpush is able to identify conversations that need optimization by looking at sentiment points. Thanks to analytics, they doubled their chatbot response accuracy from 30 percent to 60 percent from January to July 2019.

What are leading chatbot analytics vendors?

  • Botanalytics
  • BotExpress.ai
  • ChatBase
  • ChatBot
  • Chat Metrics
  • Dashbot
  • Enterprise Bot Manager (EBM)
  • Kore.ai

Botanalytics is based in San Francisco and is the leading company in conversational analytics tools. They were established in 2016, their main goal is to improve human-bot interactions and conversational UI through data. By offering analytics for bots, they empower bot developers, corporate brands and agencies to improve their chatbots with historical and real-time conversation data.

Now that you know how to measure the progress of your chatbot, it may be a good time to

And let us know if you are looking for a chatbot vendor:

Let us find the right vendor for your business

4 comments

    1. Targets really depend on the implementation. A personal companion chatbot will have much different targets than a chatbot optimized for e-commerce.

  1. Great! I am analyzing our chatbot and get some new ideas from you, such as goal completion rate and fallback rate. It must be useful to monitor these aspects.

  2. Nice read. I agree with the tenets and conclusions that you have drawn around analytics. They are important to monitors your chatbot performance to make sure it

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*