AIMultiple ResearchAIMultiple Research

Top 22 Metrics for Chatbot Analytics in 2024

Updated on Jan 12
9 min read
Written by
Cem Dilmegani
Cem Dilmegani
Cem Dilmegani

Cem is the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per Similarweb) including 60% of Fortune 500 every month.

Cem's work focuses on how enterprises can leverage new technologies in AI, automation, cybersecurity(including network security, application security), data collection including web data collection and process intelligence.

View Full Profile
Top 22 Metrics for Chatbot Analytics in 2024Top 22 Metrics for Chatbot Analytics in 2024

Web focused metrics do not calculate misunderstood requests or message delays. That is why chatbots require new metrics in order to get the most out of their benefits. (P.S. If you are already familiar with web metrics, jump to chatbot specific performance metrics such as message metrics or bot metrics.)

Chatbot analytics is an important tool in guiding brands’ experiments with chatbots. In this article, we cover all relevant question about bot analytics tools and important chatbot metrics that a company needs to measure vis-a-vis the progress of its chatbot/conversational AI system:

What is chatbot analytics?

Chatbot analytics, also referred to as conversational analytics, chatbot analytics, bot analytics, chatbot intelligence, is the process of analyzing historical bot conversations to gain insights about chatbot performance and customer experience.

Why is chatbot analytics important  now?

Maintaining a chatbot does not end with it going live. Due to increasing competition in every industry, customer experience has become  the key driver in gaining a competitive edge. After a business deploys a chatbot, it should track how people are using it. 

Reasons that make conversational analytics important are:

  • Better focus: As with other analytics approaches, chatbot analytics enables organizations to track relevant chatbot KPIs to make data-driven decisions to enhance the chatbot performance.
  • Improved data collection: As new data privacy regulations such as the GDPR and CCPA make the process of using third-party consumer data harder, conversational analytics software becomes a tool to collect first-party data insights since consumers are willingly interacting with bots.
  • Customer insight generation: These platforms enable businesses to map popular user paths, tasks, and exit points via chatbot analytics dashboards (in visual context) so that organizations can identify patterns, trends, and correlations that might not get noticed using text-based data analysis methods. This helps businesses better understand the customer journey.

What is a metric?

a metric is a quantifiable measure that is used to track and assess the status of a specific business process

Defining metrics is important. Since many of the capabilities of the chatbot will be measured through those metrics and companies can measure the efficiency of bot thanks to metrics. 

As far as chatbots are concerned, there are four metrics to keep track of: 

  • User metrics 
  • Message metrics
  • Bot metrics
  • Commercial metrics 

We will expand on each below.

(Courtesy of Botanalytics)

User Metrics

  1. Total Users: This is the most basic metric. It captures the number of people using your chatbot. This matters because its trend shows the change in the number of users, and therefore the amount of data your chatbot has been exposed to. Also, this would provide critical information regarding the market size calculations, and potentially, the effect of your chatbot.
  2. Active Users: Active users can be defined as the people who read a message in the chatbot in a defined time frame. These are your potential targets. Measuring the potential effects of a promotional activity can be estimated from that number. The number of people who read your message is critical. This is something like the promotional content on social media. Engagement is not guaranteed but the content is seen by the people.
  3. Engaged Users: Those users are the ones who communicate with the chatbot. They receive and send messages. This is important since your chatbot will be able to provide the conversation statistics based on this sub-sample. They are likely to shape your decision regarding the effectiveness of the chatbot. It doesn’t make any sense if the chatbot is not able to start the conversation with the users.
  4. New Users: This metric captures the overall success of your chatbot promotion campaign. New users will be necessary to keep an active user number. Customers’ preferences change over time and the amount of interaction with the chatbot shows an exponential decay. For that reason, new users will keep your customer base strong.
  5. User sentiment: This metric is captured by performing sentiment analysis so that you can categorize messages as positive, neutral, or negative. You can gain insights into the user experience and where/when the conversation went wrong.
(Courtesy of Chatbots Life)

Message Metrics

First four metrics capture the overall trend in your user base, but you will be needing greater detail regarding how an individual interacts with your chatbot. One such metric group is the message metrics.

  • 6- Conversation Starter Messages: This is the number of messages where the bot starts the interaction. This is critical for measuring the organic reach of your platform. It is possible to elicit a response by sending messages to users, but as time moves forward, companies would need a lower number for that metric. Since we will implement chatbot for customer relations management and digital marketing, after the initial greeting, we need continuing users to send messages to chatbot directly.
  • 7- Bot Messages: Bot messages are the total number of messages sent by the chatbot in each interaction. This measures the length of a conversation between a customer and the chatbot. We normally want the number of messages to be high, but there is one critical condition; our chatbot needs to respond correctly. In case of misunderstanding or failure to comprehend the input by the user, the chatbot will say similar words repeatedly.
  • 8- In Messages: This category shows the messages sent by the user. We need to see whether the user engages with the chatbot or not. If this category is significantly low, we don’t need to use a chatbot. Using regular social media such as a Facebook Page or a Twitter account may make more sense, rather than using a Facebook Messenger chatbot or a voice-assisted technology.
  • 9- Miss Messages: Miss messages are the ones our chatbot can’t process. This metric may be hard to calculate. Requires the times the chatbot miss interprets the input. This would be a key metric if the firm starts to engage in countries where the language is used more idiomatically.
  • 10- Total Conversations: Number of conversations started and successfully completed on a given day. This is the concept engaged users
  • 11- New Conversations: Number of new conversations started. This captures both the inexperienced users and the conversations that are initiated by the returning users on a different matter, problem, or order.

Startups, such as Pulse Chat focus on measuring the number of messages and provide the details in their platform. Their innovative solution makes it possible to maintain and improve the quality of the conversation instantly.

The above-mentioned metrics are the key metrics for our metric construction. We know that the other traditional digital marketing metrics can be constructed using those basic metrics.

Bot Metrics

We will now introduce some other metrics that are critical for measuring the performance of a chatbot.

  • 12- Retention Rate: This is the percentage of users that return to using the chatbot on the given time frame. This is important since we need to keep the customer engaged to extract valuable insight regarding the customers’ preferences by making them spend time on the chatbot. Higher retention rate can be achieved by promotional campaigns such as chat-to-receive-discount or a lottery like word guessing games. The important thing is keeping that high through an organic process. This mostly can be achieved by providing a high-quality chatbot that meets the customers’ expectations and needs.
A simple retention report that shows retention rate per day after a visit
(A simple retention report courtesy of botanalytics.co)
  • 13- Goal Completion Rate (GCR): This captures the percentage of successful engagement through chatbot. Users will probably try to reach different information or services. For an ecommerce company’s chatbot, relevant goals can be informing the user about a product’s details or purchasing a product. This shows that the number of times our bot successfully processed the input and provided the asked information. There are other ways to utilize this concept. One way is to mine data through the questions asked by the users. This would show an overall trend in consumer preferences; hence a greater emphasis can be placed on that issue while training the chatbot.
Most common phrases visitors use while interacting with a chatbot
(Courtesy of pulsechat)

In the above graph, we can see an example of such use-case. Users mostly used the chatbot for scheduling their rides, hence greater emphasis should be placed on that issue, for keeping the consumer engaged and active.

  • 14/15/16 – Goal Completion time/messages/taps: Chatbots need to provide a seamless and efficient experience and they have plenty of substitutes such as company’s web pages or apps. Minimizing the effort to complete a goal can improve user experience.
  • 17- Fallback Rate (FBR): No robot is perfect; therefore chatbots are expected to fail sometimes, but what really matters is those failures happen regularly or in some extreme cases. Fallback Rate captures that information. This is the percentage of times our chatbot failed or experienced a near-failure situation. The aim should be to minimize this since we need a chatbot that replaces humans such as customer service representatives, salespeople, and call center people. When experiencing a high fallback rate, it might be right to try finding new data sources or training sets for improving the performance. Fallback rate can be examined in three categories:
    • Confusion Rate: Bot may not understand the question that the user is asking or vice versa. Confusion triggers are a valuable indicator of how and where a chatbot needs to be improved. It also provides insights into the quality of the customer experience.
    • Reset Rate: Sometimes users may want to change the previous response in the question chain. These cases are also considered as fallbacks. You can add a reset button to the conversational interface to solve the problem. Tracking this metric may enable you to identify patterns in customer behavior for certain questions
    • Human Takeover Rate: Though sales chatbots are designed to transfer the user to the human agent so that the lead can turn into sales, there are some cases where a human agent had to take over due to repeated failure of the bot or customers prefer to have a comprehensive discussion about the product. According to a customer experience index survey, 86% of customers believe that chatbots should always have an option to transfer to a live agent.
  • 18- User Satisfaction: A new metric can be defined through the exit surveys. Customers or the people engaging with the chatbot can rate their experience to achieve further product excellence. This can be implemented as a binary variable such as “did the bot perform well? – Yes, or No” or we can create more complex evaluation forms to rank and provide points for each different category. This metric would capture the overall effectiveness of the bot from the user experience point. directly provided by the user.
  • 19- Virality: Not all bots are viral, but bots that motivate the user to include others in the conversation can achieve viral growth. If your solution can involve others, that will surely aid growth. You can read an analysis of how viral features helped the growth of the simple bot, Roll.
  • 20-Self-Service Rate: This is the frequency of completed conversations without a need for a second-tier call center. This metric would capture the overall effectiveness of the bot from your point.

Commercial Metrics

  • 21- ROI / payback period: Return on investment (ROI) is critical to know for any project. Companies need to track, at least on a high level, their spending for the bot and the benefits it generates. The benefits will depend on metrics like leads generated, fallback rate, and cost per fallback so that businesses can compare bot’s benefits with other channels.
  • 22-Leads generated: For chatbots in sales, this is one of the most important metrics. They should lead to a higher number or quality of leads captured. For chatbots in customer service, this is a less relevant metric. However, some customer service bots can also identify users’ pain points and cross/up-sell opportunities.

These are some of the common key performance metrics for chatbots. For specific projects, companies sometimes create new, ad hoc metrics to measure the specific impact of improvement projects if they are not captured by these metrics. However, for typical use cases, these metrics will cover the most critical areas.

What are key features of chatbot analytics tools?

Sentiment analysis

Chatbots can capture and process customer sentiments from conversations, without explicitly asking. Thanks to sentiment analysis, businesses can understand whether users are responding positively or negatively and make their bot more user friendly.

Customer segmentation

Chatbot analytics tools can use natural language processing (NLP) to extract conversational data and combine it with web analytics data such as demographics to segment customers. With customer segmentation, businesses are able to personalize conversations and try out A/B testing to chatbot landing pages and query responses to see how they can increase engagement.

Intent analysis

Some tools like Dashbot provide an intent mapping feature to help developers check how they match messages to intent categories.

An illustration of DashBot intend mapping feature
Source: DashBot

Analytics tools enable businesses to track search full text of transcripts to track the complete user lifecycle.

Identification of task failures

With analytics tools, organizations record and categorize all instances where a bot fails to complete a task so that developers work on bot’s responses to improve bot’s performance.

What are example chatbot analytics case studies?

Talkpush

Talkpush is a recruitment platform that leverages conversations and social media to help companies hire talent. Their recruiting chatbot Stanley can screen a large volume of candidates for the right fit. Stanley provides initial screening and can pass inquiries to a human, schedule interviews, and engage candidates while they’re waiting at an onsite interview. Talkpush’s clients are large-scale enterprises that’s why they wanted to ensure they are delivering a flawless experience to their customers.

They purchased DashBot analytics tool to improve their chatbot with data-driven insights. With the conversational analytics tool, Talkpush is able to identify conversations that need optimization by looking at sentiment points. Thanks to analytics, they doubled their chatbot response accuracy from 30 percent to 60 percent from January to July 2019.

What are leading chatbot analytics vendors & tools?

  • Botanalytics
  • BotExpress.ai
  • ChatBase
  • ChatBot
  • Chat Metrics
  • Dashbot
  • Enterprise Bot Manager (EBM)
  • Kore.ai

To learn more on chatbot analytics and bot metrics

This video can help you guide your chatbot analytics process. Ofer Ronen, CEO of Chatbase, is explaining what KPIs matter for digital assistants and how to optimize them. You will also see and understand how key KPIs are seen on dashboards and how to interpret those visualizations.

Now that you know how to measure the progress of your chatbot, it may be a good time to

  1. Identify common strategies for chatbot success
  2. Get inspired by chatbot success stories and explore our comprehensive list of chatbot use cases to see how they can help your company
  3. When you are ready to build your company’s chatbot, explore the chatbot ecosystem, top chatbot companies, compare the top chatbot platforms.
  4. Get ready for the conclusion of chatbot development by exploring our detailed guide on chatbot testing

Finally, if you believe your business would benefit from an conversational AI platform, identify all chatbot platform and voice bot platform vendors with our sortable and transparent lists.

And let us know if you are looking for a chatbot vendor:

Find the Right Vendors
Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on
Cem Dilmegani
Principal Analyst

Cem is the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per Similarweb) including 60% of Fortune 500 every month.

Cem's work focuses on how enterprises can leverage new technologies in AI, automation, cybersecurity(including network security, application security), data collection including web data collection and process intelligence.

Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.

Cem's hands-on enterprise software experience contributes to the insights that he generates. He oversees AIMultiple benchmarks in dynamic application security testing (DAST), data loss prevention (DLP), email marketing and web data collection. Other AIMultiple industry analysts and tech team support Cem in designing, running and evaluating benchmarks.

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

Sources:

AIMultiple.com Traffic Analytics, Ranking & Audience, Similarweb.
Why Microsoft, IBM, and Google Are Ramping up Efforts on AI Ethics, Business Insider.
Microsoft invests $1 billion in OpenAI to pursue artificial intelligence that’s smarter than we are, Washington Post.
Data management barriers to AI success, Deloitte.
Empowering AI Leadership: AI C-Suite Toolkit, World Economic Forum.
Science, Research and Innovation Performance of the EU, European Commission.
Public-sector digitization: The trillion-dollar challenge, McKinsey & Company.
Hypatos gets $11.8M for a deep learning approach to document processing, TechCrunch.
We got an exclusive look at the pitch deck AI startup Hypatos used to raise $11 million, Business Insider.

To stay up-to-date on B2B tech & accelerate your enterprise:

Follow on

Next to Read

Comments

Your email address will not be published. All fields are required.

3 Comments
Helen Buchanan
Apr 14, 2020 at 15:53

Does anyone have any targets not just the metric

AIMultiple
Apr 27, 2020 at 21:33

Targets really depend on the implementation. A personal companion chatbot will have much different targets than a chatbot optimized for e-commerce.

Liyuan
Dec 11, 2019 at 10:25

Great! I am analyzing our chatbot and get some new ideas from you, such as goal completion rate and fallback rate. It must be useful to monitor these aspects.

ashish
Jan 22, 2018 at 15:55

Nice read. I agree with the tenets and conclusions that you have drawn around analytics. They are important to monitors your chatbot performance to make sure it

Related research