AIMultiple ResearchAIMultiple Research

In-Depth Guide Into Chatbots Intent Recognition in 2024

Cem Dilmegani
Updated on Jan 12
4 min read

Chatbots use natural language processing (NLP) to understand the users’ intent and  provide the best possible conversational service. Intent recognition is a critical feature in chatbot architecture that determines if a chatbot will succeed at fulfilling the user’s needs in sales, marketing or customer service.

The quantity of the chatbot’s training data is key to maintaining a good conversation with the users. However, the data quality determines the bot’s ability to detect the right intent and generate the correct response.

In this article, we present techniques for companies to improve the intent recognition of their chatbots in order to make them more efficient.

How do chatbots identify intent?

Image shows process of intent recognition for chatbots.
source: Journal of Information Science Theory and Practice

NLP allows the chatbot to understand the user’s message, and machine learning classification algorithms help it to classify this message based on the training data in order to deliver the correct response. 

The steps required for the chatbot to have a meaningful conversation include:

1. Preprocessing for NLU

Natural language understanding (NLU) is a subfield in NLP that focuses on organizing the user’s unstructured input such that the chatbot can understand and analyze it. This process includes:

  1. Syntax analysis: Identifying the basic grammar rules, word organization, combination and relation to one another. This consists of :
    • Splitting the text into smaller segments (words, shorter sentences) called “Tokens,”
    • Labeling the tokens as nouns, verbs, adjectives, etc. This step is called “Part of Speech tagging (PoS).
    • Reducing words into their roots for better analysis.
    • Filter out filling words to save space and time in processing large data
  2. Semantic analysis: inferring the meaning of the input sentence by:
    • distinguishing the context of each word
    • understand the relationships between the words in the text.

NLU models utilize:

  • Supervised machine learning for syntax analysis steps (tokenization, PoS tagging), such as support vector machines (SVM), Bayesian networks, and maximum entropy algorithms.
  • Unsupervised machine learning for semantic analysis such as clustering algorithms.

For more on difference between NLU and NLP you can read our NLU vs NLP: Main Differences & Use Cases Comparison article.

2. Chatbot intent classification

The classification of customers’ potential intentions is done by classifiers. One customer would want to check the status of an order, while another customer might want to check the coupons she has, which are two different types of queries.

Classifiers are trained on relevant labeled datasets. Therefore, this is a supervised/human in the loop learning application. Classifiers utilize:

  • Rules-based pattern matching
  • Machine learning classification algorithms such as decision trees, naive Bayes, and logistic regression.
  • Deep learning such as artificial neural networks

An intent classifier is used to match the output of the NLU process to relevant pre-defined labels in the training dataset. For example, when the user tells the chatbot: “I want to book a flight from Houston to LA”, the intent classifier will classify the context and sequence of words under the label “book flight”.

3. Response generation

To generate responses, chatbots either rely on pre-defined recommendations or they could generate recommendations on the fly. 

For commercial applications, responses tend to be pre-defined to ensure that customers receive a consistent service and the bot does not respond in unintended ways leading to public relations failures.

The dialog is formulated to achieve a specific goal, such as acquiring the user’s information, providing suggestions about a product or a service, or directing the user to a live agent.

Image shows how chatbot intent recognition works.
source: structurely Conversational AI for Real Estate

What are the challenges of chatbots’ intent recognition, and how to overcome them?

Challenges

Chatbots’ goal is to produce human-like comprehensive conversations to meet the user’s expectations. However, chatbots face limitations in understanding the user’s intent because:

  • Natural language is difficult, even for humans.
  • As humans, we have the ability to represent our intent using varying linguistic structures.
  • Users sometimes make typos and use keywords instead of complete sentences.
  • Most chatbot responses are limited to the pre-defined intents in the trained dataset.

Limitations cause chatbots to misunderstand the user’s intent, deliver wrong responses, and fail to fulfill their purpose of use. Feel free to read our article on chatbot epic fails for more details.

Best Practices

However, some limitations can be addressed by integrating different solutions:

  • AI spell checking algorithms can be implemented with NLP models to autocorrect users’ misspellings and typos.
    • For example, Google Docs’s autocorrect feature points out misspellings, grammatical errors, and provides enhancements on text structure.
  • Users can create custom intents.For example, in Amazon Alexa, the user can set rules for the chatbot to perform a specific task by providing a name and a list of utterances that users would say to invoke this intent.
  • Increasing the volume of training data will decrease the margin of error in intent detection.

We expect this problem to be solved eventually by having more training data. There is already significant and increasing usage of conversational bots. Based on our statistics about chatbots,

  • 23% of organizations use chatbots in administrative departments.
  • 20% use them in the customer service department.
  • 16% use AI chatbots and assistants in sales and marketing departments.
  • 40% of millennials engage with chatbots on a daily basis.

For more on chatbots and NLP

If you are interested in learning more about chatbots, you may find these articles interesting as well:

If you are ready to deploy chatbots and want to make sure that you deploy the highest performing ones when it comes to intent detection, we have 2 lists of vendors to help you:

And if you want us to guide you to find the best chatbot tool for your business

Find the Right Vendors

Image credits

Conversational AI for Real Estate

This article was drafted by former AIMultiple industry analyst Alamira Jouman Hajjar.

Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on

Cem Dilmegani
Principal Analyst

Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 60% of Fortune 500 every month.

Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.

Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised businesses on their enterprise software, automation, cloud, AI / ML and other technology related decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.

He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.

Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.

To stay up-to-date on B2B tech & accelerate your enterprise:

Follow on

Next to Read

Comments

Your email address will not be published. All fields are required.

0 Comments