Chatbots use natural language processing (NLP) to understand the users’ intent and provide the best possible conversational service. Intent recognition is a critical feature in chatbot architecture that determines if a chatbot will succeed at fulfilling the user’s needs in sales, marketing or customer service.
The quantity of the chatbot’s training data is key to maintaining a good conversation with the users. However, the data quality determines the bot’s ability to detect the right intent and generate the correct response.
In this article, we present techniques for companies to improve the intent recognition of their chatbots in order to make them more efficient.
How do chatbots identify intent?
NLP allows the chatbot to understand the user’s message, and machine learning classification algorithms help it to classify this message based on the training data in order to deliver the correct response.
The steps required for the chatbot to have a meaningful conversation include:
1. Preprocessing for NLU
Natural language understanding (NLU) is a subfield in NLP that focuses on organizing the user’s unstructured input such that the chatbot can understand and analyze it. This process includes:
- Syntax analysis: Identifying the basic grammar rules, word organization, combination and relation to one another. This consists of :
- Splitting the text into smaller segments (words, shorter sentences) called “Tokens,”
- Labeling the tokens as nouns, verbs, adjectives, etc. This step is called “Part of Speech tagging (PoS).
- Reducing words into their roots for better analysis.
- Filter out filling words to save space and time in processing large data
- Semantic analysis: inferring the meaning of the input sentence by:
- distinguishing the context of each word
- understand the relationships between the words in the text.
NLU models utilize:
- Supervised machine learning for syntax analysis steps (tokenization, PoS tagging), such as support vector machines (SVM), Bayesian networks, and maximum entropy algorithms.
- Unsupervised machine learning for semantic analysis such as clustering algorithms.
For more on difference between NLU and NLP you can read our NLU vs NLP: Main Differences & Use Cases Comparison article.
2. Chatbot intent classification
The classification of customers’ potential intentions is done by classifiers. One customer would want to check the status of an order, while another customer might want to check the coupons she has, which are two different types of queries.
Classifiers are trained on relevant labeled datasets. Therefore, this is a supervised/human in the loop learning application. Classifiers utilize:
- Rules-based pattern matching
- Machine learning classification algorithms such as decision trees, naive Bayes, and logistic regression.
- Deep learning such as artificial neural networks
An intent classifier is used to match the output of the NLU process to relevant pre-defined labels in the training dataset. For example, when the user tells the chatbot: “I want to book a flight from Houston to LA”, the intent classifier will classify the context and sequence of words under the label “book flight”.
3. Response generation
To generate responses, chatbots either rely on pre-defined recommendations or they could generate recommendations on the fly.
For commercial applications, responses tend to be pre-defined to ensure that customers receive a consistent service and the bot does not respond in unintended ways leading to public relations failures.
The dialog is formulated to achieve a specific goal, such as acquiring the user’s information, providing suggestions about a product or a service, or directing the user to a live agent.
What are the challenges of chatbots’ intent recognition, and how to overcome them?
Chatbots’ goal is to produce human-like comprehensive conversations to meet the user’s expectations. However, chatbots face limitations in understanding the user’s intent because:
- Natural language is difficult, even for humans.
- As humans, we have the ability to represent our intent using varying linguistic structures.
- Users sometimes make typos and use keywords instead of complete sentences.
- Most chatbot responses are limited to the pre-defined intents in the trained dataset.
Limitations cause chatbots to misunderstand the user’s intent, deliver wrong responses, and fail to fulfill their purpose of use. Feel free to read our article on chatbot epic fails for more details.
However, some limitations can be addressed by integrating different solutions:
- AI spell checking algorithms can be implemented with NLP models to autocorrect users’ misspellings and typos.
- For example, Google Docs’s autocorrect feature points out misspellings, grammatical errors, and provides enhancements on text structure.
- Users can create custom intents.For example, in Amazon Alexa, the user can set rules for the chatbot to perform a specific task by providing a name and a list of utterances that users would say to invoke this intent.
- Increasing the volume of training data will decrease the margin of error in intent detection.
We expect this problem to be solved eventually by having more training data. There is already significant and increasing usage of conversational bots. Based on our statistics about chatbots,
- 23% of organizations use chatbots in administrative departments.
- 20% use them in the customer service department.
- 16% use AI chatbots and assistants in sales and marketing departments.
- 40% of millennials engage with chatbots on a daily basis.
For more on chatbots and NLP
If you are interested in learning more about chatbots, you may find these articles interesting as well:
- Top 12 Benefits of Chatbots: The Ultimate Guide.
- 30+ Chatbot Use cases / Applications in Business.
- Conversational Commerce: Benefits, Best Practices & Examples.
- Top 5 Conversational Commerce Examples & Success Stories.
- Top 5 Expectations Regarding the Future of NLP.
If you are ready to deploy chatbots and want to make sure that you deploy the highest performing ones when it comes to intent detection, we have 2 lists of vendors to help you:
And if you want us to guide you to find the best chatbot tool for your business
This article was drafted by former AIMultiple industry analyst Alamira Jouman Hajjar.
Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 60% of Fortune 500 every month.
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE, NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and media that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised businesses on their enterprise software, automation, cloud, AI / ML and other technology related decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
To stay up-to-date on B2B tech & accelerate your enterprise:Follow on
Next to Read
Your email address will not be published. All fields are required.