Data Annotation in 2024: Why it matters & Top 8 Best Practices

Annotated data is an integral part of various machine learning, artificial intelligence (AI) and GenAI applications. It is also one of the most time-consuming and labor-intensive parts of AI/ML projects. Data annotation is one of the top limitations of AI implementation for organizations. Whether you work with an AI data service, or perform annotation in-house, you need to get this process right.

Tech leaders and developers need to focus on improving data annotation for their data-hungry digital solutions. To remedy that, we recommend an in-depth understanding of data annotation.

Our research covers the following:

• What is data annotation?
• Why it matters?
• What its techniques/types are?
• What are some key challenges of annotating data?
• What are some best practices for data annotation?

What is data annotation?

Data annotation is the process of labeling data with relevant tags to make it easier for computers to understand and interpret. This data can be in the form of images, text, audio, or video, and data annotators need to label it as accurately as possible. Data annotation can be done manually by a human or automatically using advanced machine learning algorithms and tools. Learn more about automated data annotation.

For supervised machine learning, labeled datasets are crucial because ML models need to understand input patterns to process them and produce accurate results. Supervised ML models (see figure 1) train and learn from correctly annotated data and solve problems such as:

• Classification: Assigning test data into specific categories. For instance, predicting whether a patient has a disease and assigning their health data to “disease” or “no disease” categories is a classification problem.
• Regression: Establishing a relationship between dependent and independent variables. Estimating the relationship between the budget for advertising and the sales of a product is an example of a regression problem.

Figure 1: Supervised Learning Example¹

For example, training machine learning models of self-driving cars involve annotated video data. Individual objects in videos are annotated, which allows machines to predict the movements of objects.

Other terms to describe data annotation include data labeling, data tagging, data classification, or machine learning training data generation.

Why does data annotation matter?

Annotated data is the lifeblood of supervised learning models since the performance and accuracy of such models depend on the quality and quantity of annotated data. Machines can not see images and videos as we do. Data annotation makes the different data types machine-readable. Annotated data matters because:

• Machine learning models have a wide variety of critical applications (e.g., healthcare) where erroneous AI/ML models can be dangerous
• Finding high-quality annotated data is one of the primary challenges of building accurate machine-learning models

Here is a data-driven list of the top data annotation services on the market.

Gathering data is a prerequisite for annotation. To help you obtain the right datasets, here is some research:

• Top data crowdsourcing platforms on the market
• Guide to AI data collection.
• Data-driven list of data collection/harvesting services.

What are the different types of data annotation?

Different data annotation techniques can be used depending on the machine learning application. Some of the most common types are:

1. RLHF

Reinforcement learning with human feedback (RLHF) was identified in 2017.² It increased in popularity significantly in 2022 after the success of large language models (LLMS) like ChatGPT which leveraged the technology. These are the two main types of RLHF:

• Humans generating suitable responses to train LLMs
• Humans annotating (i.e. selecting) better responses among multiple LLM responses.

Human labor is expensive and AI companies are also leveraging reinforcement learning from AI feedback (RLAIF) to scale their annotations cost effectively in cases where AI models are confident about their feedback. ³

2. Text annotation

Text annotation trains machines to better understand the text. For example, chatbots can identify users’ requests with the keywords taught to the machine and offer solutions. If annotations are inaccurate, the machine is unlikely to provide a useful solution. Better text annotations provide a better customer experience. During the data annotation process, with text annotation, some specific keywords, sentences, etc., are assigned to data points. Comprehensive text annotations are crucial for accurate machine training. Some types of text annotation are:

2.1. Semantic annotation

Semantic annotation (see figure 2) is the process of tagging text documents. By tagging documents with relevant concepts, semantic annotation makes unstructured content easier to find. Computers can interpret and read the relationship between a specific part of metadata and a resource described by semantic annotation.

Figure 2: Semantic Annotation Example⁴

2.2. Intent annotation

For example, the sentence “I want to chat with David” indicates a request. Intent annotation analyzes the needs behind such texts and categorizes them, such as requests and approvals.

2.3. Sentiment annotation

Sentiment annotation (see Figure 3) tags the emotions within the text and helps machines recognize human emotions through words. Machine learning models are trained with sentiment annotation data to find the true emotions within the text. For example, by reading the comments left by customers about the products, ML models understand the attitude and emotion behind the text and then make the relevant labeling such as positive, negative, or neutral.

Figure 3: Sentiment Annotation Example⁵

3. Text categorization

Text categorization assigns categories to the sentences in the document or the whole paragraph in accordance with the subject. Users can easily find the information they are looking for on the website.

4. Image annotation

Image annotation is the process of labeling images (see figure 4)  to train an AI or ML model. For example, a machine learning model gains a high level of comprehension like a human with tagged digital images and can interpret the images it sees. With data annotation, objects in any image are labeled. Depending on the use case, the number of labels on the image may increase. There are four fundamental types of image annotation:

4.1. Image classification

First, the machine trained with annotated images then determines what an image represents with the predefined annotated images.

4.2. Object recognition/detection

Object recognition/detection is a further version of image classification. It is the correct description of the numbers and exact positions of entities in the image. While a label is assigned to the entire image in image classification, object recognition labels entities separately. For example, with image classification, the image is labeled as day or night. Object recognition individually tags various entities in an image, such as a bicycle, tree, or table.

4.3. Segmentation

Segmentation is a more advanced form of image annotation. In order to analyze the image more easily, it divides the image into multiple segments, and these parts are called image objects. There are three types of image segmentation:

• Semantic segmentation: Label similar objects in the image according to their properties, such as their size and location.
• Instance segmentation: Each entity in the image can be labeled. It defines the properties of entities such as position and number.
• Panoptic segmentation: Both semantic and instance segmentations are used by combining.

Figure 4: Image annotation example⁶

5. Video annotation

Video annotation is the process of teaching computers to recognize objects from videos. Image and video annotation are types of data annotation methods that are performed to train computer vision (CV) systems, which is a subfield of artificial intelligence (AI).

Video annotation for a retail store surveillance system:

Click here to learn more about video annotation.

6. Audio annotation

Audio annotation is a type of data annotation that involves classifying components in audio data. Like all other types of annotation (such as image and text annotation), audio annotation requires manual labeling and specialized software. Solutions based on natural language processing (NLP) rely on audio annotation, and as their market grows (projected to grow 14 times between 2017 and 2025), the demand and importance of quality audio annotation will grow as well.

Audio annotation can be done through software that allows data annotators to label audio data with relevant words or phrases. For example, they may be asked to label a sound of a person coughing as “cough.”

Audio annotation can be: 

• In-house, completed by that company’s employees.
• Outsourced (i.e., done by a third-party company.)
• Crowdsourced. Crowdsourced data annotation involves using a large network of data annotators to label data through an online platform.

Learn more about audio annotation.

7. Industry-specific data annotation

Each industry uses data annotation differently. Some industries use one type of annotation, and others use a combination to annotate their data. This section highlights some of the industry-specific types of data annotation.

• Medical data annotation: Medical data annotation is used to annotate data such as medical images (MRI scans), EMRs, and clinical notes, etc. This type of data annotation helps develop computer vision-enabled systems for disease diagnosis and automated medical data analysis.
• Retail data annotation: Retail data annotation is used to annotate retail data such as product images, customer data, and sentiment data. This type of annotation helps create and train accurate AI/ML models to determine the sentiment of customers, product recommendations, etc.
• Finance data annotation: Finance data annotation is used to annotate data such as financial documents, transactional data, etc. This type of annotation helps develop AI/ML systems, such as fraud and compliance issues detection systems.
• Automotive data annotation: This industry-specific annotation is used to annotate data from autonomous vehicles, such as data from cameras and lidar sensors. This annotation type helps develop models that can detect objects in the environment and other data points for autonomous vehicle systems.
• Industrial data annotation: Industrial data annotation is used to annotate data from industrial applications, such as manufacturing images, maintenance data, safety data, quality control, etc. This type of data annotation helps create models that can detect anomalies in production processes and ensure worker safety.

What is the difference between data annotation and data labeling?

Data annotation and data labeling mean the same thing. You will come across articles that try to explain them in different ways and make up a difference. For example, some sources claim that data labeling is a subset of data annotation where data elements are assigned labels according to predefined rules or criteria. However, based on our discussions with vendors in this space and with data annotation users, we do not see major differences between these concepts.

What are the main challenges of data annotation?

• Cost of annotating data: Data annotation can be done either manually or automatically. However, manually annotating data requires a lot of effort, and you also need to maintain the quality of the data.
• Accuracy of annotation: Human errors can lead to poor data quality, and these have a direct impact on the prediction of AI/ML models. Gartner’s study highlights that poor data quality costs companies 15% of their revenue.

What are the best practices for data annotation?

1. Start with the correct data structure: Focus on creating data labels that are specific enough to be useful but still general enough to capture all possible variations in data sets.
2. Prepare detailed and easy-to-read instructions: Develop data annotation guidelines and best practices to ensure data consistency and accuracy across different data annotators.
3. Optimize the amount of annotation work: Annotation is costlier and cheaper alternatives need to be examined. You can work with a data collection service that offers pre-labeled datasets.
4. Collect data if necessary: If you don’t annotate enough data for machine learning models, their quality can suffer. You can work with data collection companies to collect more data.
5. Leverage outsourcing or crowdsourcing if data annotation requirements become too large and time-consuming for internal resources.
6. Support humans with machines: Use a combination of machine learning algorithms (data annotation software) with a human-in-the-loop approach to help humans focus on the hardest cases and increase the diversity of the training data set. Labeling data that the machine learning model can correctly process has limited value. 
7. Focus on quality:
   1. Regularly test your data annotations for quality assurance purposes.
   2. Have multiple data annotators review each other’s work for accuracy and consistency in labeling datasets.
8. Stay compliant: Carefully consider privacy and ethical issues when annotating sensitive data sets, such as images containing people or health records. Lack of compliance with local rules can damage your company’s reputation.

By following these data annotation best practices, you can ensure that your data sets are accurately labeled and accessible to data scientists and fuel your data-hungry projects.

You can also check our video annotation tools list to choose the fit that best suits your annotation needs.

If you have questions about data annotation, we would like to help:

External links

• 1. Diego Calvo. (2019). Supervised learning. Diego Calvo. Accessed: 29/September/2023.
• 2. Christiano P.; Leike J.; Brown T.B.; Martic M.; Legg S.; Amodei D. (2017). “Deep reinforcement learning from human preferences“
• 3. Bai Y.; et al. (2022). “Constitutional AI: Harmlessness from AI Feedback”. Retrieved January 1, 2024
• 4. Articles Hubspot. (2019). What Is Text Annotation in Machine Learning, Examples and How it’s Done?. Accessed: 29/September/2023.
• 5. Sentiment Annotation – Quick Start Guide. Accessed: 29/September/2023.
• 6. Ashely John. (2020). Why Data & Data Annotation Make or Break AI. Medium. Accessed: 29/September/2023.