Amazon Mechanical Turk in 2024: Review-based Analysis

As AI use cases spread across different industries, many organizations are working with partners and AI service providers to develop machine learning models or natural language processing (NLP) systems. In this pursuit, they require data to fuel AI projects. AI data services have become popular in the past few years (Figure 1). Amazon Mechanical Turk is one such company that fulfills businesses’ data needs. 

However, recent news has shed light on some problems with the company¹, which is why business leaders might want to re-evaluate the company before investing in it.

This article offers an in-depth evaluation of Amazon Mechanical Turk to help you make an informed decision. If you are already interested in learning about Amazon Mechanical Turk alternatives, check out this article offering 3 alternatives to MTurk.

Figure 1. Global traffic of the keyword ‘AI data services’

Recent News About Amazon Mechanical Turk

Before we delve into the nitty-gritty details, let’s first catch up on recent news and updates about the company.

1. Difficult working conditions

A recent news article² on an interview with Krystal Kauffman, lead organizer, Turkopticon, highlighted the difficult working conditions for workers at Amazon Mechanical Turk. This can raise concerns with the clients who trying to implement fair trade practices.

2. MTurkers using automated tools

According to a recent study³, around ~33 to 46% of workers on Amazon Mechanical Turk’s platform or MTurkers are using large language models (LLMs) to automate tasks that are specifically meant to be done by humans. This can create issues for clients’ AI models that require human-generated data to be trained.

3. Worker pay less than the U.S. minimum wage

Another recent study

by the University of Massachusetts reported that MTurk’s worker network, which is mainly based in the United States (~75%), is paid a median of <$3 per hour – about ~40% less than the mandated federal minimum wage of >$7 per hour.

4. Significantly lower crowd size than claimed

MTurk claims on its website that its network of workers is over 500,000⁴, but according to a recent report, there are only around 100,000⁵ workers are available for work on the platform.

Amazon Mechanical Turk Performance Evaluation

Before committing to any tech partner, it is important to learn about its performance from the customer’s point of view. One of the most direct ways to do that is through the ratings it receives from third-party platforms. These ratings provide valuable insights into the quality of tasks, approval rate, and reliability of the company’s offerings.

In this section, we highlight some ratings and reviews of customers gathered from B2B review platforms to help you get an overall picture of the performance of MTurk.

User ratings of MTurk

To evaluate the user ratings for MTurk, we only considered platforms that had more than 20 reviews.

Table 1. MTurk user ratings comparison

Customer reviews

This section highlights some positive and negative customer reviews of MTurk identified from third-party B2B review platforms listed in the previous table. We analyzed all the reviews and picked the relevant ones.

Positive reviews of MTurk

1. Regarding the data collection service

A customer who uses Mturk for data collection found the platform to be quick, efficient, and user-friendly⁶.

2. Regarding conducting surveys through MTurk

The customer found the platform to be more professional than other survey platforms⁷.

Negative reviews of MTurk

1. Aggregate of MTurk reviews by workers from Trustpilot

We analyzed around 55 reviews of MTurk workers and identified that while there are some workers who were satisfied with the platform, most were not. The concerns revolved around low rates of pay, account suspension, scams on the platform, and poor management of the company⁸. Here is a visual summary of the reviews:

2. Poor quality of workers

One customer identified that the quality of the work provided by MTurkers is sometimes low⁹.

3- Concerns regarding communication with MTurkers

A customer highlighted that it can be difficult to contact the MTrukers who have completed your HITs for a follow-up¹⁰.

What is Amazon Mechanical Turk?

Amazon Mechanical Turk, or MTurk, is a crowdsourcing marketplace that enables businesses to outsource processes and jobs to a distributed workforce. These jobs, known as human intelligence tasks (HITs), are well-suited for humans but challenging for computers. Examples range from data validation and survey responses to experimental research and writing product descriptions.

MTurk serves as a resource for a variety of users, including academic researchers, businesses, tech developers, market researchers, etc. Tech developers, for example, often use the platform to gather/generate AI training data and obtain annotation services.

Services and Offerings of Amazon Mechanical Turk

This section highlights some of the offerings of MTurk.

1. Human intelligence tasks (HITs)

The bread and butter of MTurk are its Human Intelligence Tasks. These tasks can be anything from answering survey questions to annotating images with relevant information. The vast majority of these tasks are designed to be completed quickly as they are micro-tasks.

2. Data collection

MTurk’s platform also works as an AI training data collection service where developers can gather relevant human-generated data on demand.

3. Experimental research

One of the offerings of Mechanical Turk is experimental research. Academic researchers can use Mechanical Turk as a platform for conducting experiments that require human-annotated data or participant responses.

4. Business processes

Companies can also use MTurk to augment their business processes. The platform offers services like data entry, customer feedback, and writing product descriptions. MTurk also offers API integration to allow businesses to directly work with the platform.

5. Payment and compensation

Mechanical Turk offers a payment system that allows requesters to pay workers either a fixed amount or based on the quality and quantity of work completed.

Amazon Mechanical Turk Alternatives

Before investing in MTurk, it can be beneficial to consider its alternatives. This section highlights some of the key Amazon Mechanical Turk alternatives you can consider.

1. Clickworker: Clickworker is an AI data service provider that works with a crowdsourcing model with a global network of over 4.5 million workers.
2. Appen: This popular AI data service provider offers AI data services through a crowdsourcing platform with a global network of over 1 million workers.
3. Telus International: Offer AI data services through a global network of over 1 million workers.

Further Reading

• Top 3 Prolific Alternatives & Evaluation
• Top 3 Hive Alternatives & Their Evaluations

External Resources

• 1. Coldewey, D. (2023). Mechanical Turk workers are using AI to automate being human. TechCrunch. Accessed: 19/September/2023.
• 2. Skelton, S, K. (2023). AI interview: Krystal Kauffman, lead organiser, Turkopticon. ComputerWeekly.com. Accessed: 20/September/2023
• 3. Veselovsky, V., Ribeiro, M.H. and West, R., (2023). Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. Accessed: 20/September/2023.
• 4. Welcome to the Amazon Mechanical Turk Requester User Interface Guide. Accessed: 20/September/2023.
• 5. Difallah, D., Filatova, E., & Ipeirotis, P. (2018). Demographics and dynamics of mechanical turk workers. In Proceedings of the eleventh ACM international conference on web search and data mining (pp. 135-143). Accessed: 20/September/2023.
• 6. Mturk customer review data collection. G2. Accessed: 20/September/2023
• 7. MTurk customer review surveys. G2. Accessed: 20/September/2023
• 8. Aggregate of reviews by workers. Trustpilot. Accessed: 20/September/2023
• 9. negative review regarding data collection service. G2. Accessed: 20/September/2023.
• 10. Negative review about communication with workers. G2. Accessed: 21/September/2023.