AIMultiple ResearchAIMultiple Research

Amazon Mechanical Turk in 2024: Review-based Analysis

Amazon Mechanical Turk in 2024: Review-based AnalysisAmazon Mechanical Turk in 2024: Review-based Analysis

As AI use cases spread across different industries, many organizations are working with partners and AI service providers to develop machine learning models or natural language processing (NLP) systems. In this pursuit, they require data to fuel AI projects. AI data services have become popular in the past few years (Figure 1). Amazon Mechanical Turk is one such company that fulfills businesses’ data needs. 

However, recent news has shed light on some problems with the company1, which is why business leaders might want to re-evaluate the company before investing in it.

This article offers an in-depth evaluation of Amazon Mechanical Turk to help you make an informed decision. If you are already interested in learning about Amazon Mechanical Turk alternatives, check out this article offering 3 alternatives to MTurk.

Figure 1. Global traffic of the keyword ‘AI data services’

Recent News About Amazon Mechanical Turk

Before we delve into the nitty-gritty details, let’s first catch up on recent news and updates about the company.

1. Difficult working conditions

A recent news article2 on an interview with Krystal Kauffman, lead organizer, Turkopticon, highlighted the difficult working conditions for workers at Amazon Mechanical Turk. This can raise concerns with the clients who trying to implement fair trade practices.

2. MTurkers using automated tools

According to a recent study3, around ~33 to 46% of workers on Amazon Mechanical Turk’s platform or MTurkers are using large language models (LLMs) to automate tasks that are specifically meant to be done by humans. This can create issues for clients’ AI models that require human-generated data to be trained.

3. Worker pay less than the U.S. minimum wage

Another recent study

by the University of Massachusetts reported that MTurk’s worker network, which is mainly based in the United States (~75%), is paid a median of <$3 per hour – about ~40% less than the mandated federal minimum wage of >$7 per hour.

4. Significantly lower crowd size than claimed

MTurk claims on its website that its network of workers is over 500,0004, but according to a recent report, there are only around 100,0005 workers are available for work on the platform.

Amazon Mechanical Turk Performance Evaluation

Before committing to any tech partner, it is important to learn about its performance from the customer’s point of view. One of the most direct ways to do that is through the ratings it receives from third-party platforms. These ratings provide valuable insights into the quality of tasks, approval rate, and reliability of the company’s offerings.

In this section, we highlight some ratings and reviews of customers gathered from B2B review platforms to help you get an overall picture of the performance of MTurk.

User ratings of MTurk

To evaluate the user ratings for MTurk, we only considered platforms that had more than 20 reviews.

Table 1. MTurk user ratings comparison

PlatformUser ratings
out of 5
Number of
reviews
G24.129
Trustpilot1.955
Sitejabber2.663
Average score:2.8

Customer reviews

This section highlights some positive and negative customer reviews of MTurk identified from third-party B2B review platforms listed in the previous table. We analyzed all the reviews and picked the relevant ones.

Positive reviews of MTurk

1. Regarding the data collection service

A customer who uses Mturk for data collection found the platform to be quick, efficient, and user-friendly6.

Amazon Mechanical Turk positive customer review on data collection from G2.
2. Regarding conducting surveys through MTurk

The customer found the platform to be more professional than other survey platforms7.

Amazon Mechanical Turk positive customer review on surveys from G2.

Negative reviews of MTurk

1. Aggregate of MTurk reviews by workers from Trustpilot

We analyzed around 55 reviews of MTurk workers and identified that while there are some workers who were satisfied with the platform, most were not. The concerns revolved around low rates of pay, account suspension, scams on the platform, and poor management of the company8. Here is a visual summary of the reviews:

A screenshot of a compilation of worker reviews of amazon mechanical turk from Trustpilot. 71% of the reviews are 1-star.
2. Poor quality of workers

One customer identified that the quality of the work provided by MTurkers is sometimes low9.

Negative review of Amazon mechanical turk from G2 regarding low quality work.
3- Concerns regarding communication with MTurkers

A customer highlighted that it can be difficult to contact the MTrukers who have completed your HITs for a follow-up10.

A negative comment about poor communication with the workers in the amazon mechanical turk platform from G2.

What is Amazon Mechanical Turk?

Amazon Mechanical Turk, or MTurk, is a crowdsourcing marketplace that enables businesses to outsource processes and jobs to a distributed workforce. These jobs, known as human intelligence tasks (HITs), are well-suited for humans but challenging for computers. Examples range from data validation and survey responses to experimental research and writing product descriptions.

MTurk serves as a resource for a variety of users, including academic researchers, businesses, tech developers, market researchers, etc. Tech developers, for example, often use the platform to gather/generate AI training data and obtain annotation services.

Services and Offerings of Amazon Mechanical Turk

This section highlights some of the offerings of MTurk.

1. Human intelligence tasks (HITs)

The bread and butter of MTurk are its Human Intelligence Tasks. These tasks can be anything from answering survey questions to annotating images with relevant information. The vast majority of these tasks are designed to be completed quickly as they are micro-tasks.

2. Data collection

MTurk’s platform also works as an AI training data collection service where developers can gather relevant human-generated data on demand.

3. Experimental research

One of the offerings of Mechanical Turk is experimental research. Academic researchers can use Mechanical Turk as a platform for conducting experiments that require human-annotated data or participant responses.

4. Business processes

Companies can also use MTurk to augment their business processes. The platform offers services like data entry, customer feedback, and writing product descriptions. MTurk also offers API integration to allow businesses to directly work with the platform.

5. Payment and compensation

Mechanical Turk offers a payment system that allows requesters to pay workers either a fixed amount or based on the quality and quantity of work completed.

Amazon Mechanical Turk Alternatives

Before investing in MTurk, it can be beneficial to consider its alternatives. This section highlights some of the key Amazon Mechanical Turk alternatives you can consider.

  1. Clickworker: Clickworker is an AI data service provider that works with a crowdsourcing model with a global network of over 4.5 million workers.
  2. Appen: This popular AI data service provider offers AI data services through a crowdsourcing platform with a global network of over 1 million workers.
  3. Telus International: Offer AI data services through a global network of over 1 million workers.

Further Reading

External Resources

Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on

Shehmir Javaid
Shehmir Javaid is an industry analyst in AIMultiple. He has a background in logistics and supply chain technology research. He completed his MSc in logistics and operations management and Bachelor's in international business administration From Cardiff University UK.

Next to Read

Comments

Your email address will not be published. All fields are required.

0 Comments