AIMultiple ResearchAIMultiple Research

Top 5 MTurk Survey Participant Recruitment Alternatives in '24

Recruiting participants for your consumer or market research can be challenging. Thanks to crowdsourcing platforms such as Amazon Mechanical Turk, you can find the right target for your survey, request the crowd to answer your questions, and get the results quickly to grasp their thoughts. 

However, even though it seems like a seamless process, there may be some drawbacks to recruiting participants for your surveys, such as the reliability of the responses or data quality, that directly affect the results.

In this article, we highlight

  • Some statistics of Amazon Mechanical Turk
  • Challenges of using MTurk to recruit survey participants
  • The top 5 alternatives to MTurk in recruiting participants for surveys.

How does Amazon Mechanical Turk work?

Amazon Mechanical Turk or MTurk is one of the online survey participant recruitment platforms where individuals or businesses can create a task to be completed by the crowdsource signed up in the platform, and MTurkers are found to be more representative of the population than other online survey platforms. 

To learn more about the other usages of crowdsourcing, check our article on crowdsourcing sentiment analysis.

Although Amazon has announced more than 500,000 respondents, studies show that there are stable 100,000 to 200,000 active workers on the platform, and 10,000 new MTurkers sign up for the platform every year.

Top 5 alternatives to Amazon Mechanical Turk Survey Participants Recruitment Service

We conducted a research on the available survey participants recruitment services on the market. Here you can see the results of our study in the table below.

VendorsReviews*Employee sizeFree trialSurvey respondent sizeLanguages offeredMulti-channel survey distributionCustomer support
resonio by Clickworker691,0096M+45Web-based panel, Android or iOS24/7 phone support
Qualtrics CoreXM3,6825,780NA50Web-based panel24/7 live support
User Interviews8261722.4MEnglish onlyWeb-based panel, Android or iOSNA
Amazon Mechanical Turk2810,000+✖️500KNot providedWeb-based panelThrough e-mail
Pollfish23264✖️250M+50Web-based panel, Android or iOS24/7 live support
SurveyMonkey Audience29,4771,361✖️175M16Web-based panel24/7 phone support

If interested, you can also check out our data-driven list of survey participant recruitment services and survey tools.

Challenges of using Amazon Mechnical Turk for recruiting participants

Much research has been conducted on the reliability of MTurk survey responses, and a study claims that 10% of respondents are responsible for 40% of the responses on the platform. It has also been shown that the data quality of respondents has significantly decreased since 2018.

Here, we review the challenges of Amazon MTurk and propose some solutions:

1- Speeders

Some respondents usually do not carefully read the questions and use little effort to provide the proper answer. New research reveals that Among MTurk workers who have a 98% of approval rate, almost 25% of them could not pass at least one of the attentional check questions in the surveys. While investigating the reason for the inattentiveness, the same study found that prior experience with the technology significantly decreases attention.

Solution/recommendation: Implement attentional check questions and remove those who fail to pass them. You can also conduct post-hoc statistical analysis to detect fraudulent respondents.

2- Cheaters

A study showed that 5% to 7% of participants intentionally provide wrong or misleading responses. Some respondents also claim to be in another country while, in reality, they are not. Studies also show that this may take up to 10% percent of respondents, and the same study showed that they change their locations using a VPN to meet the survey requirements.

Solution/recommendation: Track the response time in seconds per item (SPI) using a tool as an important validity indicator and integrate tools into your survey to detect suspected VPN IP addresses to prevent taking the survey with those addresses. 

3- Bot accounts

Contrary to MTurk’s mission and business model, which relies on human intelligence in solving tasks, sometimes bots are used to complete the surveys automatically, and creating distortions in the results. 

Solution/recommendation: Utilize reCAPTCHA at the beginning or at the end of the survey to detect bot accounts.

4- Participants who take the survey multiple times

Research on Amazon MTurk showed that almost 20% of respondents take the surveys multiple times, decreasing the reliability of the survey responses.

Solution/recommendation: Disable the same IP addresses filling the survey more than once by implementing third-party tools.

Transparency statement:

AIMultiple serves numerous emerging tech companies, including resonio by Clickworker.

Further Reading

Here is also our data-driven list of market research tools.

Please contact us if you have any additional questions about conducting surveys:

Find the Right Vendors
Access Cem's 2 decades of B2B tech experience as a tech consultant, enterprise leader, startup entrepreneur & industry analyst. Leverage insights informing top Fortune 500 every month.
Cem Dilmegani
Principal Analyst
Follow on

Begüm Yılmaz
Begüm is an Industry Analyst at AIMultiple. She holds a bachelor's degree from Bogazici University and specializes in sentiment analysis, survey research, content writing services and Customer Relationship Management (CRM) systems.

Next to Read

Comments

Your email address will not be published. All fields are required.

0 Comments