Top 5 MTurk Survey Participant Recruitment Alternatives in '24
Recruiting participants for your consumer or market research can be challenging. Thanks to crowdsourcing platforms such as Amazon Mechanical Turk, you can find the right target for your survey, request the crowd to answer your questions, and get the results quickly to grasp their thoughts.
However, even though it seems like a seamless process, there may be some drawbacks to recruiting participants for your surveys, such as the reliability of the responses or data quality, that directly affect the results.
In this article, we highlight
- Some statistics of Amazon Mechanical Turk
- Challenges of using MTurk to recruit survey participants
- The top 5 alternatives to MTurk in recruiting participants for surveys.
How does Amazon Mechanical Turk work?
Amazon Mechanical Turk or MTurk is one of the online survey participant recruitment platforms where individuals or businesses can create a task to be completed by the crowdsource signed up in the platform, and MTurkers are found to be more representative of the population than other online survey platforms.
To learn more about the other usages of crowdsourcing, check our article on crowdsourcing sentiment analysis.
Although Amazon has announced more than 500,000 respondents, studies show that there are stable 100,000 to 200,000 active workers on the platform, and 10,000 new MTurkers sign up for the platform every year.
Top 5 alternatives to Amazon Mechanical Turk Survey Participants Recruitment Service
We conducted a research on the available survey participants recruitment services on the market. Here you can see the results of our study in the table below.
Vendors | Reviews* | Employee size | Free trial | Survey respondent size | Languages offered | Multi-channel survey distribution | Customer support |
---|---|---|---|---|---|---|---|
resonio by Clickworker | 69 | 1,009 | ✅ | 6M+ | 45 | Web-based panel, Android or iOS | 24/7 phone support |
Qualtrics CoreXM | 3,682 | 5,780 | ✅ | NA | 50 | Web-based panel | 24/7 live support |
User Interviews | 826 | 172 | ✅ | 2.4M | English only | Web-based panel, Android or iOS | NA |
Amazon Mechanical Turk | 28 | 10,000+ | ✖️ | 500K | Not provided | Web-based panel | Through e-mail |
Pollfish | 232 | 64 | ✖️ | 250M+ | 50 | Web-based panel, Android or iOS | 24/7 live support |
SurveyMonkey Audience | 29,477 | 1,361 | ✖️ | 175M | 16 | Web-based panel | 24/7 phone support |
If interested, you can also check out our data-driven list of survey participant recruitment services and survey tools.
Challenges of using Amazon Mechnical Turk for recruiting participants
Much research has been conducted on the reliability of MTurk survey responses, and a study claims that 10% of respondents are responsible for 40% of the responses on the platform. It has also been shown that the data quality of respondents has significantly decreased since 2018.
Here, we review the challenges of Amazon MTurk and propose some solutions:
1- Speeders
Some respondents usually do not carefully read the questions and use little effort to provide the proper answer. New research reveals that Among MTurk workers who have a 98% of approval rate, almost 25% of them could not pass at least one of the attentional check questions in the surveys. While investigating the reason for the inattentiveness, the same study found that prior experience with the technology significantly decreases attention.
Solution/recommendation: Implement attentional check questions and remove those who fail to pass them. You can also conduct post-hoc statistical analysis to detect fraudulent respondents.
2- Cheaters
A study showed that 5% to 7% of participants intentionally provide wrong or misleading responses. Some respondents also claim to be in another country while, in reality, they are not. Studies also show that this may take up to 10% percent of respondents, and the same study showed that they change their locations using a VPN to meet the survey requirements.
Solution/recommendation: Track the response time in seconds per item (SPI) using a tool as an important validity indicator and integrate tools into your survey to detect suspected VPN IP addresses to prevent taking the survey with those addresses.
3- Bot accounts
Contrary to MTurk’s mission and business model, which relies on human intelligence in solving tasks, sometimes bots are used to complete the surveys automatically, and creating distortions in the results.
Solution/recommendation: Utilize reCAPTCHA at the beginning or at the end of the survey to detect bot accounts.
4- Participants who take the survey multiple times
Research on Amazon MTurk showed that almost 20% of respondents take the surveys multiple times, decreasing the reliability of the survey responses.
Solution/recommendation: Disable the same IP addresses filling the survey more than once by implementing third-party tools.
Transparency statement:
AIMultiple serves numerous emerging tech companies, including resonio by Clickworker.
Further Reading
- 4 SurveyMonkey Audience Alternatives
- Top 4 Qualtrics Competitors
- Top 5 Pollfish Alternatives
- Market Survey Research: Benefits, 3 Use Cases & Tips
- Online Survey Analytics: How It Works & 5 Essential Tips
Here is also our data-driven list of market research tools.
Please contact us if you have any additional questions about conducting surveys:
Comments
Your email address will not be published. All fields are required.