Comparing the Reliability of Amazon Mechanical Turk and Survey Monkey to Traditional Market Research Surveys
Abstract
In the product design process, it is often desirable to quickly obtain information about current user behaviors for topics that cannot be obtained through existing data or instrumentation. Perhaps we would like to understand the use of products we do not have access to or perhaps the action we would like to know about (such as using a coupon) is an action taken outside of a system that can be instrumented. Traditionally, large market research surveys would be conducted to answer these questions, but often designers need answers much faster. We present a study investigating the reliability of fast survey platforms such as Amazon Mechanical Turk and Survey Monkey as compared to larger market research studies for technology behavior research and show that results can be obtained in hours for much smaller costs with accuracy within 10% of traditional larger surveys. This demonstrates that we can rely more heavily on these platforms in the product design process and provide much faster planning iterations that are informed by actual usage data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Studies have shown the validity of using SurveyMonkey to conduct surveys used in academic research.
“Studies have shown the validity of using this platform to conduct surveys used in academic research, in addition to the numerous studies which use such platforms for conducting surveys globally56–60.”
From Perception, performance, and detectability of conversational artificial intelligence across 32 university courses (2023) · cites this paper as [56]
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 17,604 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment.
Tversky, A., & Kahneman, D. (1983) · Psychological Review · doi:10.1037/0033-295x.90.4.293 · Cited by 3,991 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 2× · see citing works
“showed that the diversity within MTurk samples is similar to that of the population at large [10].”
Result"I thought she would like to read it"
Bentley, F. R., Peesapati, S. T., & Church, K. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858056 · Cited by 17 · see citing works
Self-citationTechnology assessment and risk analysis
Nehněvajsa, J., & Menkes, J. (1981) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(81)90057-3 · Cited by 13 · see citing works
Flagged first, then by citations
1 of the 12 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Music, Search, and IoT: How People (Really) Use Voice Assistants
Ammari, T., Kaye, J., Tsai, J. Y., & Bentley, F. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3311956 · Cited by 356 · see citing works
“[8] has shown that using samples of participants from Mechanical Turk can be reliable in understanding technology use when compared to large-scale professional market research surveys or the analysis of usage logs held by large corporations.”
Self-citationUnderstanding the Long-Term Use of Smart Speaker Assistants
Bentley, F., Luvogt, C., Silverman, M., Wirasinghe, R., White, B., & Lottridge, D. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264901 · Cited by 316 · see citing works
“Larger households (of 3 or more) use the device proportionally more on weekends compared to smaller households (M householdsize1 = .238, M householdsize2 =.265, M househouldsize3+ = .380; F [2,58] = 4.651, p=.014).”
BackgroundInfluentialSelf-citationPersonalizing Persuasive Strategies in Gameful Systems to Gamification User Types
Orji, R., Tondello, G. F., & Nacke, L. E. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174009 · Cited by 230 · see citing works
Ibrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154 · see citing works
“Studies have shown the validity of using this platform to conduct surveys used in academic research, in addition to the numerous studies which use such platforms for conducting surveys globally 56–60 .”
Jr. Torrentira Moises, C. (2020) · European Journal of Education Studies · doi:10.46827/ejes.v7i11.3336 · Cited by 134 · see citing works
“Other tools that have been used by researchers such as Amazon Mechanical Turk and Survey Monkey for market surveys (Bentley et al, 2017), Survey Gizmo (Abd Halim et al., 2018), Research Now SSI, and Opinion Access (Kimball, 2019) among others.”
MethodThe Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123 · see citing works
“Granted there are other ways to operationalize this group (e.g., recruiting AI practitioners), as a frst step, we chose students because it allows us to explore how their AI coursework impacts the way they make sense of explanations from AI systems.”
BackgroundThe Majoritarian Threat to Liberal Democracy
Grossman, G., Kronick, D., Levendusky, M. S., & Meredith, M. N. (2021) · Journal of Experimental Political Science · doi:10.1017/xps.2020.44 · Cited by 102 · see citing works
Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
Shi, W., Wang, X., Oh, Y. J., Zhang, J., Sahay, S., & Yu, Z. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376843 · Cited by 93 · see citing works
“AMT has been frequently used in social science studies [34, 2, 21, 33, 35, 4, 27] and is suitable for our experiments.”
MethodCharnley, F., Knecht, F., Muenkel, H., Pletosu, D., Rickard, V. A., Sambonet, C., Schneider, M., & Zhang, C. (2022) · Sustainability · doi:10.3390/su14084589 · Cited by 100 · see citing works
“The limitations of crowdsourcing survey platforms are associated mainly with whether they are truly representative and can be considered a reliable source of decision-making data [48].”
BackgroundThe Adoption Analysis of Voice-Based Smart IoT Products
Pal, D., Arpnikanondt, C., Funilkul, S., & Chutimaskul, W. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2991791 · Cited by 89 · see citing works
“Previous research in [65] has shown that using samples of participants from MTurk are more reliable in understanding technology use when compared to large-scale professional surveys.”
ResultTsourela, M., & Nerantzaki, D.-M. (2020) · Future Internet · doi:10.3390/fi12110191 · Cited by 79 · see citing works
“MTurk was considered to be the best option for the current research, which investigates consumers’ technology perceptions across a broader population, which is in line with the type of study that it is good at handling [48].”
BackgroundOrji, R., Reilly, D. F., Oyibo, K., & Orji, F. A. (2018) · Behaviour and Information Technology · doi:10.1080/0144929x.2018.1520302 · Cited by 61 · see citing works
“AMT is suitable for our study, which investigates technology preferences across a broader population in line with the type of study that AMT is good at handling (Bentley, Daskalova, and White 2017).”
MethodUnderstanding Online News Behaviors
Bentley, F. R., Quehl, K., Wirfs-Brock, J., & Bica, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300820 · Cited by 55 · see citing works
“However, previous research has shown large agreement between the behaviors of this panel and the broader US population [3].”
BackgroundResultSelf-citationExploring the Use of Large Language Models for Improving the Awareness of Mindfulness
Kumar, H., Wang, Y., Shi, J., Musabirov, I., Farb, N. A. S., & Williams, J. J. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3585614 · Cited by 50 · see citing works
“We recruit participants from Amazon Mechanical Turk (AMT), which has been used frequently in social and behavioral science studies [3, 7].”
MethodOffenwanger, A., Milligan, A. J., Chang, M., Bullard, J., & Yoon, D. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445383 · Cited by 52 · see citing works
“Our qualitative interviews with HCI researchers and background reading revealed two problematic aspects to gender balancing, which is a model of gender equity that has been used for many years in good faith [6, 15].”
BackgroundConversations and idea generation: Evidence from a field experiment
Hasan, S., & Koning, R. M. (2019) · Research Policy · doi:10.1016/j.respol.2019.103811 · Cited by 47 · see citing works
Personal Information Disclosure via Voice Assistants: The Personalization–Privacy Paradox
Pal, D., Arpnikanondt, C., & Razzaque, M. A. (2020) · SN Computer Science · doi:10.1007/s42979-020-00287-9 · Cited by 45 · see citing works
“Previous research in [88] has shown that using samples of participants from MTurk are more reliable in understanding technology use when compared to large-scale professional surveys.”
ResultLottridge, D. M., & Bentley, F. R. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173634 · Cited by 41 · see citing works
Self-citationRationing by Inconvenience: How Insurance Denials Induce Administrative Burdens
Yaver, M. (2024) · Journal of Health Politics Policy and Law · doi:10.1215/03616878-11186111 · Cited by 40 · see citing works
Climate change risk perception in the USA and alignment with sustainable travel behaviours
Fletcher, J., Higham, J. E. S., & Longnecker, N. E. (2021) · PLoS ONE · doi:10.1371/journal.pone.0244545 · Cited by 35 · see citing works
“Studies comparing participants recruited via SurveyMonkey have found responses to be within 10% comparison of traditional market research companies [51] and within a discrepancy of 5 to 10% between demographics of online respondents and their US distribution [52].”
MethodThe case for voter-centered audits of search engines during political elections
Mustafaraj, E., Lurie, E., & Devine, C. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372835 · Cited by 28 · see citing works
“However, recent research has indicated that AMT is better than or equal in terms of diversity to other survey participant pools [3].”
BackgroundJohnson, K. A., & John, B. S. (2021) · Journalism Studies · doi:10.1080/1461670x.2021.1910542 · Cited by 28 · see citing works
“…results returned by Surveymonkey are comparable to results returned by Mechanical Turk, as well as other traditional large market research study platforms, leading to the conclusion that the Surveymonkey platform is reliable (Bentley, Daskalova, and White 2017; Bentley, O’Neill, and Quehl 2020).”
ResultTunçel, N., & Kavak, B. (2021) · International Journal of Consumer Studies · doi:10.1111/ijcs.12693 · Cited by 33 · see citing works
Exploring Online Video Watching Behaviors
Bentley, F. R., Silverman, M., & Bica, M. (2019) · ACM International Conference on Interactive Experiences for TV and Online Video · doi:10.1145/3317697.3323355 · Cited by 26 · see citing works
“Previous studies have shown that MTurk samples can be quite accurate when studying technology use in the broader American population [3].”
BackgroundSelf-citationAlhasani, M., Alkhawaji, A., & Orji, R. (2022) · Human Behavior and Emerging Technologies · doi:10.1155/2022/7376748 · Cited by 32 · see citing works
Importance of Wellness Concepts in the Hotel Industry: Perspectives from the Millennials
Lee, P., Lee, M. J., & Cheng, T.-T. (2019) · Journal of Quality Assurance in Hospitality & Tourism · doi:10.1080/1528008x.2019.1616041 · Cited by 24 · see citing works
“It is also proven that using Amazon MTurk for traditional market research surveys is reliable (e.g., Bentley, Daskalova, & White, 2017).”
Bentley, F. R., O'Neill, K., Quehl, K., & Lottridge, D. M. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376671 · Cited by 22 · see citing works
“This holds for a wide variety of psychology studies including Prisoner’s Dilemma, the Asian disease problem [7], and the Linda Problem[8].”
BackgroundResultSelf-citationDye, T., Tavárez, Z. Q., Perez-Ramos, J. G., Fernandez, I. D., Vega, C. V., Ocasio, D. V., Avendaño, E., Cordero, N. R. C., DiMare, C., Dozier, A. M., et al. (2021) · Journal of Community Genetics · doi:10.1007/s12687-021-00538-z · Cited by 21 · see citing works
“reliable method for reaching a global audience for survey research (Bentley et al. 2017).”
MethodExplanatory or Dispositional Optimism: Which Trait Predicts Eco-Friendly Tourist Behavior?
Rafiq, F., Chishty, S. K., & Adil, M. (2022) · Sustainability · doi:10.3390/su14052994 · Cited by 22 · see citing works
“However, Bentley, Daskalova, and White [56] noted that MTurk helps generate accurate data relatively faster and at a lower cost.”
BackgroundWho Broke Amazon Mechanical Turk?: An Analysis of Crowdsourcing Data Quality over Time
Marshall, C. C., Goguladinne, P. S. R., Maheshwari, M., Sathe, A., & Shipman, F. M. (2023) · Proceedings of the 15th ACM Web Science Conference 2023 · doi:10.1145/3578503.3583622 · Cited by 20 · see citing works
“This type of work is perhaps the most frustrating to qualitative researchers who appreciated the type of diverse participation Mechanical Turk offered [5] [31] [33].”
BackgroundEvaluating and Revising the Digital Citizenship Scale
Connolly, R. W., & Miller, J. (2022) · Informatics · doi:10.3390/informatics9030061 · Cited by 17 · see citing works
“Sampling participants from crowdsource populations (such as SMA) has become a well-respected research practice used in many fields [28,29].”
Understanding Mass-Market Mobile TV Behaviors in the Streaming Era
Bentley, F. R., & Lottridge, D. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300491 · Cited by 17 · see citing works
“Both surveys used SurveyMonkey to reach their target audiences, a service that has shown to be accurate to within several percentage points of the American population [2] when studying technology use and is used by polling companies such as FiveThirtyEight to assess American opinions.”
MethodSelf-citationThe Who in Explainable AI: How AI Background Shapes Perceptions of AI ExplanationsPreprint
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M. J., & Riedl, M. O. (2021) · arXiv (Cornell University) · Cited by 17 · see citing works
Bartek, M. A., Truitt, A. R., Widmer-Rodriguez, S., Tuia, J. E., Bauer, Z., Comstock, B. A., Edwards, T. C., Lawrence, S. O., Monsell, S. E., Patrick, D. L., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8821 · Cited by 16 · see citing works
“Originally designed to allow the rapid completion of complex but repetitive work, MTurk has been adopted by behavioral scientists and market researchers to serve as a virtual laboratory to quickly and inexpensively administer thought experiments via online surveys, perform market research for organizations, and give i…”
BackgroundWallner, C., Kruber, S., Adebayo, S. O., Ayandele, O., Namatame, H., Olonisakin, T. T., Olapegba, P. O., Sawamiya, Y., Suzuki, T., Yamamiya, Y., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192013212 · Cited by 17 · see citing works
“However, according to the literature, we do not assume any substantial bias from this type of survey [57,58].”
BackgroundOC-2-KB: integrating crowdsourcing into an obesity and cancer knowledge base curation system
Lossio-Ventura, J. A., Hogan, W. R., Modave, F. P., Guo, Y. M., He, Z., Yang, X., Zhang, H., & Bian, J. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0635-5 · Cited by 16 · see citing works
“The Amazon MTurk platform allows anyone to become a requester in order to create HITs, while people that perform those tasks are called workers [58].”
BackgroundThe Myth of the Chilling EffectPreprint
Bedi, S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3794037 · Cited by 16 · see citing works
Coimagining the Future of Voice Assistants with Cultural Sensitivity
Seaborn, K., Sawa, Y., & Watanabe, M. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/3238737 · Cited by 15 · see citing works
“Participants were recruited online using SurveyMonkey Audience, which provides a level of quality roughly equivalent to within 10% of traditional market surveys [62].”
MethodPlatform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks
Qarout, R., Checco, A., Demartini, G., & Bontcheva, K. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5264 · Cited by 15 · see citing works
“In the crowdsourcing field, a limited number of studies have examined result consistency (Blanco et al. 2011; Sun and Stolee 2016; Bentley, Daskalova, and White 2017; Cheng et al. 2015).”
BackgroundNew data strategies: nonprobability sampling, mobile, big data
Link, M. (2018) · Quality Assurance in Education · doi:10.1108/qae-06-2017-0029 · Cited by 15 · see citing works
Pan, M., & Ryan, A. (2023) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981221143106 · Cited by 13 · see citing works
“MTurk samples have also been demonstrated to be just as diverse in relation to demographics as samples from digital mobile surveys (62).”
BackgroundSwitching behaviors in peer-to-peer mobile payment applications: the role of sociability
Mai, X. T., & Nguyen, T. (2023) · Journal of Systems and Information Technology · doi:10.1108/jsit-05-2023-0082 · Cited by 15 · see citing works
Bessenyei, K., Suruliraj, B., Bagnell, A. L., McGrath, P. J., Wozney, L., Huguet, A., Elger, B. S., Meier, S. M., & Orji, R. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2021.100134 · Cited by 14 · see citing works
Is the Privacy Paradox a Domain-Specific Phenomenon
Hirschprung, R. S. (2023) · Computers · doi:10.3390/computers12080156 · Cited by 20 · see citing works
Quality and representativeness of research online with Yahoo! Crowdsourcing
Seaborn, K., & Nakamura, S. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1588579 · Cited by 11 · see citing works
Usability of Humanly Computable Passwords
Samadi, S., Vempala, S., & Kalai, A. T. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13333 · Cited by 11 · see citing works
Online information privacy and price: A theoretical model and empirical tests
Wu, Z., & Luo, J. (2021) · Information & Management · doi:10.1016/j.im.2021.103583 · Cited by 10 · see citing works
Digital Citizenship and the Big Five Personality Traits
Roberts, M. J. D., Connolly, R. W., Conley, J., & Miller, J. (2023) · Informatics · doi:10.3390/informatics10030058 · Cited by 8 · see citing works
A YouTube Dataset with User-level Usage Data: Baseline Characteristics and Key Insights
Lall, S., Agarwal, M., & Sivakumar, R. (2020) · International Conference on Communications · doi:10.1109/icc40277.2020.9148782 · Cited by 8 · see citing works
“Previous studies have shown that mTurk samples can be accurate when studying technology use in the broader population [7].”
BackgroundImpact of Market Development Indicators on Company Performance
Gerhardt, V. J., Siluk, J. C. M., Michelin, C. D. F., Júnior, Á. L. N., & da Veiga, C. P. (2021) · IEEE Engineering Management Review · doi:10.1109/emr.2021.3133706 · Cited by 8 · see citing works
"I know what the companies are wanting more of"
Schirra, S., & Allison, C. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3188681 · Cited by 8 · see citing works
“…about crowdsourced participants has focused on the MTurk platform, with past work looking at topics such as the potential of MTurk for conducting online research [9], MTurk participant screening strategies [5], and reliability of MTurk sampling compared to traditional market research panels [1].”
BackgroundBig Tech Dominance Despite Global Mistrust
Ibrahim, H., Debicki, M., Rahwan, T., & Zaki, Y. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3339183 · Cited by 7 · see citing works
“Studies have shown the validity of using Survey Monkey for conducting market research surveys, in addition to the numerous studies which use such platforms for conducting surveys globally [38], [39], [40], [41], [42].”
MethodUnderstanding Secondary Content Practices for Television Viewing
Bentley, F. R. (2017) · ACM International Conference on Interactive Experiences for TV and Online Video · doi:10.1145/3077548.3077554 · Cited by 7 · see citing works
“In earlier work, we have found that these panels accurately represent technology usage behaviors within 7% of larger, professionallycommissioned market research samples and are accurate enough to discover which behaviors are common, prevalent, or rare in the broader American population [4].”
BackgroundSelf-citationFrom “Made In” to Mukokuseki: Exploring the Visual Perception of National Identity in Robots
Seaborn, K., Kotani, H., & Pennefather, P. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3681782 · Cited by 6 · see citing works
“The basic protocol was registered at OSF before data collection on Oct. 6 th , 2021 1 .”
Zeamer, C. A., & Estey, A. J. (2020) · Journal of Financial Counseling and Planning · doi:10.1891/jfcp-18-00056 · Cited by 7 · see citing works
A Real-world Dataset of Netflix Videos and User Watch-Behavior: Analysis and Insights
Lall, S., & Sivakumar, R. (2021) · International Conference on Communications · doi:10.1109/icc42927.2021.9500669 · Cited by 6 · see citing works
“Studies have shown that mTurk samples can be accurate when studying technology use in the broader population [5].”
BackgroundCase Law Grounding: Using Precedents to Align Decision-Making for Humans and AI
Chen, Q. Z., & Zhang, A. X. (2025) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3715928.3737487 · Cited by 4 · see citing works
Ignite the Scent: The Effectiveness of Implied Explosion in Perfume Ads
Droulers, O., Yu, J., & Lacoste‐Badie, S. (2024) · Journal of Advertising Research · doi:10.2501/jar-2024-010 · Cited by 5 · see citing works
How Do We Create a Fantabulous Password?
Woo, S. S. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380222 · Cited by 4 · see citing works
“They generally agree that online Mturk study design is an acceptable method and can match with traditional survey method results within 10% accuracy [3].”
MethodHow Active Management Survives
Heaton, J. B., & Pennington, G. L. (2018) · Financial Planning Review · doi:10.1002/cfp2.1031 · Cited by 4 · see citing works
Goode, S. (2022) · Behaviour and Information Technology · doi:10.1080/0144929x.2021.2016968 · Cited by 4 · see citing works
“The data quality on Amazon Mechanical Turk is believed to be high (Bentley, Daskalova, and White 2017; Ratcliff and Hendrickson 2021) and we estimate low errors in these domicility settings; however, the possibility of error cannot be ruled out.”
BackgroundAlhasani, M., Alshareef, A. M., & Orji, R. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2595307 · Cited by 3 · see citing works
“Prior research has demonstrated that data collected through MTurk are at least as reliable as those obtained via traditional methods, including in clinical research (Bentley et al., 2017; Buhrmester et al., 2011; Shapiro et al., 2013).”
Factors Shifting Preference Toward Telemedicine in the Delivery of Skin Cancer Reconstruction Care
Du, E. Y., Moody, R. A., Simpson, M. C., Nellore, A., Sinnwell, J. G., Heiland, L. D., Grace, S. A., & Chen, C. L. (2022) · The Laryngoscope · doi:10.1002/lary.30215 · Cited by 3 · see citing works
“It is easy to set up, simple, and multiple studies have found that the responses collected are comparable to more conventional methods, with similar or even better quality and representation of data.(20,21) However, as recruitment is online, there is bias inherent in utilizing MTurk in investigating telemedicine; in a…”
BackgroundResultMoving Toward Economic and Digital Sustainability in Marketing Analytics
Haverila, M. J., Al-Azad, M. S., Haverila, K. C., Mohiuddin, M., & Su, Z. (2024) · Journal of Global Information Management · doi:10.4018/jgim.349129 · Cited by 3 · see citing works
Lu, T., Ke, J.-Y., Prager, F., & Martínez, J. N. (2022) · doi:10.31979/mti.2022.2147 · Cited by 3 · see citing works
Revealing the Preference Falsification in Russia
Zhdanov, A., & Tseitlin, B. (2024) · Comparative Sociology · doi:10.1163/15691330-bja10123 · Cited by 2 · see citing works
PrivRo: A Privacy-Preserving Crowdsourcing Service With Robust Quality Awareness
Lian, R., Zheng, Y., & Wang, C. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3377158 · Cited by 2 · see citing works
“The requester: // RequireGen ( r ) 1: Generate the vector r ; 2: r ← F ( r ); common crowdsourcing scenario where that task is in the form of providing answers to a set of questions.”
What Makes Fake News Appeal to You? Empirical Evidence from the Tweets Related to COVID-19 Vaccines
Park, M., & Chai, S. (2024) · SAGE Open · doi:10.1177/21582440241257671 · Cited by 2 · see citing works
“The studies find out over-whelmingly supportive of the reliability, and high quality of collected data by using MTurk (Bentley et al., 2017).”
InfluentialA Food Venue Recommender System Based on Multilingual Geo-Tagged Tweet Analysis
Siriaraya, P., Nakaoka, Y., Wang, Y., & Kawai, Y. (2018) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2018.8508553 · Cited by 2 · see citing works
“This platform has shown to be an efficient means for gathering good quality data in survey research for a number of research areas [10].”
BackgroundGuadagno, R. E., & Olivieri, A. F. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009000796.004 · Cited by 1 · see citing works
Bentley, F. R. (2018) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3267305.3274109 · Cited by 1 · see citing works
“We have found that this platform is quite reliable in understanding technology use [14], but if the types of participants that use MTurk have changed in these 2.”
BackgroundSelf-citationWhat and how the public knows about the Fed
Lopus, J. S., Duzhak, E. A., & Hoff, K. J. (2025) · The Journal of Economic Education · doi:10.1080/00220485.2025.2503324 · Cited by 1 · see citing works
“3 This is a widely used program that has been found to be generally reliable and efficient (Bentley, Daskalova, and White 2017).”
Suicide Attitudes and Terror Mangement Theory
Kheibari, A. (2019) · UKnowledge (University of Kentucky) · doi:10.13023/etd.2019.297 · Cited by 1 · see citing works
Utilizing Multilingual Social Media Analysis for Food Venue Recommendation
Siriaraya, P., Wang, Y., Kawai, Y., Nakaoka, Y., & Akiyama, T. (2019) · Lecture notes in social networks · doi:10.1007/978-3-030-33698-1_3 · Cited by 1 · see citing works
Lall, S., Moravapalle, U. P., & Sivakumar, R. (2020) · ACM Multimedia Systems Conference (MMSys) · doi:10.1145/3339825.3391864 · Cited by 1 · see citing works
“Previous studies have shown that mTurk samples can be accurate when studying technology use in the broader population [24].”
BackgroundEmployer Learning, Biased Beliefs, and Labor Market Discrimination
Lepage, L. (2021) · Deep Blue (University of Michigan) · doi:10.7302/3065 · Cited by 1 · see citing works
Density and Public Transit – Victims of COVID-19 Pandemic?Preprint
Grover, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2917712/v1 · Cited by 1 · see citing works
Investigating Stability and Reliability of Crowdsourcing Output (short paper).
Qarout, R., Checco, A., & Bontcheva, K. (2018)
Measuring Psychological Effects of Peer-to-Peer Reputation Systems Involving In-Person Exchanges
Yousef, M. (2019) · University of Minnesota Digital Conservancy (University of Minnesota)
Fuza, Z. I. M., Isa, S. M., Shattar, N. A., Ismail, N., Bakar, S. K. A., & Patah, M. O. R. A. (2019) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v9-i13/6238
“Bentley et al. (2017) found that questionnaires with 40 questions have approximately 10% lower response rate than questionnaires with 10 questions only.”
BackgroundConcepção e divulgação de um e-survey para pesquisa em administração aplicada em enfermagem
Francisco, Q. A. D. S., & Martinez, M. R. (2020) · RAS Revista de administração em saúde · doi:10.23973/ras.80.246
Wallner, C., Lehnhardt, M., & Behr, B. (2022) · Aesthetic Plastic Surgery · doi:10.1007/s00266-022-02849-z
“However, studies clearly demonstrate the superiority of this new and very effective form of study [2, 3].”
BackgroundDensity and Public Transit – Victims of COVID Pandemic?Preprint
Grover, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3646372/v1
Finding Antiracists: Construction of an Antiracism Attitude Scale
Yancey, G. (2024) · Sociological Focus · doi:10.1080/00380237.2024.2330470
“Bentley, Daskalova, and White (2017) found that surveys using Amazon Mechanical Turk and Survey Money tend to underrepresent females; however, they also found that the error rate of Amazon Mechanical Turk and Survey Monkey panels was under 10 percent, although it could be higher than 5 percent.”
BackgroundWalsh, A., & Walsh, P. R. (2024) · International Journal of Energy Sector Management · doi:10.1108/ijesm-04-2024-0028
Administrative Burdens and Inequities in COBRA Take‐Up
Yaver, M. (2025) · World Medical & Health Policy · doi:10.1002/wmh3.70025
Cash for Commuting: Behavioral and Environmental Impacts of California's Parking Cash-Out PolicyPreprint
Prager, F., Lu, T., Membere, A., Chhetri, P., Cowo, M., & Kass, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5383592
Yaver, M. (2026) · doi:10.1017/9781009649803.012
Improving Reproducibility in Crowdsourced Labeling Across Platforms
Zhong, Y., Katsuno, K., Matsubara, M., & Morishima, A. (2026) · ACM Transactions on Social Computing · doi:10.1145/3812206
“With test theory, we can choose good questions after we have gotten the answers from many people to all the candidate questions.”
Alhasani, M., & Orji, R. (2026) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-026-09447-7
“MTurk has been widely recognized as a valid source for behavioural and clinical research, with prior studies confirming the reliability of data obtained through the platform (Bentley et al. 2017; Buhrmester et al. 2011; Hirsh et al. 2012; Jia et al. 2016; Shapiro et al. 2013).”
- Authors
- Bentley, F. R., Daskalova, N., & White, B.
- Affiliations
- Yahoo (United States) (US)
- Venue
- Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-06
- Pages
- 1092-1099
- OpenAlex
- W2610617162
- Cited by
- 91
- References
- 12
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 13 · 2023: 9 · 2022: 10 · 2021: 13 · 2020: 12 · 2019: 11 · 2018: 12 · 2017: 2
- FWCI
- 9.33
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- Amazon Mechanical Turk, SurveyMonkey, online survey tools, market survey, survey reliability, product design, user behavior, crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
