The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment
Open accessAbstract
Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon's Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Previous research found a statistically significant difference in results between MTurk participants and participants recruited using more traditional methods, but those differences lacked practical significance [63].”
From Internalized Racism and Mental Health: The Moderating Role of Collective Racial Self-Esteem (2024) · cites this paper as [63]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
(1963) · Medical Entomology and Zoology · Cited by 15,085
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Constants across cultures in the face and emotion.
Ekman, P., & Friesen, W. V. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030377 · Cited by 5,509
WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. J. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,397
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941
Talking about tactile experiences
Obrist, M., Seah, S. A., & Subramanian, S. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466220 · Cited by 1,634
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Crowdsourcing graphical perception
Heer, J., & Bostock, M. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 819
Rand, D. G. (2011) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 699
Methods for Evaluating Interactive Information Retrieval Systems with Users
Kelly, D. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000012 · Cited by 693
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · ACM Conference on Electronic Commerce (EC) · doi:10.1145/1807342.1807376 · Cited by 612
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · doi:10.1145/1600150.1600175 · Cited by 562
Reips, U. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50005-8 · Cited by 493
The Online Laboratory: Conducting Experiments in a Real Labor Market
Horton, J. J., Rand, D. G., & Zeckhauser, R. (2010) · National Bureau of Economic Research · doi:10.3386/w15961 · Cited by 402
A mixed‐effects multinomial logistic regression model
Hedeker, D. (2003) · Statistics in Medicine · doi:10.1002/sim.1522 · Cited by 373
Vehtari, A. (2002) · Journal of the Royal Statistical Society Series B (Methodological) · Cited by 333
Sue, V. M., & Ritter, L. A. (2008) · Development in Practice · doi:10.1080/09614520802031274 · Cited by 315
Sue, V. M., & Ritter, L. A. (2007) · Sage eBooks · doi:10.4135/9781412983754 · Cited by 278
USING ELECTRONIC MAIL FOR DATA COLLECTION IN ORGANIZATIONAL RESEARCH.
Sproull, L. S. (1986) · Academy of Management Journal · doi:10.2307/255867 · Cited by 237
Technology Trends in Survey Data Collection
Couper, M. P. (2005) · Social Science Computer Review · doi:10.1177/0894439305278972 · Cited by 200
From Mice to Men - 24 Years of Evaluation in CHI
Barkhuus, L., & Rode, J. A. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.2180963 · Cited by 180
Bayesian inference for categorical data analysis
Agresti, A., & Hitchcock, D. B. (2005) · Stat. Methods Appl. · doi:10.1007/s10260-005-0121-y · Cited by 174
Crowdsourcing performance evaluations of user interfaces
Komarov, S., Reinecke, K., & Gajos, K. Z. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470684 · Cited by 157
Bayesian Measures of Model Complexity and Fit
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00353 · Cited by 151
Hultsch, D. F., MacDonald, S. W. S., Hunter, M. A., Maitland, S. B., & Dixon, R. A. (2002) · International Journal of Behavioral Development · doi:10.1080/01650250143000247 · Cited by 150
Pixels vs. Paper: Comparing Online and Traditional Survey Methods in Sport Psychology
Lonsdale, C., Hodge, K. P., & Rose, E. A. (2006) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.28.1.100 · Cited by 107
Computer-Assisted Self-Interviewing Tailored for Special Populations and Topics
de Leeuw, E. D., Hox, J. J., & Kef, S. (2003) · Field Methods · doi:10.1177/1525822x03254714 · Cited by 86
The Web Experiment Method: Advantages, disadvantages, and solutions
Reips, U. (2000) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-19760 · Cited by 85
Do Mechanical Turks dream of square pie charts?
Kosara, R., & Ziemkiewicz, C. (2010) · doi:10.1145/2110192.2110202 · Cited by 74
Computerized Questionnaires and the Quality of Survey Data
Hanscom, B. S., Lurie, J. D., Homa, K. A., & Weinstein, J. N. (2002) · Spine · doi:10.1097/00007632-200208150-00020 · Cited by 68
POTTER, T., & CORNEILLE, O. (2008) · Psychonomic Bulletin & Review · doi:10.3758/pbr.15.3.615 · Cited by 58
Questionnaire mode effects in interactive information retrieval experiments
Kelly, D., Harper, D. J., & Landau, B. J. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.02.007 · Cited by 56
EXPLORING TASK PROPERTIES IN CROWDSOURCING – AN EMPIRICAL STUDY ON MECHANICAL TURK
Schulze, T., Seedorf, S., Geiger, D., Kaufmann, N., & Schader, M. (2011) · MADOC (University of Mannheim) · Cited by 46
Behavioral Research and Data Collection via the Internet
Birnbaum, M. H., & Reips, U. (2005) · Cited by 42
USING INTERNET-BASED SURVEYS FOR ACADEMIC RESEARCH: OPPORTUNITIES AND PROBLEMS
Roztocki, N. (2001) · Cited by 40
Employee Reactions to Paper and Electronic Surveys: An Experimental Comparison
Croteau, A., Dyer, L., & Miguel, M. (2010) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2010.2052852 · Cited by 34
Bulmer, M. (2004) · SAGE Publications Ltd eBooks · doi:10.4135/9781446263204 · Cited by 31
Electronic versus paper surveys: analysis of potential psychometric biases
McCoy, S., Marks, P. V., Carr, C. L., & Mbarika, V. (2004) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2004.1265634 · Cited by 15
The online laboratory: conducting experiments in a real labor market
Horton, J. J., Rand, D. G., & Zeckhauser, R. (2011) · Experimental Economics · doi:10.1007/s10683-011-9273-9 · Cited by 9
Facial caricature generation using a quadratic deformation model
Obaid, M., Lond, D., Mukundan, R., & Billinghurst, M. (2009) · International Conference on Advances in Computer Entertainment Technology (ACE) · doi:10.1145/1690388.1690437 · Cited by 8
Electronic methods of collecting survey data: a review of 'E-Research'
Tuten, T. L. (1997) · Social Science Open Access Repository (GESIS – Leibniz Institute for the Social Sciences) · Cited by 5
Flagged first, then by citations
2 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Bartneck C, Obaid M, Zawieska K (2013) Agents with faces - what can we learn from lego minfigures. In: 1st International Conference on Human-Agent Interaction. URL http://bartneck.de/publications/2013/agentsWithFaces/bartneckLEGOAgent.pdf.
Cited by
Sorted by citations
Betsch, C., Schmid, P., Heinemeier, D., Korn, L., Holtmann, C., & Böhm, R. (2018) · PLoS ONE · doi:10.1371/journal.pone.0208601 · Cited by 1,276
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
Americans’ COVID-19 stress, coping, and adherence to CDC guidelines
Park, C. L., Russell, B. S., Fendrich, M., Finkelstein-Fox, L., Hutchison, M., & Becker, J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05898-9 · Cited by 728
Russell, B. S., Hutchison, M., Tambling, R. R., Tomkunas, A. J., & Horton, A. L. (2020) · Child Psychiatry & Human Development · doi:10.1007/s10578-020-01037-x · Cited by 778
Can we control it? Autonomous robots threaten human identity, uniqueness, safety, and resources
Złotowski, J. A., Yogeeswaran, K., & Bartneck, C. (2016) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2016.12.008 · Cited by 418
Self-citation
Metadata
- Authors
- Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K.
- Affiliations
- University of Canterbury (NZ)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Łukasiewicz Research Network - Industrial Research Institute for Automation and Measurements (PL)
- Corresponding author
- Bartneck, Christoph
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-04-14
- Volume
- 10
- Issue
- 4
- Pages
- e0121595
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4397064
- OpenAlex
- W2138247806
- Cited by
- 317
- References
- 49
- Citations by year
- 2026: 5 · 2025: 19 · 2024: 22 · 2023: 21 · 2022: 31 · 2021: 43 · 2020: 59 · 2019: 33 · 2018: 28 · 2017: 29 · 2016: 19 · 2015: 7
- FWCI
- 41.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UC Research Repository (University of Canterbury) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Keywords
- Amazon Mechanical Turk, participant recruitment, online surveys, response similarity, online forums, crowdsourcing
- MeSH terms
- Culture, Emotions, Employment, Facial Expression, Humans, Judgment, New Zealand, Personnel Selection, Surveys and Questionnaires, Research Design, Universities, Internet, Remuneration, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
