The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment
Open accessAbstract
Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon's Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Previous research found a statistically significant difference in results between MTurk participants and participants recruited using more traditional methods, but those differences lacked practical significance [63].”
From Internalized Racism and Mental Health: The Moderating Role of Collective Racial Self-Esteem (2024) · cites this paper as [63]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
(1963) · Medical Entomology and Zoology · Cited by 15,085
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Constants across cultures in the face and emotion.
Ekman, P., & Friesen, W. V. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030377 · Cited by 5,509
WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. J. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,397
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941
Research works printed but not linked: 1
- Bartneck C, Obaid M, Zawieska K (2013) Agents with faces - what can we learn from lego minfigures. In: 1st International Conference on Human-Agent Interaction. URL http://bartneck.de/publications/2013/agentsWithFaces/bartneckLEGOAgent.pdf.
Cited by
Sorted by citations
Betsch, C., Schmid, P., Heinemeier, D., Korn, L., Holtmann, C., & Böhm, R. (2018) · PLoS ONE · doi:10.1371/journal.pone.0208601 · Cited by 1,276
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
Americans’ COVID-19 stress, coping, and adherence to CDC guidelines
Park, C. L., Russell, B. S., Fendrich, M., Finkelstein-Fox, L., Hutchison, M., & Becker, J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05898-9 · Cited by 728
Russell, B. S., Hutchison, M., Tambling, R. R., Tomkunas, A. J., & Horton, A. L. (2020) · Child Psychiatry & Human Development · doi:10.1007/s10578-020-01037-x · Cited by 778
Can we control it? Autonomous robots threaten human identity, uniqueness, safety, and resources
Złotowski, J. A., Yogeeswaran, K., & Bartneck, C. (2016) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2016.12.008 · Cited by 418
Self-citation
Metadata
- Authors
- Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K.
- Affiliations
- University of Canterbury (NZ)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Łukasiewicz Research Network - Industrial Research Institute for Automation and Measurements (PL)
- Corresponding author
- Bartneck, Christoph
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-04-14
- Volume
- 10
- Issue
- 4
- Pages
- e0121595
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4397064
- OpenAlex
- W2138247806
- Cited by
- 317
- References
- 49
- Citations by year
- 2026: 5 · 2025: 19 · 2024: 22 · 2023: 21 · 2022: 31 · 2021: 43 · 2020: 59 · 2019: 33 · 2018: 28 · 2017: 29 · 2016: 19 · 2015: 7
- FWCI
- 41.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UC Research Repository (University of Canterbury) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Keywords
- Amazon Mechanical Turk, participant recruitment, online surveys, response similarity, online forums, crowdsourcing
- MeSH terms
- Culture, Emotions, Employment, Facial Expression, Humans, Judgment, New Zealand, Personnel Selection, Surveys and Questionnaires, Research Design, Universities, Internet, Remuneration, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
