The DOI matches its Crossref record; 21 of the 23 references it cites were found in a registry or an index.
Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants
Open accessAbstract
Participant attentiveness is a concern for many researchers using Amazon's Mechanical Turk (MTurk). Although studies comparing the attentiveness of participants on MTurk versus traditional subject pool samples have provided mixed support for this concern, attention check questions and other methods of ensuring participant attention have become prolific in MTurk studies. Because MTurk is a population that learns, we hypothesized that MTurkers would be more attentive to instructions than are traditional subject pool samples. In three online studies, participants from MTurk and collegiate populations participated in a task that included a measure of attentiveness to instructions (an instructional manipulation check: IMC). In all studies, MTurkers were more attentive to the instructions than were college students, even on novel IMCs (Studies 2 and 3), and MTurkers showed larger effects in response to a minute text manipulation. These results have implications for the sustainable use of MTurk samples for social science research and for the conclusions drawn from research with MTurk and college subject pool samples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In crowd-sourced settings such as MTurk, respondents are often incentivized to respond as attentively, compliantly, and quickly as possible to both open higher-paying opportunities and prevent being rejected from future employment [90, 100].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [100]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Comparable attention checks have been shown to improve the quality of data obtained through web-based surveys above that of classic paper and pencil questionnaires [57].”
From Dispositional factors in the explanation of symptoms of depression, anxiety, health anxiety and COVID-19 Phobia (2024) · cites this paper as [57]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Advances In Experimental Social Psychology
(2022) · Advances in experimental social psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,559
Mental Accounting and Consumer Choice
Thaler, R. (1985) · Marketing Science · doi:10.1287/mksc.4.3.199 · Cited by 4,885
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Mental Accounting and Consumer Choice
Thaler, R. H. (2008) · Marketing Science · doi:10.1287/mksc.1070.0330 · Cited by 3,477
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Mental Accounting and Consumer Choice / Теория ментального учета и потребительский выбор
Thaler, R. H. (1985) · Marketing Science · Cited by 1,296
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Judgment in a Social Context: Biases, Shortcomings, and the Logic of Conversation
Schwarz, N. (1994) · Advances in experimental social psychology · doi:10.1016/s0065-2601(08)60153-7 · Cited by 373
Self-citationEmotion rating from short blog texts
Gill, A. J., Gergle, D., French, R. M., & Oberlander, J. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357229 · Cited by 232
Clifford, S., & Jerit, J. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.5 · Cited by 208
Encoding Under Trust and Distrust: The Spontaneous Activation of Incongruent Cognitions.
Schul, Y., Mayo, R., & Burnstein, E. (2004) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.86.5.668 · Cited by 167
Mayo, R., Alfasi, D., & Schwarz, N. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0035127 · Cited by 43
Self-citation
Research works printed but not linked: 1
- Kapelner, A., & Chandler, D. (2010). Preventing satisficing in online surveys: A “kapcha” to ensure higher data quality. In Proceedings of CrowdConf 2010. Available at www.academia.edu/2788541/Preventing_Satisficing_in_Online_Surveys
Cited by
Sorted by citations
Gorilla in our midst: An online behavioral experiment builder
Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N. Z., & Evershed, J. K. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01237-x · Cited by 2,087
Do Investors Value Sustainability? A Natural Experiment Examining Ranking and Fund Flows
HARTZMARK, S. M., & SUSSMAN, A. B. (2019) · The Journal of Finance · doi:10.1111/jofi.12841 · Cited by 1,711
Methods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
Metadata
- Authors
- Hauser, D. J., & Schwarz, N.
- Affiliations
- University of Michigan (US)
- University of Southern California (US)
- Corresponding author
- Hauser, David J.
- Venue
- Behavior Research Methods
- Published
- 2015-03-12
- Volume
- 48
- Issue
- 1
- Pages
- 400-407
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2076387137
- Cited by
- 1,739
- References
- 23
- Citations by year
- 2026: 60 · 2025: 104 · 2024: 145 · 2023: 180 · 2022: 209 · 2021: 249 · 2020: 248 · 2019: 176 · 2018: 160 · 2017: 122 · 2016: 64 · 2015: 19
- FWCI
- 184.66
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Behavior Research Methods (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Amazon Mechanical Turk, university students
- MeSH terms
- Attention, Data Collection, Female, Humans, Learning, Male, Surveys and Questionnaires, Social Sciences, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
