The DOI matches its Crossref record; 18 of the 20 references it cites were found in a registry or an index.
Inattentive Responding in MTurk and Other Online Samples
Abstract
The focal article by Landers and Behrend (2015) makes the case that samples collected on microtask websites like Amazon's Mechanical Turk (MTurk) are inherently no better or worse than traditional samples of convenience from university students or organizations. We wholeheartedly agree. However, having successfully used MTurk and other online sources for data collection, we feel that the focal article was insufficient regarding the caution required in identifying inattentive respondents and the problems that can arise if such individuals are not removed from the dataset. Although we focus on MTurk, similar issues arise for most “low-stakes” assessments, including student samples, which seem to be increasingly collected online.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous research also demonstrates that MTurk data is as reliable as other common convenience samples (i.e., college student samples) (Fleischer et al., 2015).”
From Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective (2021) · cites this paper as Fleischer et al., 2015
Citations
Cites
Egeth, H. E., & Kahneman, D. J. (1975) · The American Journal of Psychology · doi:10.2307/1421603 · Cited by 6,976
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865
Wright, K. B. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00259.x · Cited by 2,550
Introduction to Measurement Theory
Allen, M. J. (1979) · Cited by 2,124
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315
Landers, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714
Factors Defined by Negatively Keyed Items: The Result of Careless Respondents?
Schmitt, N., & Stuits, D. M. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900405 · Cited by 536
Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis
Woods, C. M. (2006) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-005-9004-7 · Cited by 402
Evidence for response bias as a source of error variance in applied assessment.
McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010) · Psychological Bulletin · doi:10.1037/a0019216 · Cited by 282
MMPI-2 random responding indices: Validation using a self-report methodology.
Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L. H., Clark, C. D., & Monroe, K. (1992) · Psychological Assessment · doi:10.1037/1040-3590.4.3.340 · Cited by 177
Generation Digital: Politics, Commerce, and Childhood in the Age of the Internet
Montgomery, K. C. (2007) · Cited by 146
Employee attitude surveys: Examining the attitudes of noncompliant employees.
Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.2.284 · Cited by 139
Attitudes toward Surveys: Development of a Measure and Its Relationship to Respondent Behavior
Rogelberg, S. G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Horvath, M. J. (2001) · Organizational Research Methods · doi:10.1177/109442810141001 · Cited by 122
Response Bias and Reliability in Sensitive Topic Surveys
Marquis, K. H., Marquis, M. S., & Polich, J. M. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478282 · Cited by 96
An IRT Examination of the Psychometric Functioning of Negatively Worded Personality Items
Sliter, K. A., & Zickar, M. J. (2013) · Educational and Psychological Measurement · doi:10.1177/0013164413504584 · Cited by 87
A review of recent response-time analyses in educational testing.
Lee, Y.-H., & Chen, H. (2011) · Psychological test and assessment modeling · Cited by 86
Montgomery, K. C. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/3222.001.0001 · Cited by 86
Detection of Random Responding on the MMPI--A
Baer, R. A., Ballenger, J. F., Berry, D. T. R., & Wetter, M. W. (1997) · Journal of Personality Assessment · doi:10.1207/s15327752jpa6801_11 · Cited by 75
Research works printed but not linked: 2
- Huang, 2015, Does the negatively-worded item effect exist in personality measures? A meta-analysis
- Fleischer, 2015, Detecting inattentive pilot test examinees in an MTurk sample
Cited by
Sorted by citations
A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?
Walter, S. L., Seibert, S. E., Goering, D. D., & O’Boyle, E. H. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9552-y · Cited by 593
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Gamification and the impact of extrinsic motivation on needs satisfaction: Making work fun?
Mitchell, R., Schuster, L., & Jin, H. S. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.11.022 · Cited by 367
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168
Metadata
- Authors
- Fleischer, A., Mead, A. D., & Huang, J.
- Affiliations
- Illinois Institute of Technology (US)
- Corresponding author
- Fleischer, Avi
- ORCID
- Huang, Jialin
- Venue
- Industrial and Organizational Psychology
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-06
- Volume
- 8
- Issue
- 2
- Pages
- 196-202
- ISSN
- 1754-9426 (ISSN-L)
- 1754-9434
- OpenAlex
- W2199108421
- Cited by
- 151
- References
- 20
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 10 · 2023: 13 · 2022: 22 · 2021: 19 · 2020: 25 · 2019: 23 · 2018: 13 · 2017: 6 · 2016: 3
- FWCI
- 11.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Industrial and Organizational Psychology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Amazon Mechanical Turk, online sample, sample quality, convenience samples, low-stakes assessment, student samples, data cleaning, response validity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
