Inattentive responding in MTurk and other online samples
Abstract
The focal article by Landers and Behrend (2015) makes the case that samples collected on microtask websites like Amazon's Mechanical Turk (MTurk) are inherently no better or worse than traditional samples of convenience from university students or organizations. We wholeheartedly agree. However, having successfully used MTurk and other online sources for data collection, we feel that the focal article was insufficient regarding the caution required in identifying inattentive respondents and the problems that can arise if such individuals are not removed from the dataset. Although we focus on MTurk, similar issues arise for most “low-stakes” assessments, including student samples, which seem to be increasingly collected online.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MTurk data is as reliable as other common convenience samples (e.g., college student samples).
“Previous research also demonstrates that MTurk data is as reliable as other common convenience samples (i.e., college student samples) (Fleischer et al., 2015).”
From Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective (2021) · cites this paper as Fleischer et al., 2015
Cites
Egeth, H. E., & Kahneman, D. J. (1975) · The American Journal of Psychology · doi:10.2307/1421603 · Cited by 6,976 · see citing works
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · see citing works
Wright, K. B. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00259.x · Cited by 2,550 · see citing works
Introduction to Measurement Theory
Allen, M. J. (1979) · Cited by 1,971 · see citing works
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315 · see citing works
Landers, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714 · see citing works
Factors Defined by Negatively Keyed Items: The Result of Careless Respondents?
Schmitt, N., & Stuits, D. M. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900405 · Cited by 536 · see citing works
Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis
Woods, C. M. (2006) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-005-9004-7 · Cited by 402 · see citing works
Evidence for response bias as a source of error variance in applied assessment.
McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010) · Psychological Bulletin · doi:10.1037/a0019216 · Cited by 282 · see citing works
MMPI-2 random responding indices: Validation using a self-report methodology.
Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L. H., Clark, C. D., & Monroe, K. (1992) · Psychological Assessment · doi:10.1037/1040-3590.4.3.340 · Cited by 177 · see citing works
Generation Digital: Politics, Commerce, and Childhood in the Age of the Internet
Montgomery, K. C. (2007) · Cited by 146 · see citing works
Employee attitude surveys: Examining the attitudes of noncompliant employees.
Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.2.284 · Cited by 139 · see citing works
Attitudes toward Surveys: Development of a Measure and Its Relationship to Respondent Behavior
Rogelberg, S. G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Horvath, M. J. (2001) · Organizational Research Methods · doi:10.1177/109442810141001 · Cited by 122 · see citing works
Response Bias and Reliability in Sensitive Topic Surveys
Marquis, K. H., Marquis, M. S., & Polich, J. M. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478282 · Cited by 96 · see citing works
An IRT Examination of the Psychometric Functioning of Negatively Worded Personality Items
Sliter, K. A., & Zickar, M. J. (2013) · Educational and Psychological Measurement · doi:10.1177/0013164413504584 · Cited by 87 · see citing works
A review of recent response-time analyses in educational testing.
Lee, Y.-H., & Chen, H. (2011) · Psychological test and assessment modeling · Cited by 86 · see citing works
Montgomery, K. C. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/3222.001.0001 · Cited by 86 · see citing works
Detection of Random Responding on the MMPI--A
Baer, R. A., Ballenger, J. F., Berry, D. T. R., & Wetter, M. W. (1997) · Journal of Personality Assessment · doi:10.1207/s15327752jpa6801_11 · Cited by 75 · see citing works
Flagged first, then by citations
2 of the 20 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?
Walter, S. L., Seibert, S. E., Goering, D. D., & O’Boyle, E. H. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9552-y · Cited by 548 · see citing works
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474 · see citing works
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319 · see citing works
Gamification and the impact of extrinsic motivation on needs satisfaction: Making work fun?
Mitchell, R., Schuster, L., & Jin, H. S. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.11.022 · Cited by 317 · see citing works
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168 · see citing works
Zickar, M. J., & Keith, M. G. (2022) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-120920-052946 · Cited by 163 · see citing works
Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective
Ball, C., Huang, K.-T., & Francis, J. (2021) · Telematics and Informatics · doi:10.1016/j.tele.2021.101728 · Cited by 152 · see citing works
“Previous research also demonstrates that MTurk data is as reliable as other common convenience samples (i.e., college student samples) (Fleischer et al., 2015).”
The MTurkification of Social and Personality Psychology
Anderson, C. A., Allen, J. J., Plante, C. N., Quigley‐McBride, A., Lovett, A., & Rokkum, J. N. (2018) · Personality and Social Psychology Bulletin · doi:10.1177/0146167218798821 · Cited by 127 · see citing works
Wheaton, M. G., Messner, G. R., & Marks, J. B. (2021) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2020.100605 · Cited by 111 · see citing works
Amazon's Mechanical Turk: A Comment
Ford, J. B. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1277380 · Cited by 112 · see citing works
Patihis, L., & Pendergrast, M. H. (2018) · Clinical Psychological Science · doi:10.1177/2167702618773315 · Cited by 95 · see citing works
Germelmann, C. C., Herrmann, J.-L., Kacha, M., & Darke, P. R. (2020) · Journal of Advertising · doi:10.1080/00913367.2020.1745110 · Cited by 92 · see citing works
Berry, C., Kees, J., & Burton, S. (2022) · Journal of Advertising · doi:10.1080/00913367.2022.2079026 · Cited by 90 · see citing works
Using Mechanical Turk Data in IS Research: Risks, Rewards, and Recommendations
Jia, R., Steelman, Z., & Reich, B. H. (2017) · Communications of the Association for Information Systems · doi:10.17705/1cais.04114 · Cited by 89 · see citing works
To grit or not to grit, that is the question!
Vazsonyi, A. T., Ksinan, A. J., Jiskrova, G. K., Mikuška, J., Javakhishvili, M., & Cui, G. (2018) · Journal of Research in Personality · doi:10.1016/j.jrp.2018.12.006 · Cited by 87 · see citing works
Davids, C. M., Watson, L. B., & Gere, M. P. (2018) · Sex Roles · doi:10.1007/s11199-018-0940-6 · Cited by 85 · see citing works
Dimitriu, R., & Guesalaga, R. (2017) · Psychology and Marketing · doi:10.1002/mar.21007 · Cited by 79 · see citing works
When Patients Overreport Symptoms: More Than Just Malingering
Merckelbach, H. L. G. J., Dandachi‐FitzGerald, B., van Helvoort, D. H. J., Jelícic, M., & Otgaar, H. (2019) · Current Directions in Psychological Science · doi:10.1177/0963721419837681 · Cited by 79 · see citing works
Millroth, P., & Frey, R. (2021) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2021.102454 · Cited by 80 · see citing works
A panel for lemons? Positivity bias, reputation systems and data quality on MTurk
Matherly, T. (2019) · European Journal of Marketing · doi:10.1108/ejm-07-2017-0491 · Cited by 75 · see citing works
Lie prevalence, lie characteristics and strategies of self-reported good liars
Verigin, B. L., Meijer, E. H., Bogaard, G., & Vrij, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225566 · Cited by 73 · see citing works
Meuwese, D., Dijkstra, K., Maas, J., & Koole, S. L. (2021) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2021.101594 · Cited by 61 · see citing works
Mell, J. N., DeChurch, L. A., Leenders, R., & Contractor, N. (2019) · Academy of Management Journal · doi:10.5465/amj.2018.0325 · Cited by 60 · see citing works
Swiping away the moments that make up a dull day: Narcissism, boredom, and compulsive smartphone use
Ksinan, A. J., Mališ, J., & Vazsonyi, A. T. (2019) · Current Psychology · doi:10.1007/s12144-019-00228-7 · Cited by 51 · see citing works
Bae, M. (2022) · Internet Research · doi:10.1108/intr-05-2021-0269 · Cited by 49 · see citing works
Hardiness moderates the effects of COVID-19 stress on anxiety and depression
Bartone, P. T., McDonald, K., Hansma, B. J., & Solomon, J. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.08.045 · Cited by 48 · see citing works
A Primer for Conducting Survey Research using MTurk
Chambers, S., Nimon, K., & Anthony-McMann, P. E. (2016) · International Journal of Adult Vocational Education and Technology · doi:10.4018/ijavet.2016040105 · Cited by 41 · see citing works
Smith, B. R. (2019) · Journal of the Midwest Association for Information Systems · doi:10.17705/3jmwa.000050 · Cited by 40 · see citing works
Person–Environment Fit and Work Satisfaction: Exploring the Conditional Effects of Age
Rauvola, R. S., Rudolph, C. W., Ebbert, L. K., & Zacher, H. (2019) · Work Aging and Retirement · doi:10.1093/workar/waz011 · Cited by 39 · see citing works
Towards Understanding and Detecting Cyberbullying in Real-world Images
Vishwamitra, N., Hu, H., Luo, F., & Cheng, L. (2021) · Proceedings 2021 Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24260 · Cited by 39 · see citing works
Using pre-test explanations to improve test-taker reactions: Testing a set of “wise” interventions
McCarthy, J. M., Bauer, T. N., Truxillo, D. M., Campion, M. C., Iddekinge, C. H. V., & Campion, M. A. (2017) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2017.04.002 · Cited by 36 · see citing works
Liu, I., Ni, S., & Peng, K. (2020) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2019.0283 · Cited by 36 · see citing works
Moving Toward a New Model of Sexual Consent: The Development of the Process-Based Consent Scale
Glace, A. M., Zatkin, J. G., & Kaufman, K. L. (2020) · Violence Against Women · doi:10.1177/1077801220952159 · Cited by 35 · see citing works
Development of a Measure of Informal Workplace Social Interactions
Winslow, C. J., Sabat, I. E., Anderson, A. J., Kaplan, S. A., & Miller, S. J. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02043 · Cited by 34 · see citing works
Wörtler, B., Yperen, N. W. V., & Barelds, D. P. H. (2020) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2020.1844663 · Cited by 32 · see citing works
Tailoring service recovery messages to consumers’ affective states
Cummings, K. H., & Yule, J. A. (2020) · European Journal of Marketing · doi:10.1108/ejm-02-2019-0122 · Cited by 30 · see citing works
Investigating the Accessibility of Crowdwork Tasks on Mechanical Turk
Uzor, S., Jacques, J. T., Dudley, J. J., & Kristensson, P. O. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445291 · Cited by 28 · see citing works
Meier, E., Aziz, S., Wuensch, K. L., & Dolbier, C. L. (2020) · Journal of Leisure Research · doi:10.1080/00222216.2020.1778589 · Cited by 26 · see citing works
Insufficient Effort Responding as a Potential Confound between Survey Measures and Objective Tests
Huang, J. L., & DeSimone, J. (2020) · Journal of business and psychology · doi:10.1007/s10869-020-09707-2 · Cited by 20 · see citing works
Геннадьевич, М. К., Shcherbakova, O., & Kaufman, J. C. (2021) · Creativity Research Journal · doi:10.1080/10400419.2021.1953823 · Cited by 26 · see citing works
Protecting their turf: When and why supervisors undermine employee boundary spanning.
Mell, J. N., Quintane, E., Hirst, G., & Carnegie, A. B. (2021) · Journal of Applied Psychology · doi:10.1037/apl0000960 · Cited by 26 · see citing works
Alarcon, G. M., Gamble, R., Jessup, S. A., Walter, C., Ryan, T. J., Wood, D. W., & Calhoun, C. (2017) · Cogent Psychology · doi:10.1080/23311908.2017.1389640 · Cited by 24 · see citing works
Alt-White? A Gendered Look at “Victim” Ideology and the Alt-Right
Boehme, H. M., & Isom, D. A. (2019) · Victims & Offenders · doi:10.1080/15564886.2019.1679308 · Cited by 24 · see citing works
Do truth-telling oaths improve honesty in crowd-working?
Jacquemet, N., James, A. G., Luchini, S., Murphy, J. J., & Shogren, J. F. (2021) · PLoS ONE · doi:10.1371/journal.pone.0244958 · Cited by 24 · see citing works
Using surveytainment to counter declining survey data quality
Kostyk, A., Zhou, W., & Hyman, M. R. (2018) · Journal of Business Research · doi:10.1016/j.jbusres.2018.10.024 · Cited by 23 · see citing works
Development and Validation of the Ableist Microaggression Impact Questionnaire
Aydemir-Döke, D., & Herbert, J. T. (2021) · Rehabilitation Counseling Bulletin · doi:10.1177/00343552211014259 · Cited by 21 · see citing works
Can we trust online crowdworkers?
Jonell, P., Kucherenko, T., Torre, I., & Beskow, J. (2020) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3383652.3423860 · Cited by 20 · see citing works
Factor Structure and Psychometric Properties of the Digital Communication Empathy Scale (DCES)
Collins, A. M., Warburton, W., Bussey, K., & Sweller, N. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103183 · Cited by 19 · see citing works
Talking Past Each Other: Construal Level, Utilitarian Motives, and Entrepreneurial Team Formation
Gray, S. M., Howell, T., & Sackett, E. (2024) · Organization Science · doi:10.1287/orsc.2022.16693 · Cited by 19 · see citing works
Wheaton, M. G., & Kaiser, N. (2021) · International Journal of Cognitive Therapy · doi:10.1007/s41811-021-00114-w · Cited by 18 · see citing works
Workplace Anger Costs Women Irrespective of Race
Marshburn, C. K., Cochran, K. J., Flynn, E., & Levine, L. J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.579884 · Cited by 18 · see citing works
A Psychometric Evaluation of the Intrasexual Competition Scale
Albert, G. O., Richardson, G. B., Arnocky, S. A., Bird, B. M., Fisher, M. L., Hlay, J. K., McHale, T. S., & Hodges‐Simeon, C. R. (2022) · Archives of Sexual Behavior · doi:10.1007/s10508-021-02167-6 · Cited by 16 · see citing works
Preventing satisficing: A narrative review
Blazek, D. R., & Siegel, J. T. (2023) · International Journal of Social Research Methodology · doi:10.1080/13645579.2023.2239086 · Cited by 16 · see citing works
Enhancing Self-Administered Questionnaire Response Quality Using Code of Conduct Reminders
Saunders, C., & Kulchitsky, J. D. (2021) · International Journal of Market Research · doi:10.1177/14707853211055060 · Cited by 17 · see citing works
Schell, C., Godinho, A., & Cunningham, J. A. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01556-2 · Cited by 15 · see citing works
Huang, K.-T., Ball, C., & Francis, J. (2023) · American Behavioral Scientist · doi:10.1177/00027642231156775 · Cited by 19 · see citing works
Dual effects of self-focused and other-focused power on sexual harassment intentions
Stockdale, M. S., Gilmer, D. O., & Dinh, T. (2019) · Equality Diversity and Inclusion An International Journal · doi:10.1108/edi-09-2018-0160 · Cited by 15 · see citing works
On Text-based Personality Computing: Challenges and Future Directions
Fang, Q., Giachanou, A., Bagheri, A., Boeschoten, L., van Kesteren, E., Kamalabad, M. S., & Oberski, D. L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.691 · Cited by 14 · see citing works
Kalesnikaite, V., & Baker, K. P. (2024) · Public Management Review · doi:10.1080/14719037.2024.2303030 · Cited by 14 · see citing works
The Self-Report Symptom Inventory as an Instrument for Detecting Symptom Over-Reporting
Bošković, I., Merckelbach, H. L. G. J., Merten, T., Hope, L., & Jelícic, M. (2019) · European Journal of Psychological Assessment · doi:10.1027/1015-5759/a000547 · Cited by 14 · see citing works
Playing alone: can game design elements satisfy user needs in gamified mHealth services?
Mitchell, R., Schuster, L., & Jin, H. S. (2021) · Health Promotion International · doi:10.1093/heapro/daab168 · Cited by 14 · see citing works
PACT : Developing and Evaluating a Digital Sexual Consent Program for Youth
Javidi, H., Widman, L. M., Maheux, A. J., McCrimmon, J., Evans-Paulson, R., & Becker, W. (2023) · The Journal of Sex Research · doi:10.1080/00224499.2023.2208560 · Cited by 13 · see citing works
Relations among psychopathy, moral competence, and moral intuitions in student and community samples
Gay, J. G., Vitacco, M. J., Hackney, A. A., Beussink, C. N., & Lilienfeld, S. O. (2018) · Legal and Criminological Psychology · doi:10.1111/lcrp.12128 · Cited by 13 · see citing works
The structured interview's resistance to gender discrimination under cognitive load
Kith, S., Cassidy, S. A., & Powell, D. M. (2022) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12361 · Cited by 13 · see citing works
When altruistic decisions shape egoistic motives: motivation shift in sequential charitable support
Fanggidae, J. P., Jin, H. S., Kim, H. J., & Kelly, L. (2022) · International Journal of Advertising · doi:10.1080/02650487.2022.2140962 · Cited by 13 · see citing works
Shoppers’ susceptibility to information overload: scale development and validation
Hunter, G. L., Taylor, S. A., & Kallen, P. H. (2022) · The Journal of Marketing Theory and Practice · doi:10.1080/10696679.2022.2121287 · Cited by 12 · see citing works
Milczarski, W., Borkowska, A., Paruzel‐Czachura, M., & Białek, M. (2023) · International Journal of Bilingualism · doi:10.1177/13670069231206701 · Cited by 13 · see citing works
Nivedhitha, K. S. (2022) · Behaviour and Information Technology · doi:10.1080/0144929x.2022.2093792 · Cited by 14 · see citing works
Risk attitudes: The central tendency bias
Akbari, K., Eigruber, M., & Vetschera, R. (2023) · EURO Journal on Decision Processes · doi:10.1016/j.ejdp.2023.100042 · Cited by 11 · see citing works
Credentials or Chemistry? Entrepreneur Gender and Cofounder Selection
Gray, S., Howell, T., Strassman, J., & Yamamoto, K. (2024) · Academy of Management Journal · doi:10.5465/amj.2022.0640 · Cited by 11 · see citing works
Destigmatization and Its Imbalanced Effects in Labor Markets
Negro, G., Williams, M. J., Pontikes, E., & Lopiano, G. R. (2021) · Management Science · doi:10.1287/mnsc.2020.3881 · Cited by 11 · see citing works
Niles, J. K., Gutiérrez, D., Dukes, A. T., Mullen, P. R., & Goode, C. D. (2022) · Journal of Addictions & Offender Counseling · doi:10.1002/jaoc.12099 · Cited by 12 · see citing works
Conducting Survey Research Using MTurk
Chambers, S., & Nimon, K. (2018) · Advances in library and information science (ALIS) book series · doi:10.4018/978-1-5225-5164-5.ch016 · Cited by 11 · see citing works
Romance 2.0 on Instagram! “What Type of Girlfriend Would You Date?”
Jin, S. A., Ryu, E., & Muqaddam, A. (2019) · Evolutionary Psychology · doi:10.1177/1474704919826845 · Cited by 11 · see citing works
Is value statement an effective informal control for stimulating pro-environmental behaviors?
de Aguiar, A. B. (2020) · Revista Contabilidade & Finanças · doi:10.1590/1808-057x202011500 · Cited by 11 · see citing works
Individual differences in effective interview design: Factors affecting question choice
Speer, A. B., Tenbrink, A. P., Schwendeman, M. G., & Wegmeyer, L. J. (2020) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12297 · Cited by 11 · see citing works
Jones, A. T., Hadsell, L., & Burrus, R. T. (2019) · Eastern Economic Journal · doi:10.1057/s41302-019-00138-3 · Cited by 10 · see citing works
Grosser, T. J., Sterling, C., & Piplani, R. S. (2023) · Journal of Organizational Behavior · doi:10.1002/job.2734 · Cited by 10 · see citing works
Generalizability of Effect Sizes Within Aviation Research: More Samples Are Needed
Mehta, R., Rice, S. C., Winter, S. R., Spence, T. B., Edwards, M., & Candelaria-Oquendo, K. (2019) · International Journal of Aviation Aeronautics and Aerospace · doi:10.15394/ijaaa.2019.1404 · Cited by 9 · see citing works
Higher-Order Musical Temporal Structure in Bird Song
Bilger, H., Vertosick, E., Vickers, A. J., Kaczmarek, K., & Prum, R. O. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.629456 · Cited by 9 · see citing works
Howard, M. C. (2019) · Performance Improvement Quarterly · doi:10.1002/piq.21301 · Cited by 9 · see citing works
Peng, S., Man, K., Veldkamp, B. P., Cai, Y., & Tu, D. (2023) · Organizational Research Methods · doi:10.1177/10944281231181642 · Cited by 9 · see citing works
The company you keep: Satisfaction with life, economic freedom, and preference-policy mismatch
Hadsell, L., & Jones, A. T. (2020) · Journal of Comparative Economics · doi:10.1016/j.jce.2019.12.002 · Cited by 8 · see citing works
Davis, R. E., & Renzetti, C. M. (2021) · Journal of Interpersonal Violence · doi:10.1177/0886260520985497 · Cited by 8 · see citing works
Niles, A. N., & O’Donovan, A. (2018) · Cognitive Therapy and Research · doi:10.1007/s10608-018-9923-8 · Cited by 8 · see citing works
Verifiability and Symptom Endorsement in Genuine, Exaggerated, and Malingered Pain
Akca, A. Y. E., Slootmaekers, L., & Bošković, I. (2020) · Psychological Injury and Law · doi:10.1007/s12207-020-09375-w · Cited by 8 · see citing works
LaCosse, J., & Plant, E. A. (2018) · European Journal of Social Psychology · doi:10.1002/ejsp.2391 · Cited by 7 · see citing works
Eyewitness identification performance is not affected by time-of-day optimality
Yaremenko, S., Sauerland, M., & Hope, L. (2021) · Scientific Reports · doi:10.1038/s41598-021-82628-z · Cited by 7 · see citing works
Marketing Influences on Perceptions of Reduced Nicotine Content Cigarettes
Johnson, A. C., Mays, D. M., Villanti, A. C., NIAURA, R. S., Rehberg, K., Phan, L., Mercincavage, M., Luta, G., & Strasser, A. A. (2019) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntz167 · Cited by 7 · see citing works
Wijngaards, I., Burger, M., & van Exel, J. N. A. (2020) · Journal of Well-Being Assessment · doi:10.1007/s41543-021-00040-w · Cited by 7 · see citing works
Liu, W., Zhou, K., Ouyang, X., Chen, S., & Gao, K. (2025) · Internet Research · doi:10.1108/intr-03-2024-0422 · Cited by 6 · see citing works
Toloka platform as a source of online survey participants: an experience of assessing data quality
Gavrilov, K. (2022) · Sociology methodology methods mathematical modeling (Sociology 4M) · doi:10.19181/4m.2021.53.5 · Cited by 6 · see citing works
Good for the Many or Best for the Few?
Dove, G., Balestra, M., Mann, D., & Nov, O. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415239 · Cited by 6 · see citing works
Clemens, V., Sabel, C. A., Foege, J. N., & Nüesch, S. (2022) · Schmalenbach Journal of Business Research · doi:10.1007/s41471-022-00133-z · Cited by 6 · see citing works
The verb–self link: An implicit association test study
Weis, P. P., Nikadon, J., Herbert, C., & Formanowicz, M. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02105-0 · Cited by 6 · see citing works
Webster, J. M., Golding, J. M., Malik, S. E., & Riederer, A. M. (2020) · Psychology Crime and Law · doi:10.1080/1068316x.2020.1742342 · Cited by 6 · see citing works
Grammatical Redundancy in Scales: Using the “ConGRe” Process to Create Better Measures
Alley, L., Kadolkar, I., Gupta, A., Cortina, J. M., & Winsler, K. (2024) · Journal of Management · doi:10.1177/01492063241291542 · Cited by 5 · see citing works
Kim, N., & Oh, J. (2022) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.2022.2073895 · Cited by 5 · see citing works
Novel Approaches for Improving data Quality from Self-Administered Questionnaires
Hyman, M. R., Kostyk, A., Zhou, W., & Paas, L. J. (2019) · International Journal of Market Research · doi:10.1177/1470785319870622a · Cited by 5 · see citing works
Predicting Personality with Smartphone Cameras: A Pilot Study
Liu, I., Ni, S., & Peng, K. (2020) · IEEE International Conference on Human-Machine Systems (ICHMS) · doi:10.1109/ichms49158.2020.9209354 · Cited by 5 · see citing works
- Authors
- Fleischer, A., Mead, A. D., & Huang, J.
- Affiliations
- Illinois Institute of Technology (US)
- Corresponding author
- Fleischer, Avi
- ORCID
- Huang, Jialin
- Venue
- Industrial and Organizational Psychology
- Publisher
- Cambridge University Press (CUP)
- Published
- 2015-06
- Volume
- 8
- Issue
- 2
- Pages
- 196-202
- ISSN
- 1754-9426 (ISSN-L)
- 1754-9434
- OpenAlex
- W2199108421
- Cited by
- 151
- References
- 20
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 10 · 2023: 13 · 2022: 22 · 2021: 19 · 2020: 25 · 2019: 23 · 2018: 13 · 2017: 6 · 2016: 3
- FWCI
- 11.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Industrial and Organizational Psychology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Amazon Mechanical Turk, online sample, sample quality, convenience samples, low-stakes assessment, student samples, data cleaning, response validity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
