I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items
Open accessPublished version of a preprint
Abstract
Not all individuals put in the required thought and effort while responding to self-report survey items, and participant carelessness is a source of invalidity in psychological data (Huang, Liu, & Bowling, 2015). Many techniques and methods have been created to screen for this carelessness (Curran, 2016; Johnson, 2005), including the inclusion of items that researchers presume thoughtful individuals will answer in a given way (e.g., disagreement with “I am paid biweekly by leprechauns”, Meade & Craig, 2012). However, no studies have examined if these items always identify those who are careless, or if there exist individuals who have legitimate and justifiable reasons for picking out-of-bounds responses. This paper reports on two studies in which individuals spoke aloud a series of these questions, as well as their responses and justification for those responses. Coding of these responses found that a) individuals do occasionally report valid justifications for presumed invalid responses, b) there is relatively high variance in this behavior over different items, and c) items developed for this specific purpose tend to work better than those drawn from other sources or created ad hoc. These results suggest that care should be taken when implementing these types of items to screen for carelessness.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Three attention check questions were included to measure the quality of responses and minimize careless responses.
“three attention check questions were included to measure the quality of responses”
From Fear of missing out, social media influencers, and the social, psychological and financial wellbeing of young consumers (2025) · cites this paper as [95]
Cites
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · see citing works
The international personality item pool and the future of public-domain personality measures
Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006) · Journal of Research in Personality · doi:10.1016/j.jrp.2005.08.007 · Cited by 3,185 · see citing works
(1999) · Medical Entomology and Zoology · Cited by 3,151 · see citing works
MMPI-2 : Minnesota Multiphasic Personality Inventory-2 : manual for administration and scoring
Hathaway, S. R., & McKinley, J. (1989) · University of Minnesota Press eBooks · Cited by 2,450 · see citing works
Methods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356 · see citing works
Self-citationDetecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315 · see citing works
Self-citationAscertaining the validity of individual protocols from Web-based personality inventories
Johnson, J. A. (2004) · Journal of Research in Personality · doi:10.1016/j.jrp.2004.09.009 · Cited by 618 · see citing works
Insufficient effort responding: Examining an insidious confound in survey data.
Huang, J. L. W., Liu, M., & Bowling, N. A. (2014) · Journal of Applied Psychology · doi:10.1037/a0038510 · Cited by 421 · see citing works
A Meta‐Analytic Investigation of Job Applicant Faking on Personality Measures
Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. Τ., & Smith, M. A. (2006) · International Journal of Selection and Assessment · doi:10.1111/j.1468-2389.2006.00354.x · Cited by 400 · see citing works
Huang, J. L. W., Bowling, N. A., Liu, M., & Li, Y. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9357-6 · Cited by 315 · see citing works
Computerized Adaptive Assessment of Personality Disorder: Introducing the CAT–PD Project
Simms, L. J., Goldberg, L. R., Roberts, J. E., Watson, D., Welte, J. W., & Rotterman, J. H. (2011) · Journal of Personality Assessment · doi:10.1080/00223891.2011.577475 · Cited by 306 · see citing works
An Update on Survey Measures of Web-Oriented Digital Literacy
Hargittai, E. (2008) · Social Science Computer Review · doi:10.1177/0894439308318213 · Cited by 232 · see citing works
Detection of malingering on the MMPI: A meta-analysis
Berry, D. T. R., Baer, R. A., & Harris, M. J. (1991) · Clinical Psychology Review · doi:10.1016/0272-7358(91)90005-f · Cited by 210 · see citing works
A STUDY OF FAKING BEHAVIOR ON A FORCED‐CHOICE SELF‐DESCRIPTION CHECKLIST
Dunnette, M. D., McCartney, J. L., Carlson, H. C., & Kirchner, W. K. (1962) · Personnel Psychology · doi:10.1111/j.1744-6570.1962.tb01843.x · Cited by 111 · see citing works
THE FAKABILITY OF THE EDWARDS PERSONAL PREFERENCE SCHEDULE IN PERSONNEL SELECTION
Orpen, C. (1971) · Personnel Psychology · doi:10.1111/j.1744-6570.1971.tb00353.x · Cited by 22 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532 · see citing works
Ward, M. K., & Meade, A. W. (2023) · Annual Review of Psychology · doi:10.1146/annurev-psych-040422-045007 · Cited by 512 · see citing works
Zickar, M. J., & Keith, M. G. (2022) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-120920-052946 · Cited by 163 · see citing works
Arias, V. B., Garrido, L. E., Jenaro, C., Martínez‐Molina, A., & Martínez, B. A. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01401-8 · Cited by 144 · see citing works
Barends, A. J., & de Vries, R. E. (2019) · Personality and Individual Differences · doi:10.1016/j.paid.2019.02.015 · Cited by 124 · see citing works
Jaeger, S. R., & Cardello, A. V. (2022) · Food Quality and Preference · doi:10.1016/j.foodqual.2022.104676 · Cited by 113 · see citing works
Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting
Schroeders, U., Schmidt, C., & Gnambs, T. (2021) · Educational and Psychological Measurement · doi:10.1177/00131644211004708 · Cited by 93 · see citing works
Psychometric Properties and Correlates of Precarious Manhood Beliefs in 62 Nations
Bosson, J. K., Jurek, P., Vandello, J. A., Kosakowska-Berezecka, N., Olech, M., Besta, T., Bender, M., Hoorens, V., Becker, M., Timur Sevincer, A., et al. (2021) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022121997997 · Cited by 88 · see citing works
Ulitzsch, E., Pohl, S., Khorramdel, L., Kroehne, U., & von Davier, M. (2021) · Psychometrika · doi:10.1007/s11336-021-09817-7 · Cited by 78 · see citing works
Attention checks and how to use them: Review and practical recommendations
Muszyński, M. A. (2023) · Ask Research and Methods · doi:10.18061/ask.v32i1.0001 · Cited by 56 · see citing works
Weigold, A., & Weigold, I. K. (2021) · Social Science Computer Review · doi:10.1177/08944393211006847 · Cited by 49 · see citing works
Silber, H., Roßmann, J., & Gummer, T. (2022) · Field Methods · doi:10.1177/1525822x221115830 · Cited by 44 · see citing works
Using Market-Research Panels for Behavioral Science: An Overview and Tutorial
Moss, A. J., Hauser, D., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221140388 · Cited by 44 · see citing works
Jones, A., Earnest, J., Adam, M., Clarke, R., Yates, J., & Pennington, C. R. (2022) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000546 · Cited by 49 · see citing works
Pennington, C. R., Monk, R. L., Heim, D., Rose, A. K., Gough, T., Clarke, R., Knibb, G., Patel, R., Rai, P., Ravat, H., et al. (2023) · Psychology of Addictive Behaviors · doi:10.1037/adb0000919 · Cited by 22 · see citing works
Measuring Attentiveness in Self-Administered Surveys
Berinsky, A. J., Frydman, A., Margolis, M. F., Sances, M., & Valerio, D. C. (2024) · Public Opinion Quarterly · doi:10.1093/poq/nfae004 · Cited by 22 · see citing works
Jeremy Freese, A., & Jin, O. S. (2025) · Annual Review of Sociology · doi:10.1146/annurev-soc-090524-043117 · Cited by 21 · see citing works
Do you get what you pay for? Preventing insufficient effort responding in MTurk and student samples
Toich, M. J., Schutt, E., & Fisher, D. M. (2021) · Applied Psychology · doi:10.1111/apps.12344 · Cited by 18 · see citing works
Roberts, J. A., & David, M. E. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2238341 · Cited by 18 · see citing works
Arias, V. B., Ponce, F. P., Garrido, L. E., Nieto, M. D., Martínez‐Molina, A., & Martínez, B. A. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02308-w · Cited by 22 · see citing works
Mullarkey, M. C., Dobias, M., Sung, J. Y., Ahuvia, I. L., Shumake, J., Beevers, C. G., & Schleider, J. L. (2022) · JMIR Mental Health · doi:10.2196/33473 · Cited by 15 · see citing works
How a Few Inconsistent Respondents Can Confound the Structure of Personality Survey Data
Arias, V. B., Ponce, F. P., & Martínez‐Molina, A. (2022) · European Journal of Psychological Assessment · doi:10.1027/1015-5759/a000719 · Cited by 17 · see citing works
Zhang, J., Li, Y., Zeng, Y., & Lu, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03800-0 · Cited by 16 · see citing works
Bartosiak, A., Lee, J. E., & Loibl, C. (2025) · PLOS ONE · doi:10.1371/journal.pone.0319034 · Cited by 16 · see citing works
“Following common practice in reviewing if participants are fully reading and understanding the questions and to minimize careless responses [95–97], three attention check questions were included to measure the quality of responses (e.g., “Based on the responses below, regardless of what your…”
Kay, C. S., & Saucier, G. (2023) · Personality and Individual Differences · doi:10.1016/j.paid.2022.112073 · Cited by 14 · see citing works
Between friends: Forgiveness, unforgiveness, and wrongdoing in same-sex friendships
Boon, S. D., Hojjat, M., Paulin, M. M., & Stackhouse, M. R. D. (2021) · Journal of Social and Personal Relationships · doi:10.1177/02654075211062272 · Cited by 13 · see citing works
Cognitive abilities and visual complexity impact first impressions in five-second testing
Kuric, E., Demcak, P., Krajcovic, M., & Nguyen, G. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2023.2272747 · Cited by 12 · see citing works
The threat of AI chatbot responses to crowdsourced open-ended survey questions
Traylor, F. (2024) · Energy Research & Social Science · doi:10.1016/j.erss.2024.103857 · Cited by 14 · see citing works
Kay, C. S., & Dimakis, S. (2022) · Journal of Homosexuality · doi:10.1080/00918369.2022.2132576 · Cited by 10 · see citing works
Survey Attention and Self-Reported Political Behavior
Alvarez, R. M., & Li, Y. (2022) · Public Opinion Quarterly · doi:10.1093/poq/nfac048 · Cited by 11 · see citing works
Careless responding in online studies is associated with alcohol use: A mega-analysis.
Jones, A., Gillespie, S. M., Pennington, C. R., Strickland, J. C., & Robinson, E. (2023) · Psychology of Addictive Behaviors · doi:10.1037/adb0000924 · Cited by 11 · see citing works
Curran, P. G., & Denison, A. J. (2019) · doi:10.31234/osf.io/ge6fa · Cited by 11 · see citing works
Self-citationCareless responding detection revisited: Accuracy of direct and indirect measures
Goldammer, P., Stöckli, P. L., Escher, Y. A., Annen, H., Jonas, K. W., & Antonakis, J. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02484-3 · Cited by 10 · see citing works
Screening respondents to increase data quality in consumer tests
Castura, J. C., Pohjanheimo, T. A., Laaksonen, O. A., McEwan, J. A., Varela, P., & Næs, T. (2023) · Food Quality and Preference · doi:10.1016/j.foodqual.2023.105030 · Cited by 9 · see citing works
Ozaki, K. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02407-2 · Cited by 8 · see citing works
Stackhouse, M. R. D., Boon, S. D., & Paulin, M. M. (2023) · European Journal of Social Psychology · doi:10.1002/ejsp.2929 · Cited by 7 · see citing works
Almekhled, B., & Petrie, H. (2024) · Journal of Innovative Digital Transformation · doi:10.1108/jidt-10-2023-0040 · Cited by 7 · see citing works
Careless Response Processes are Heterogeneous: Comment on Goldammer et al. (2020)
Denison, A. J., & Wiernik, B. M. (2022) · Meta-Psychology · doi:10.15626/mp.2020.2637 · Cited by 6 · see citing works
Vhaduri, S., Cho, J., & Meng, K. (2023) · Smart Health · doi:10.1016/j.smhl.2023.100398 · Cited by 6 · see citing works
This will not change us: Leader's use of continuity rhetoric to promote collective change
Syfers, L., Rast, D. E., & Gaffney, A. M. (2023) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2023.104550 · Cited by 5 · see citing works
Alfons, A., & Welz, M. (2024) · Social and Personality Psychology Compass · doi:10.1111/spc3.12941 · Cited by 5 · see citing works
Lan, R., Shen, Z., & Lian, R. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02617-1 · Cited by 4 · see citing works
DeAtley, T., Johnson, A. C., Stone, M. D., Audrain-McGovern, J. E., Mercincavage, M., & Strasser, A. A. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20156454 · Cited by 4 · see citing works
Investigating Respondent Attention to Experimental Text Lengths
Rettig, T., & Blom, A. G. (2023) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smad044 · Cited by 4 · see citing works
Kay, C. S. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02452-x · Cited by 4 · see citing works
The Ethics of Conducting Research with Human Participants in HCI
Petrie, H. (2025) · Springer eBooks · doi:10.1007/978-3-319-27648-9_86-1 · Cited by 4 · see citing works
Ertuna, L., Uyanık, G. K., & Gençaslan, D. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1815225 · Cited by 3 · see citing works
I Caught It, Now What Do I Do With It? Controlling For Insufficient Effort Responding
Ragland, E., Miller, B. K., Simmering, M. J., & Fuller, C. M. (2025) · Journal of Business and Psychology · doi:10.1007/s10869-025-10025-8 · Cited by 3 · see citing works
Pihlajamäki, M., Eisele, G. V., Peeters, L., Kirtley, O. J., Myin‐Germeys, I., & Lafit, G. (2026) · Assessment · doi:10.1177/10731911251410340 · Cited by 3 · see citing works
Almekhled, B., & Petrie, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42286-7_27 · Cited by 3 · see citing works
The Impact of Attentiveness Interventions on Survey Data
Fuller, C. M., Simmering, M. J., Waterwall, B., Ragland, E., Twitchell, D. P., & Wall, A. (2025) · Educational and Psychological Measurement · doi:10.1177/00131644241311851 · Cited by 2 · see citing works
A sequential Moken scaling approach to evaluate response quality in survey research
Wind, S. A., Lugu, B. K., & Wang, Y. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02147-9 · Cited by 2 · see citing works
Older People’s Attitudes to Personal Robots
Hou, J. (2023) · OALib · doi:10.4236/oalib.1110635 · Cited by 1 · see citing works
Almekhled, B., & Petrie, H. (2023) · Electronic workshops in computing · doi:10.14236/ewic/bcshci2023.24 · Cited by 2 · see citing works
Beliefs about seeking and receiving help: a mixed-methods analysis
Scotney, V. S., & Tay, L. (2025) · The Journal of Positive Psychology · doi:10.1080/17439760.2025.2549765 · Cited by 1 · see citing works
Inattentive respondent detection methods for multiple-choice format items
Ozaki, K., Hirota, A., Suzuki, T., Nakashita, A., Park, Y., & Masuda, S. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.106689 · Cited by 1 · see citing works
Leadership as Development: Validating the Developmental Self-Concept Scale for Leaders
Pomerantz, A. L., Brown, R. P., & Johnson, S. K. (2026) · Journal of Leadership & Organizational Studies · doi:10.1177/15480518261442990 · Cited by 1 · see citing works
McLatchie, N. M., & Thomae, M. (2023) · Humor - International Journal of Humor Research · doi:10.1515/humor-2022-0134 · Cited by 1 · see citing works
The boundary-crossing constraint revisited: movement verbs across varieties of Spanish
Bocanegra, R. C. (2023) · Cognitive Linguistics · doi:10.1515/cog-2023-0030 · Cited by 1 · see citing works
Kuric, E., Demcak, P., Smrecek, P., & Špiláková, B. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20852-9 · Cited by 2 · see citing works
Detecting Careless Responding in Survey Data Using Stochastic Gradient BoostingPreprint
Schroeders, U., Schmidt, C. Μ., & Gnambs, T. (2020) · doi:10.31234/osf.io/f94zk · Cited by 1 · see citing works
Understanding, Detecting, and Adjusting for Response Bias in Surveys
Takahashi, Y., Arai, S., Masuda, S., Ozaki, K., & Todo, N. (2024) · The Annual Report of Educational Psychology in Japan · doi:10.5926/arepj.63.322
Blackhart, G. C., Purgason, R., Rusielewicz, D., Hernandez, D. K., & Hillhouse, L. (2025) · International Journal of Social Research Methodology · doi:10.1080/13645579.2025.2506078
The Effectiveness and Potential Risks of Using Careless Responding Indices to Improve Data QualityPreprint
Qiu, T., Ma, C., & Li, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5340451
The Prevention and Detection of Deception in Self-Report Survey Data
Clairmont, A., Wolf, M. G., & Maul, A. (2021) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-64802-604-120251018
Marsh, D. R., & Dik, B. J. (2026) · International Journal of Applied Positive Psychology · doi:10.1007/s41042-026-00284-y
On Knowing Who You Want to Become: Validating a Measure of Aspirational Self-Clarity
Brown, R. P., Hertel, A. W., Pomerantz, A. L., Lodi‐Smith, J., Westmoreland, K. E., & Parsons, S. V. (2026) · Journal of Personality Assessment · doi:10.1080/00223891.2026.2650793
Börkan, B., & Cuhadar, I. (2026) · Measurement Interdisciplinary Research and Perspectives · doi:10.1080/15366367.2026.2665781
Adaptation and validation of bogus items scale: A Chinese version
Li, J., Chen, S., Xing, Y.-X., Li, C.-J., Ma, C., & Li, Y. M. (2026) · Applied Psychology · doi:10.1111/apps.70100
Shockley, K. M., Clark, M. A., Dodd, H., King, E. B., & Buchanan, B. E. (2026) · Personnel Psychology · doi:10.1111/peps.70032
Rogers, J., & Hui, B. (2026) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2026.100340
The roles of sex and Greek life affiliation in food and alcohol disturbance
Edwards, A. B., & Kerr, K. L. (2026) · Journal of American College Health · doi:10.1080/07448481.2026.2705284
Mapping sleep disturbance and nap desire to dimensions of psychopathology in U.S. adults
Stanton, A. M., Ashar, D. A., & Seehuus, M. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1828219
Insufficient Effort Responding and Adolescent Respondents: Measurement, Extent, and Prediction
Krause, T., Vogl, S., & Sälzer, C. (2026) · Sociological Methods & Research · doi:10.1177/00491241261484518
Faraci, P., & Nasonte, G. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03163-1
Masculinity contest cultures and job satisfaction: better to be competent or embedded?
Regina, J., & Allen, T. D. (2026) · Journal of Business and Psychology · doi:10.1007/s10869-026-10159-3
Goldammer, P., Escher, Y. A., Stöckli, P. L., Annen, H., & Heimann, A. L. (2026) · Organizational Research Methods · doi:10.1177/10944281261488778
- Authors
- Curran, P. G., & Hauser, K. A.
- Affiliations
- Grand Valley State University (US)
- George Washington University (US)
- Corresponding author
- Curran, Paul G.
- ORCID
- Curran, Paul G.
- Venue
- Journal of Research in Personality
- Publisher
- Elsevier BV
- Published
- 2019-10
- Volume
- 82
- Pages
- 103849
- ISSN
- 0092-6566 (ISSN-L)
- OpenAlex
- W2965274117
- Cited by
- 79
- References
- 15
- Citations by year
- 2026: 16 · 2025: 9 · 2024: 10 · 2023: 22 · 2022: 11 · 2021: 7 · 2020: 2 · 2019: 2
- FWCI
- 11.05
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Journal of Research in Personality (publishedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Keywords
- response validity, item development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
