Attention by design: Using attention checks to detect inattentive respondents and improve data quality
Abstract
Abstract This paper examines attention checks and manipulation validations to detect inattentive respondents in primary empirical data collection. These prima facie attention checks range from the simple such as reverse scaling first proposed a century ago to more recent and involved methods such as evaluating response patterns and timed responses via online data capture tools. The attention check validations also range from easily implemented mechanisms such as automatic detection through directed queries to highly intensive investigation of responses by the researcher. The latter has the potential to introduce inadvertent researcher bias as the researcher's judgment may impact the interpretation of the data. The empirical findings of the present work reveal that construct and scale validations show consistently significant improvement in the fit statistics—a finding of great use for researchers working predominantly with scales and constructs for their empirical models. However, based on the rudimentary experimental models employed in the analysis, attention checks generally do not show a consistent, systematic improvement in the significance of test statistics for experimental manipulations. This latter result indicates that, by their very nature, attention checks may trigger an inherent trade‐off between loss of sample subjects—lowered power and increased Type II error—and the potential of capitalizing on chance alone—the possibility that the previously significant results were in fact the result of Type I error. The analysis also shows that the attrition rates due to attention checks—upwards of 70% in some observed samples—are far larger than typically assumed. Such loss rates raise the specter that studies not validating attention may inadvertently increase their Type I error rate. The manuscript provides general guidelines for various attention checks, discusses the psychological nuances of the methods, and highlights the delicate balance among incentive alignment, monetary compensation, and the subsequently triggered mood of respondents.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Validation methods included contrasting methods used to identify respondents who provided poor quality data (e.g., directed responses, inconsistent reporting, speeding, straight lining, or zigzagging).
“Validation methods included contrasting methods used to identify respondents who provided poor quality data (e.g., directed responses, inconsistent reporting, speeding, straight lining, or zigzagging; [21]).”
From Dimensions of caregiver strain may partially mediate the relationship between youth symptomology and counseling utilization among Latinxs (2024) · cites this paper as [21]
Cites
Ryan, R. M., & Deci, E. L. (2000) · American Psychologist · doi:10.1037//0003-066x.55.1.68 · Cited by 27,815 · see citing works
Multivariate data analysis : a global perspective
Hair, J. F. (2010) · Cited by 12,258 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
(1932) · Medical Entomology and Zoology · Cited by 9,568 · see citing works
Chaiken, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.39.5.752 · Cited by 5,516 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Fundamentals of item response theory
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4185 · Cited by 3,417 · see citing works
Lepper, M. R., Greene, D., & Nisbett, R. E. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0035519 · Cited by 2,471 · see citing works
Pay Enough or Don't Pay at All*
Gneezy, U., & Rustichini, A. (2000) · Quarterly Journal of Economics · doi:10.1162/003355300554917 · Cited by 2,175 · see citing works
Robust Statistics: The Approach Based on Influence Functions.
Jennison, C., Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., & Stahel, W. A. (1987) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981480 · Cited by 1,753 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Empirical research methods in operations management
Flynn, B. B., Sakakibara, S., Schroeder, R. G., Bates, K. A., & Flynn, E. J. (1990) · Journal of Operations Management · doi:10.1016/0272-6963(90)90098-x · Cited by 1,636 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
An assessment of survey research in POM: from constructs to theory
Malhotra, M. K., & Grover, V. (1998) · Journal of Operations Management · doi:10.1016/s0272-6963(98)00021-7 · Cited by 1,316 · see citing works
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315 · see citing works
Peterson, R. A. (2001) · Journal of Consumer Research · doi:10.1086/323732 · Cited by 1,258 · see citing works
Detrimental effects of reward: Reality or myth?
Eisenberger, R., & Cameron, J. (1996) · American Psychologist · doi:10.1037//0003-066x.51.11.1153 · Cited by 1,016 · see citing works
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Detrimental effects of reward. Reality or myth?
Eisenberger, R., & Cameron, J. (1996) · American Psychologist · doi:10.1037/0003-066x.51.11.1153 · Cited by 949 · see citing works
The Impact of Accountability on Judgment and Choice: Toward A Social Contingency Model
Tetlock, P. E. (1992) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60287-7 · Cited by 926 · see citing works
A Contingency Model for the Selection of Decision Strategies
Beach, L. R., & Mitchell, T. R. (1978) · Academy of Management Review · doi:10.5465/amr.1978.4305717 · Cited by 922 · see citing works
The Disgust Scale: item analysis, factor structure, and suggestions for refinement.
Olatunji, B., Williams, N. L., Tolin, D., Abramowitz, J., Sawchuk, C. N., Lohr, J., & Elwood, L. S. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.3.281 · Cited by 833 · see citing works
Performance incentives and means: How regulatory focus influences goal attainment.
Shah, J. Y., Higgins, T. E., & Friedman, R. (1998) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.74.2.285 · Cited by 822 · see citing works
Construct Validity: Advances in Theory and Methodology
Strauss, M. E., & Smith, G. T. (2008) · Annual Review of Clinical Psychology · doi:10.1146/annurev.clinpsy.032408.153639 · Cited by 806 · see citing works
McGrath, J. E. (1981) · American Behavioral Scientist · doi:10.1177/000276428102500205 · Cited by 738 · see citing works
Doing Survey Research: A Guide to Quantitative Methods
Nardi, P. M. (2005) · Cited by 733 · see citing works
Performance incentives and means: How regulatory focus influences goal attainment.
Shah, J. Y., Higgins, T. E., & Friedman, R. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.2.285 · Cited by 616 · see citing works
Behavior in operations management: Assessing recent findings and revisiting old assumptions
Bendoly, E., Donohue, K. L., & Schultz, K. L. (2006) · Journal of Operations Management · doi:10.1016/j.jom.2005.10.001 · Cited by 545 · see citing works
Methods in behavioral research
Rippere, V. (1979) · Behaviour Research and Therapy · doi:10.1016/0005-7967(79)90091-3 · Cited by 541 · see citing works
Tetlock, P. E., Skitka, L. J., & Boettger, R. D. (1989) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.57.4.632 · Cited by 541 · see citing works
Barnette, J. J. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970592 · Cited by 529 · see citing works
Accountability: A social magnifier of the dilution effect.
Tetlock, P. E., & Boettger, R. D. (1989) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.57.3.388 · Cited by 400 · see citing works
Methods in behavioral research
Cozby, P. C. (1977) · Cited by 374 · see citing works
Measuring mindfulness? An item response theory analysis of the Mindful Attention Awareness Scale
Van Dam, N. T., Earleywine, M., & Borders, A. (2010) · Personality and Individual Differences · doi:10.1016/j.paid.2010.07.020 · Cited by 338 · see citing works
Evidence of a detrimental effect of extrinsic incentives on breaking a mental set
McGraw, K. O., & McCullers, J. C. (1979) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(79)90039-8 · Cited by 338 · see citing works
Incentive-Aligned Conjoint Analysis
Ding, M., Grewal, R., & Liechty, J. (2005) · Journal of Marketing Research · doi:10.1509/jmkr.42.1.67.56890 · Cited by 322 · see citing works
McAllister, D. W., Mitchell, T. R., & Beach, L. R. (1979) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(79)90027-8 · Cited by 283 · see citing works
Two methods of reducing overconfidence
Arkes, H. R., Christensen, C., Lai, C., & Blumer, C. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90049-5 · Cited by 277 · see citing works
The effect of accountability on susceptibility to decision errors
Simonson, I., & Nye, P. (1992) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(92)90020-8 · Cited by 269 · see citing works
Effects of justification and a mechanical aid on judgment performance
Ashton, R. H. (1992) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(92)90040-e · Cited by 245 · see citing works
Behavioral operations: The state of the field
Croson, R. T. A., Schultz, K. L., Siemsen, E., & Yeo, M. L. (2012) · Journal of Operations Management · doi:10.1016/j.jom.2012.12.001 · Cited by 236 · see citing works
The Role of Perceived Quality Risk in Pricing Remanufactured Products
Abbey, J. D., Kleber, R., Souza, G. C., & Voigt, G. (2016) · Production and Operations Management · doi:10.1111/poms.12628 · Cited by 208 · see citing works
Self-citationReplication in behavioral research.
Rosenthal, R. (1990) · Journal of social behavior and personality · Cited by 206 · see citing works
Auction fever: The effect of opponents and quasi-endowment on product valuations
Heyman, J. E., Orhun, A. Y., & Ariely, D. (2004) · Journal of Interactive Marketing · doi:10.1002/dir.20020 · Cited by 188 · see citing works
Wording Effects in Self-Esteem Scales: Methodological Artifact or Response Style?
Horan, P. M., DiStefano, C., & Motl, R. W. (2003) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1003_6 · Cited by 185 · see citing works
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works
Does having to justify one's judgments change the nature of the judgment process?
Hagafors, R., & Brehmer, B. (1983) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(83)90122-8 · Cited by 182 · see citing works
Effects of rater accountability on the accuracy and the favorability of performance ratings.
Mero, N. P., & Motowidlo, S. J. (1995) · Journal of Applied Psychology · doi:10.1037/0021-9010.80.4.517 · Cited by 180 · see citing works
It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D., & Schwarz, N. (2015) · SAGE Open · doi:10.1177/2158244015584617 · Cited by 173 · see citing works
Learning from feedback: Exactingness and incentives.
Hogarth, R. M., Gibbs, B., McKenzie, C. R. M., & Marquis, M. A. (1991) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.17.4.734 · Cited by 171 · see citing works
Weldon, E., & Gargano, G. M. (1988) · Personality and Social Psychology Bulletin · doi:10.1177/0146167288141016 · Cited by 159 · see citing works
Angner, E., & Loewenstein, G. F. (2012) · Elsevier eBooks · doi:10.1016/b978-0-444-51676-3.50022-1 · Cited by 143 · see citing works
Demand Forecasting Behavior: System Neglect and Change Detection
Kremer, M., Moritz, B. B., & Siemsen, E. (2011) · Management Science · doi:10.1287/mnsc.1110.1382 · Cited by 142 · see citing works
DiStefano, C., & Motl, R. W. (2008) · Personality and Individual Differences · doi:10.1016/j.paid.2008.10.020 · Cited by 122 · see citing works
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118 · see citing works
Predecisional Distortion of Information by Auditors and Salespersons
Russo, J. E., Meloy, M. G., & Wilks, T. J. (2000) · Management Science · doi:10.1287/mnsc.46.1.13.15127 · Cited by 112 · see citing works
Self-citationThe Review Process and the Accuracy of Auditor Judgments
Trotman, K. T. (1985) · Journal of Accounting Research · doi:10.2307/2490836 · Cited by 98 · see citing works
Extra-task performance as a measure of learning a primary task.
Bahrick, H. P., Noble, M. E., & Fitts, P. M. (1954) · Journal of Experimental Psychology · doi:10.1037/h0059267 · Cited by 91 · see citing works
Using Laboratory Experiments to Build Better Operations Management Models
Katok, E. (2011) · Foundations and Trends® in Technology Information and Operations Management · doi:10.1561/0200000022 · Cited by 82 · see citing works
Learning from feedback: Exactingness and incentives.
Hogarth, R. M., Gibbs, B., McKenzie, C. R. M., et al. (1991) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.17.4.734 · Cited by 82 · see citing works
Levin, I. P., Chapman, D. P., & Johnson, R. D. (1988) · Journal of Behavioral Decision Making · doi:10.1002/bdm.3960010105 · Cited by 79 · see citing works
The Sum and Its Parts: Judgmental Hierarchical Forecasting
Kremer, M., Siemsen, E., & Thomas, D. J. (2015) · Management Science · doi:10.1287/mnsc.2015.2259 · Cited by 78 · see citing works
THE USEFULNESS OF BEHAVIORAL LABORATORY EXPERIMENTS IN SUPPLY CHAIN MANAGEMENT RESEARCH
Siemsen, E. (2011) · Journal of Supply Chain Management · doi:10.1111/j.1745-493x.2011.03227.x · Cited by 67 · see citing works
Money vs gaming: Effects of salient monetary payments in double oral auctions
Jamal, K., & Sunder, S. (1991) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(91)90046-v · Cited by 67 · see citing works
Bendoly, E. (2012) · Journal of Operations Management · doi:10.1016/j.jom.2012.06.003 · Cited by 59 · see citing works
Effects of extrinsic incentives on the quality of frequency assessments
Wright, W. F., & Aboul-Ezz, M. E. (1988) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(88)90023-4 · Cited by 52 · see citing works
Meloy, M. G., Russo, J. E., & Miller, E. G. (2006) · Journal of Marketing Research · doi:10.1509/jmkr.43.2.267 · Cited by 51 · see citing works
Self-citationThe Impact of Incorrect Responses to Reverse-Coded Survey Items.
Hughes, G. (2009) · Research in the schools · Cited by 50 · see citing works
Motl, R. W., Conroy, D. E., & Horan, P. M. (2000) · PubMed · Cited by 48 · see citing works
Prioritizing and Monitoring Concurrent Project Work: Effects on Switching Behavior
Bendoly, E., Swink, M. L., & Simpson, W. P. (2013) · Production and Operations Management · doi:10.1111/poms.12083 · Cited by 45 · see citing works
Barnette, J. J. (1999) · Educational and Psychological Measurement · doi:10.1177/0013164499591003 · Cited by 42 · see citing works
Factors influencing the use of internal summary evaluations versus external information in choice
Hulland, J. S., & Kleinmuntz, D. N. (1994) · Journal of Behavioral Decision Making · doi:10.1002/bdm.3960070202 · Cited by 38 · see citing works
The nature and role of demand characteristics in scientific inquiry.
Rosnow, R. L. (2002) · Prevention & Treatment · doi:10.1037/1522-3736.5.1.537c · Cited by 37 · see citing works
Incorporating behavioral theory in OM empirical models
Bendoly, E. (2006) · Journal of Operations Management · doi:10.1016/j.jom.2006.09.003 · Cited by 27 · see citing works
Toward intelligent product information systems for consumers
Russo, J. E. (1987) · Journal of Consumer Policy · doi:10.1007/bf00411632 · Cited by 15 · see citing works
Using Laboratory Experiments to Build Better Operations Management Models
Katok, E. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601984951 · Cited by 7 · see citing works
The nature and role of demand characteristics in scientific inquiry.
Rosnow, R. L. (2002) · Prevention & Treatment · doi:10.1037//1522-3736.5.1.537c · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Consumers’ willingness to pay for upcycled foods
Bhatt, S., Ye, H., Deutsch, J., Ayaz, H., & Suri, R. (2020) · Food Quality and Preference · doi:10.1016/j.foodqual.2020.104035 · Cited by 112 · see citing works
Help me help you: How HR analytics forecasts foster organizational creativity
Di Prima, C., Cepel, M., Kotaskova, A., & Ferraris, A. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2024.123540 · Cited by 46 · see citing works
Perfectionism, Substance Use, and Mental Health in College Students: A Longitudinal Analysis
Nelsen, S. K., Kayaalp, A., & Page, K. J. (2023) · Journal of American College Health · doi:10.1080/07448481.2021.1891076 · Cited by 21 · see citing works
Youth mental health service preferences and utilization patterns among Latinx caregivers
Vázquez, A. L., Alvarez, M. D. L. C., Navarro Flores, C. M., González Vera, J. M., Barrett, T. S., & Domenech Rodríguez, M. M. (2021) · Children and Youth Services Review · doi:10.1016/j.childyouth.2021.106258 · Cited by 14 · see citing works
Vázquez, A. L., Culianos, D., Flores, C. M. N., Alvarez, M. D. L. C., Barrett, T. S., & Domenech Rodríguez, M. M. (2022) · Child & Youth Care Forum · doi:10.1007/s10566-021-09656-8 · Cited by 15 · see citing works
Vázquez, A. L., Navarro Flores, C. M., Alvarez, M. D. L. C., & Domenech Rodríguez, M. M. (2021) · Journal of Latinx Psychology · doi:10.1037/lat0000192 · Cited by 13 · see citing works
Don’t Be Fooled: The Misinformation Effect of Explanations in Human–AI Collaboration
Spitzer, P., Holstein, J., Morrison, K., Holstein, K., Satzger, G., & Kühl, N. (2026) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2574511 · Cited by 12 · see citing works
Vázquez, A. L., Culianos, D., Gudiño, O. G., Navarro Flores, C. M., Barrett, T. S., & Domenech Rodríguez, M. M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302575 · Cited by 3 · see citing works
“Validation methods included contrasting methods used to identify respondents who provided poor quality data (e.g., directed responses, inconsistent reporting, speeding, straight lining, or zigzagging; [21]).”
Method
- Authors
- Abbey, J. D., & Meloy, M. G.
- Affiliations
- Texas A&M University (US)
- Pennsylvania State University (US)
- ORCID
- Abbey, James D.
- Venue
- Journal of Operations Management
- Publisher
- Wiley
- Published
- 2017-07-05
- Volume
- 53-56
- Issue
- 1
- Pages
- 63-70
- ISSN
- 0272-6963 (ISSN-L)
- 1873-1317
- OpenAlex
- W2726093832
- Cited by
- 522
- References
- 80
- Citations by year
- 2026: 80 · 2025: 91 · 2024: 87 · 2023: 79 · 2022: 61 · 2021: 62 · 2020: 32 · 2019: 21 · 2018: 7 · 2017: 2
- FWCI
- 43.27
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Journal of Operations Management (publishedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Management Science and Operations Research › Business Strategies and Management Research
- Keywords
- attention checks, manipulation checks, data quality, response validation, researcher bias, type I error, type II error, attrition rates, incentive alignment, financial compensation, scale validation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
