The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Are Manipulation Checks Necessary?
Open accessAbstract
Researchers are concerned about whether manipulations have the intended effects. Many journals and reviewers view manipulation checks favorably, and they are widely reported in prestigious journals. However, the prototypical manipulation check is a verbal (rather than behavioral) measure that always appears at the same point in the procedure (rather than its order being varied to assess order effects). Embedding such manipulation checks within an experiment comes with problems. While we conceptualize manipulation checks as measures, they can also act as interventions which initiate new processes that would otherwise not occur. The default assumption that manipulation checks do not affect experimental conclusions is unwarranted. They may amplify, undo, or interact with the effects of a manipulation. Further, the use of manipulation checks in mediational analyses does not rule out confounding variables, as any unmeasured variables that correlate with the manipulation check may still drive the relationship. Alternatives such as non-verbal and behavioral measures as manipulation checks and pilot testing are less problematic. Reviewers should view manipulation checks more critically, and authors should explore alternative methods to ensure the effectiveness of manipulations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences
Cohen, P., Cohen, P., West, S. G., & Aiken, L. S. (2014) · Psychology Press eBooks · doi:10.4324/9781410606266 · Cited by 20,737
(1963) · Medical Entomology and Zoology · Cited by 15,085
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
MacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085542 · Cited by 5,688
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“…checks, experimental methods, research design, mediation, emotion INTRODUCTION The past decade has witnessed a growing malaise about the validity of scientific research, particularly in medicine and psychology (Ioannidis, 2005; Pashler and Wagenmakers, 2012; Open Science Collaboration, 2015).”
Background(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences.
Fong, G. T., Cohen, J., & Cohen, P. (1985) · Journal of the American Statistical Association · doi:10.2307/2287930 · Cited by 4,475
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Self-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846
Spencer, S. J., Zanna, M. P., & Fong, G. T. (2005) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.89.6.845 · Cited by 2,452
Explanation in Causal Inference: Methods for Mediation and Interaction
VanderWeele, T. J. (2015) · Cited by 2,217
Linguistic Inquiry and Word Count (LIWC): Pronounced “Luke,” . . . and Other Useful Facts
Chung, C. K., & Pennebaker, J. W. (2012) · Applied Natural Language Processing · doi:10.4018/978-1-60960-741-8.ch012 · Cited by 2,151
(2016) · Cambridge University Press eBooks · doi:10.1017/9781107415782 · Cited by 2,031
Unobtrusive Measures: Nonreactive Research in the Social Sciences
Page, E. B., Webb, E. J., Campell, D. T., Schwart, R. D., & Sechrest, L. (1966) · American Educational Research Journal · doi:10.2307/1162043 · Cited by 1,968
Strack, F., Martin, L. L., & Stepper, S. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.5.768 · Cited by 1,898
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Self-citationEffects of an expanding-spaced vs massed exposure schedule on fear reduction and return of fear
Rowe, M. K., & Craske, M. G. (1998) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(97)10016-x · Cited by 1,676
Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00051.x · Cited by 1,629
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 1×
“MTurkers have been incentivized to be equally or more attentive than college subject pool students (Hauser and Schwarz, 2016), especially MTurkers who meet certain worker restrictions that are easy to implement (Peer et al., 2014).”
BackgroundBeech, H. R. (1963) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(63)90032-1 · Cited by 1,539
Methods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey
Galesic, M., & Bosnjak, M. (2009) · Public Opinion Quarterly · doi:10.1093/poq/nfp031 · Cited by 1,297
Lieberman, M. D., Eisenberger, N. I., Crockett, M. J., Tom, S. M., Pfeifer, J. H., & Way, B. M. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01916.x · Cited by 1,150 · cited 1×
“(2007) found that asking participants to name the emotion in pictures of emotional facial expressions reduced the response of the amygdala to the faces (Creswell et al., 2007; Lieberman et al., 2007).”
BackgroundCounterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2014) · Cambridge University Press · doi:10.1017/cbo9781107587991 · Cited by 1,037
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951
The nonverbal mediation of self-fulfilling prophecies in interracial interaction
Word, C. O., Zanna, M. P., & Cooper, J. (1974) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(74)90059-6 · Cited by 902
Neural correlates of dispositional mindfulness during affect labeling
Creswell, J. D., Way, B. M., Eisenberger, N. I., & Lieberman, M. D. (2007) · Psychosomatic Medicine · doi:10.1097/psy.0b013e3180f6171f · Cited by 835
Attitude construction: evaluation in context
Schwarz, N. (2007) · Social Cognition · doi:10.1521/soco.2007.25.5.638 · Cited by 627
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625
Design approaches to experimental mediation
Pirlott, A. G., & MacKinnon, D. P. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.012 · Cited by 575
Methods of Research in Social Psychology.
Singleton, R., Carlsmith, J. M., Ellsworth, P. C., & Aronson, E. (1977) · Contemporary Sociology · doi:10.2307/2064798 · Cited by 437
Self-citationRegistered Replication Report: Strack, Martin, & Stepper (1988)
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616674458 · Cited by 430
Methods of research in social psychology
Carlsmith, M. (1976) · Medical Entomology and Zoology · Cited by 384
O’Reilly, J., Eichhorst, W., Gábos, A., Hadjivassiliou, K., Lain, D., Leschke, J., McGuinness, S., Kureková, L. M., Nazio, T., Ortlieb, R., et al. (2015) · Sage Open · doi:10.1177/2158244015574962 · Cited by 360
Bem, D. J., & McConnell, H. K. (1970) · Journal of Personality and Social Psychology · doi:10.1037/h0020916 · Cited by 349
Unobtrusive Measures: Non-Reactive Research in the Social Sciences.
Hill, R. J., Webb, E. J., Campbell, D. T., Swartz, R. D., & Sechrest, L. (1966) · American Sociological Review · doi:10.2307/2090801 · Cited by 339
The Psychology of Affiliation: Experimental Studies of the Sources of Gregariousness
Musteikis, A., & Schachter, S. (1961) · The American Catholic Sociological Review · doi:10.2307/3709132 · Cited by 328
The Influence of Attributions on the Relevance of Negative Feelings to Personal Satisfaction
Kehner, D., Locke, K. D., & Aurain, P. C. (1993) · Personality and Social Psychology Bulletin · doi:10.1177/0146167293191003 · Cited by 245
Registered Replication Report: Schooler and Engstler-Schooler (1990)
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., et al. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614545653 · Cited by 223
Sigall, H., & Mills, J. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_5 · Cited by 217
Kassam, K., & Mendes, W. B. (2013) · PLoS ONE · doi:10.1371/journal.pone.0064959 · Cited by 199
An Introduction to Registered Replication Reports at Perspectives on Psychological Science
Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614543974 · Cited by 183
It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D. J., & Schwarz, N. (2015) · Sage Open · doi:10.1177/2158244015584617 · Cited by 168
Self-citationContext Effects in Attitude Surveys: Applying Cognitive Theory to Social Research
Schwarz, N., & Strack, F. (1991) · European Review of Social Psychology · doi:10.1080/14792779143000015 · Cited by 159
A study of catharsis of aggression.
Mallick, S. K., & McCandless, B. R. (1966) · Journal of Personality and Social Psychology · doi:10.1037/h0023987 · Cited by 150
Noah, T., Schul, Y., & Mayo, R. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000121 · Cited by 120
On the Limitations of Manipulation Checks: An Obstacle Toward Cumulative Science
Fayant, M., Sigall, H., Lemonnier, A., Retsin, E., & Alexopoulos, T. (2017) · International Review of Social Psychology · doi:10.5334/irsp.102 · Cited by 113
Kagan, J. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00049.x · Cited by 84
Research Methods in Cognition and Emotion
Parrott, W. G., & Hertel, P. T. (1999) · Wiley eBooks · doi:10.1002/0470013494.ch4 · Cited by 79
Reflection on the Smiling Registered Replication Report
Strack, F. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616674460 · Cited by 78
Manipulation checks: Advantage or disadvantage?
Kidd, R. F. (1976) · Cited by 69
Schwarz (1999) · doi:10.1097/acm.0000000000002002 · Cited by 67
Using analysis of covariance (ANCOVA) with fallible covariates.
Culpepper, S. A., & Aguinis, H. (2011) · Psychological Methods · doi:10.1037/a0023355 · Cited by 65
Commentary: Using Potential Outcomes to Understand Causal Mediation Analysis
Imai, K., Jo, B., & Stuart, E. A. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.606743 · Cited by 58
Manipulation Checks as Manipulation: Another Look at the Ease-of-Retrieval Heuristic
Kühnen, U. (2009) · Personality and Social Psychology Bulletin · doi:10.1177/0146167209346746 · Cited by 55
Commentary: Are Three Waves of Data Sufficient for Assessing Mediation?
Reichardt, C. S. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.606740 · Cited by 53
Learning and the replicability of priming effects
Ramscar, M. (2016) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2016.07.001 · Cited by 40
Schwarz, N., & Strack, F. (2014) · Cited by 38
An alternative approach to analysis of mental states in experimental social cognition research
Lench, H. C., Taylor, A. B., & Bench, S. W. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0351-0 · Cited by 37
TACIT: An open-source text analysis, crawling, and interpretation tool
Dehghani, M., JOHNSON, K., Garten, J., Balasubramanian, V., Singh, A., Shankar, Y., Rajkumar, A., Parmar, N., Joe, H., Pulickal, L., et al. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0722-4 · Cited by 37
Open Science CollaborationPreprint
Nosek, B. A., Spies, J. R., Cohn, M., Bartmess, E. N., Lakens, D., Holman, D., Cohoon, J., Lewis, M., Gordon-McKeon, S., & IJzerman, H. (2012) · OSF Preprints (OSF Preprints) · Cited by 29
Registered Replication Reports
Simons, D. J. (2014) · APS observer · Cited by 16
THE RISE AND FALL OF THE HIGH-IMPACT EXPERIMENT
Ellsworth, P. C. (2011) · doi:10.4324/9780203848012-15 · Cited by 8
Self-citationNot by Chance Alone: My Life as a Social Psychologist
Aronson, E. (2010) · Cited by 6
Cited by
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
Texting with Humanlike Conversational Agents: Designing for Anthropomorphism
Seeger, A.-M., University of Mannheim, G., Pfeiffer, J., Justus Liebig University Giessen, G., Heinzl, A., & University of Mannheim, G. (2021) · Journal of the Association for Information Systems · doi:10.17705/1jais.00685 · Cited by 200
Obtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts
Wiernik, B. M., & Dahlke, J. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919885611 · Cited by 135
Potential negative consequences of mindfulness in the moral domain
Schindler, S., Pfattheicher, S., & Reinhard, M. (2019) · European Journal of Social Psychology · doi:10.1002/ejsp.2570 · Cited by 116
Wise reasoning benefits from emodiversity, irrespective of emotional intensity.
Grossmann, I., Oakes, H., & Santos, H. C. (2019) · Journal of Experimental Psychology: General · doi:10.1037/xge0000543 · Cited by 85
Metadata
- Authors
- Hauser, D. J., Ellsworth, P. C., & Gonzalez, R.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2018-06-21
- Volume
- 9
- Pages
- 998-998
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W2806284750
- Cited by
- 480
- References
- 70
- FWCI
- 26.81
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.