Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Replication is one of the most important tools for the verification of facts within the empirical sciences. A detailed examination of the notion of replication reveals that there are many different meanings to this concept and the relevant procedures, but hardly any systematic literature. This paper analyzes the concept of replication from a theoretical point of view. It demonstrates that the theoretical demands are scarcely met in everyday work within the social sciences. Some demands are just not feasible, whereas others are constricted by restrictions relating to publication. A new classification scheme based on a functional approach that distinguishes between different types of replication is proposed. Next, it will be argued that replication addresses the important connection between existing and new knowledge. To do so it has to be applied explicitly and systematically. The paper ends with a description of procedures how this could be done and a set of recommendations how to handle the concept of replication in the future to exploit its potential to the full.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“RRs provide a mechanism for publishing replication research”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“which are rarely published in the standard model 13,14 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule [48].”
From The natural selection of bad science (2016) · cites this paper as [48]
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,706
Design and Analysis: A Researcher's Handbook
Beaton, A. E., & Keppel, G. (1975) · American Educational Research Journal · doi:10.2307/1162588 · Cited by 5,822
Scientific Explanation and the Causal Structure of the World
Salmon, W. C. (1985) · Princeton University Press eBooks · doi:10.1515/9780691221489 · Cited by 2,459
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.2307/1869945 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.1086/ahr/92.3.658 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Traynham, J. G., Shapin, S., & Schaffer, S. (1987) · Journal of Interdisciplinary History · doi:10.2307/204299 · Cited by 2,045
Design and Analysis: A Researcher's Handbook
Aaker, D. A., & Keppel, G. (1976) · Journal of Marketing Research · doi:10.2307/3150757 · Cited by 2,011
Research works printed but not linked: 12
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- Beloff J., 1985, The repeatability problem in parapsychology, 1
- Collins H. M., 1985, Changing order
- CookT. D. (1990). Rersearch methodology: Strengthening causal interpretations of nonexperimental data. In SechrestL., PerrinE. & BunkerJ., US Department of health and human services (pp. 9–30). Public Health Service, Agency for Health Care Policy and Research.
- Edge H., 1985, The repeatability problem in parapsychology, 53
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 21
- Rao K. R., 1981, Journal of Parapsychology, 45, 311
- Rosenthal R., 1991, Replication research in the social sciences, 1
- Sargent C. L., 1981, European Journal of Parapsychology, 3, 423
- Schacter D. L., 2000, Wir sind Erinnerung. Gedächtnis und Persönlichkeit
- Walach H., 2000, Neuroendocrinology Letters, 21, 221
- Weizsäcker E. U. v., 1986, Offene Systeme I: Beiträge zur Zeitstruktur von Information, Entropie und Evolution, 82
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
“Direct replication is the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding (7, 8) and is the means of establishing reproducibility of a finding with new data.”
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
“As Schmidt (2009) noted, “within the social sciences, only the discovery of a new fact is credited” (Schmidt, 2009, p. 95; see also Lindsay & Ehrenberg, 1993).”
InfluentialNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
“…and had increased more rapidly in the social and many biological sciences [where theories and methods tend to be less codified and replication is rare (Fanelli 2010b; Schmidt 2009; Evanschitzky et al. 2007; Tsang and Kwan 1999; Kelly 2006; Palmer 2000; Jones et al. 2010; Hubbard and Vetter 1996)].”
BackgroundInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
“Practically, attempts to closely replicate research findings can reveal important differences in what is considered a direct replication (Schmidt, 2009), thus leading to refinements of the initial theory (e.g., Aronson, 1992; Greenwald, Pratkanis, Leippe, & Baumgardner, 1986).”
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule (Schmidt, 2009).”
BackgroundKleiman, E. M., Turner, B. J., Fedor, S., Beale, E. E., Huffman, J. C., & Nock, M. K. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000273 · Cited by 831
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
“Schmidt (2009), in a review connecting the discussion of replication theory with replication practice, eliminated Lykken’s (1968) literal replication (because it essentially requires the original investigator to gather data from additional participants) and reframed the latter two types as direct…”
BackgroundRegistered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716
“A direct replication is the attempt to duplicate the conditions and procedure that existing theory and evidence anticipate as necessary for obtaining the effect (Open Science Collaboration, 2012, 2013; Schmidt, 2009).”
BackgroundThe Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
“…in research is an important component of cumulative science (Asendorpf et al., 2013; Jasny, Chin, Chong, & Vignieri, 2011; Nosek, Spies, & Motyl, 2012; Rosenthal, 1990; Schmidt, 2009), yet relatively few close replication attempts are reported in psychology (Makel, Plucker, & Hegarty, 2012).”
BackgroundFacts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
“Research, including and especially RCTs, requires resources, making replications difficult to conduct if not funded (e.g., Schmidt, 2009).”
BackgroundInfluentialAn Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
“The project attempts direct replications—“repetition of an experimental procedure” in order to “verify a piece of knowledge” (Schmidt, 2009, p. 92, 93).”
MethodThe alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
““With every difference that is introduced the confirmatory power of the replication increases,” because we have shown that the phenomenon does not hinge on a particular operationalization but “generalizes to a larger area of application” (Schmidt, 2009, p. 93).”
BackgroundA Social Context Model of Envy and Social Undermining
Duffy, M. K., Scott, K. L., Shaw, J. D., Tepper, B. J., & Aquino, K. (2012) · Academy of Management Journal · doi:10.5465/amj.2009.0804 · Cited by 557
“Modeling identification as a teamlevel construct challenges the robustness of the Study 1 findings across contexts and operationalizations of key constructs, which improves the value of our replication (Schmidt, 2009).”
BackgroundNelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
“…the frequency of replications (Asendorpf et al. 2013, Koole & Lakens 2012), discussions of how to properly design replications (Brandt et al. 2014, Schmidt 2009), special issues dedicated to replications (Nosek & Lakens 2013, Pashler & Wagenmakers 2012), multilab replication efforts (Ebersole et…”
BackgroundTop 10 Replicated Findings From Behavioral Genetics
Plomin, R., DeFries, J. C., Knopik, V. S., & Neiderhiser, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615617439 · Cited by 517
“Although much has been written about the diagnosis, cause, and prescription for fixing these cracks in the bedrock of psychological science (Ledgerwood, 2014a, 2014b), there is consensus throughout science that the final arbiter is replication ( Jasny, Chin, Chong, & Vignieri, 2011; Schmidt, 2009).”
InfluentialZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Replication, falsification, and the crisis of confidence in social psychology
Earp, B. D., & Trafimow, D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00621 · Cited by 472
“…distinction between “conceptual” or “reproductive” replications (cf. Cartwright, 1991)—which may conceivably be used to bolster confidence in a particular theory— and “direct” or “close” replications, which may be used to bolster confidence in a finding (Schmidt, 2009; see also Earp et al., 2014).”
BackgroundInfluentialPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Nosek, B. A., & Errington, T. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000691 · Cited by 444
“Credibility of scientific claims is established with evidence for their replicability using new data [1].”
On the scientific superiority of conceptual replications for scientific progress
Crandall, C. S., & Sherman, J. W. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.002 · Cited by 344
“But the ability of a phenomenon to be replicated is a necessary condition for widespread acceptance by scientists (Schmidt, 2009).”
BackgroundHow school leadership influences student learning: A test of “The four paths model”
Leithwood, K., Sun, J., & Schumacker, R. (2020) · Educational Administration Quarterly · doi:10.1177/0013161x19878772 · Cited by 329
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
“Credibility of scientific claims is rooted in their independent verifiability (Nosek and Errington, 2020a; Putnam, 1975; Schmidt, 2009).”
BackgroundImplementing Reproducible Research
Stodden, V. (2014) · doi:10.1201/b16868 · Cited by 314
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312
“Developing understanding for the meaning of research findings is often clarified more productively through conceptual replication (Schmidt, 2009).”
InfluentialSoderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
Rewarding Replications: A Sure and Simple Way to Improve Psychological Science
Koole, S. L., & Lakens, D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462586 · Cited by 281
“Systematic efforts at replication are indeed scarce within psychology (Schmidt, 2009; Smith, 1970; Tsang & Kwan, 1999).”
BackgroundMethodInfluential(2021) · Research Design in Clinical Psychology · doi:10.1017/9781108993647.010 · Cited by 276
Marsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
“Others have converged on two (Makel et al., 2012; Schmidt, 2009): (a) ‘direct’, where there are no intentional or significant alterations of the initial study – considered “the means of establishing reproducibility of a finding with new data” (Open Science Collaboration, 2015, p.”
BackgroundInfluentialKröger, H., Pakpahan, E., & Hoffmann, R. (2015) · The European Journal of Public Health · doi:10.1093/eurpub/ckv111 · Cited by 259
A Large Scale Test of the Effect of Social Class on Prosocial Behavior
Korndörfer, M., Egloff, B., & Schmukle, S. C. (2015) · PLOS ONE · doi:10.1371/journal.pone.0133193 · Cited by 255
“As Schmidt [13] put it, “Replication is one of the most important tools for the verification of facts within the empirical sciences” (p.”
InfluentialGupta, V. K., & Batra, S. (2016) · International Small Business Journal: Researching Entrepreneurship · doi:10.1177/0266242615577708 · Cited by 239
“Our use of SME sample contributes to accumulated data in EO (Eden, 2002), which advances cumulative knowledge about EO theory (Schmidt, 2009).”
MethodTHE MEANING OF FAILED REPLICATIONS: A REVIEW AND PROPOSAL
Clemens, M. A. (2017) · Journal of Economic Surveys · doi:10.1111/joes.12139 · Cited by 213
Age at onset and clinical correlates in body dysmorphic disorder
Bjornsson, A. S., Didie, E. R., Grant, J. E., Menard, W., Stalker, E., & Phillips, K. A. (2013) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2013.03.019 · Cited by 245
“Much has been written about the need for replication in science (see eg, [40]).”
BackgroundNosek, B. A., & Errington, T. M. (2017) · eLife · doi:10.7554/elife.23383 · Cited by 242
“Direct replication is defined as attempting to reproduce a previously observed result with a procedure that provides no a priori reason to expect a different outcome (Open Science Collaboration, 2015; Schmidt, 2009).”
Research Design in Clinical Psychology
Kazdin, A. E. (2021) · doi:10.1017/9781108993647 · Cited by 240
Distinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236
“Spreading proof across different lines of experiment—a process called ‘‘conceptual replication’’ [6]—has several consequences for predictive value.”
BackgroundBoss phubbing, trust, job satisfaction and employee performance
Roberts, J. A., & David, M. E. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2019.109702 · Cited by 200
“The second study was designed as a replication study that would also extend study 1 in two important ways (Schmidt, 2009).”
BackgroundRevisiting Mental Simulation in Language Comprehension: Six Replication Attempts
Zwaan, R. A., & Pecher, D. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051382 · Cited by 215
“, [9],[10],[11],[12]) that express concern about the lack of replications in the field.”
BackgroundMethodA Unified Framework to Quantify the Credibility of Scientific Findings
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918787489 · Cited by 214
“To guide the classification of replications according to their methodological similarity to an original study, we use the replication taxonomy depicted in Figure 1 (from LeBel, Berker, Campbell, & Loving, 2017), which is a simplified version of earlier taxonomies (Hendrick, 1991; Schmidt, 2009).”
MethodFinkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202
Psychological Capital Intervention (PCI): A Replication and Extension
Dello Russo, S., & Stoykova, P. (2015) · Human Resource Development Quarterly · doi:10.1002/hrdq.21212 · Cited by 196
Canning, E. A., & Harackiewicz, J. M. (2015) · Motivation Science · doi:10.1037/mot0000015 · Cited by 189
“The importance of replication studies has received considerable attention in psychology and in the social sciences more generally (Makel & Plucker, 2014; Ritchie, Wiseman, & French, 2012; Schmidt, 2009).”
BackgroundJournal of Computational Neuroscience
(2013) · Journal of Computational Neuroscience · doi:10.1007/10827.1573-6873 · Cited by 190
Violence Against Women: Globalizing the Integrated Ecological Model
Fulu, E., & Miedema, S. (2015) · Violence Against Women · doi:10.1177/1077801215596244 · Cited by 178
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 191
Thompson, P. S., Bergeron, D. M., & Bolino, M. C. (2020) · Journal of Applied Psychology · doi:10.1037/apl0000481 · Cited by 171
Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019) · Psychophysiology · doi:10.1111/psyp.13437 · Cited by 184
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J., Berniūnas, R., Boudesseul, J., Colombo, M., et al. (2021) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 179
“…ran an experiment and obtained 2 http://experimental-philosophy.yale.edu/xphipage/Experimental%20Philosophy-Replications.html certain results; I should be able to run the same experiment and obtain the same results, if I am to be justified in placing confidence in what you reported (Schmidt, 2009).”
BackgroundResultInfluentialTo Have Control Over or to Be Free From Others? The Desire for Power Reflects a Need for Autonomy
Lammers, J., Stoker, J. I., Rink, F., & Galinsky, A. D. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216634064 · Cited by 179
“Performing exact replications is valuable by itself (Schmidt, 2009), but replications across cultures offer the additional benefit of providing a robustness test against cultural differences.”
BackgroundDe Los Reyes, A., & Epkins, C. C. (2023) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2022.2158843 · Cited by 166
“…(a) independent research teams investigating the same phenomenon using the same procedures do not arrive at similar conclusions (i.e., replication; Schmidt, 2009) and/or (b) similar conclusions do not manifest when testing a hypothesis―within the same study or dataset―using varied procedures for…”
BackgroundLack of selectivity for syntax relative to word meanings throughout the language network
Fedorenko, E., Blank, I. A., Siegelman, M., & Mineroff, Z. (2020) · Cognition · doi:10.1016/j.cognition.2020.104348 · Cited by 170
“As a result, we selected three paradigms from studies that have argued for syntax selectivity or for dissociations between lexico-semantic and syntactic processing, and that continue to be cited as evidence of such, and attempted to conceptually replicate (Schmidt, 2009) them.”
BackgroundCrowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
“original researchers and other scientists (Baribault et al., 2018; Doyen, Klein, Simons, & Cleeremans, 2014; Earp, in press; Hendrick, 1990; Schmidt, 2009; Simons, 2014).”
BackgroundConceptualizing and evaluating the replication of research results
Fabrigar, L. R., & Wegener, D. T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.009 · Cited by 165
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 165
Replication in prevention science
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0217-6 · Cited by 166
“Perhaps the most pervasive reason for differences in study results is that it is very difficult to exactly repeat even a simple experiment (Schmidt 2009).”
BackgroundZagenczyk, T. J., Gibney, R., Few, W. T., & Scott, K. L. (2011) · Journal of Labor Research · doi:10.1007/s12122-011-9111-z · Cited by 163
Social marketing theory development goals: an agenda to drive change
Rundle-Thiele, S. R., David, P., Willmott, T. J., Pang, B., Eagle, L. C., & Hay, R. (2019) · Journal of Marketing Management · doi:10.1080/0267257x.2018.1559871 · Cited by 204
“Therefore, one of the most important techniques for verification of scientific knowledge is the replication procedure, however, we note that there is limited evidence of replication studies in the social sciences (Schmidt, 2009).”
BackgroundLessons From Pinocchio: Cues to Deception May Be Highly Exaggerated
Luke, T. J. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619838258 · Cited by 159
Reappraising test anxiety increases academic performance of first-year college students.
Brady, S. T., Hard, B. M., & Gross, J. J. (2018) · Journal of Educational Psychology · doi:10.1037/edu0000219 · Cited by 154
Seven Easy Steps to Open Science: An Annotated Reading List
Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., Orben, A., Parsons, S., & Schulte-Mecklenbeck, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000387 · Cited by 153
“Some practical recommendations on data management are provided by Schönbrodt, Gollwitzer, and Abele-Brehm (2017).”
BackgroundRestubog, S. L. D., Zagenczyk, T. J., Bordia, P., Bordia, S., & Chapman, G. J. (2012) · Journal of Management · doi:10.1177/0149206312443557 · Cited by 176
“Indeed, recent research by Schmidt (2009) stresses the important—and largely neglected—role of replication in the behavioral sciences.”
BackgroundReplicability and Reproducibility in Comparative Psychology
Stevens, J. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00862 · Cited by 190
“This is problematic because substituting a different species is not a direct replication (Schmidt, 2009; Makel et al., 2012), and it is not clear what a failure to replicate or an alternative outcome means across species.”
BackgroundReplication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
Philosophy of science and the replicability crisis
Romero, F. (2019) · Philosophy Compass · doi:10.1111/phc3.12633 · Cited by 184
“A second notion that researchers often use is conceptual replication: “Repetition of a test of a hypothesis or a result of earlier research work with different methods” (Schmidt, 2009, p. 91).”
BackgroundMethodInfluentialCross-Validation Approaches for Replicability in Psychology
Koul, A., Becchio, C., & Cavallo, A. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01117 · Cited by 143
“Accordingly, multiple propositions of scope, functioning and typology have been identified (Lykken, 1968; Schmidt, 2009; Hüffmeier et al., 2016; Schmidt and Oh, 2016; LeBel et al., 2017).”
BackgroundLinden, A. H., & Hönekopp, J. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620964193 · Cited by 180
“In this context, the differentiation between close and conceptual replications has become fruitful (S. Schmidt, 2009; Zwaan et al., 2018).”
BackgroundShoss, M. K., Jiang, L., & Probst, T. M. (2016) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000060 · Cited by 184
“Our two studies complement each other in a number of other important ways that enable us to achieve constructive replication (Lykken, 1968; Schmidt, 2009), which emphasizes the use of different methodologies to test underlying conceptual ideas.”
BackgroundStrengthening the Practice of Exercise and Sport-Science Research
Halperin, I., Vigotsky, A. D., Foster, C., & Pyne, D. B. (2018) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.2017-0322 · Cited by 130
“8,50 With direct replication, researchers repeat the methods of the original study as closely as possible.”
BackgroundMethodResultInfluentialJiang, L., & Johnson, M. J. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-017-9509-6 · Cited by 161
Shnabel, N., Bar‐Anan, Y., Kende, A., Bareket, O., & Lazar, Y. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000037 · Cited by 165
Babalola, M. T., Stouten, J., Euwema, M. C., & Ovadje, F. (2018) · Journal of Management · doi:10.1177/0149206316638163 · Cited by 127
“Schmidt (2009) contends that replication indicates underlying assumptions are functional and are transferable for testing new ideas.”
Vogl, E., Pekrun, R. H., Murayama, K., Stockinger, K., & Schubert, S. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02474 · Cited by 167
Well-Being, Personal Success and Business Performance Among Entrepreneurs: A Two-Wave Study
Dijkhuizen, J., Gorgievski, M. J., van Veldhoven, M. J. P. M., & Schalk, R. (2017) · Journal of Happiness Studies · doi:10.1007/s10902-017-9914-6 · Cited by 165
“Longitudinal studies on this relationship are scarce, and a conceptual validation and replication among a more general sample of entrepreneurs is seen as a valuable tool for verification of facts (Schmidt 2009).”
MethodCollecting Psycholinguistic Response Time Data Using Amazon Mechanical Turk
Enochson, K., & Culbertson, J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0116946 · Cited by 143
“While our main aim is to show that AMT, and the web in general, can be fruitfully used to conduct research in this field, we also note the value of replication studies more generally [14,15].”
BackgroundShifting Republican views on climate change through targeted advertising
Goldberg, M. H., Gustafson, A., Rosenthal, S. A., & Leiserowitz, A. (2021) · Nature Climate Change · doi:10.1038/s41558-021-01070-1 · Cited by 146
Research Practices That Can Prevent an Inflation of False-Positive Rates
Murayama, K., Pekrun, R., & Fiedler, K. (2014) · Personality and Social Psychology Review · doi:10.1177/1088868313496330 · Cited by 129
“In a multi-study paper, researchers commonly conceptually replicate their primary findings in subsequent studies (Schmidt, 2009).”
How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?
Kepes, S., & McDaniel, M. A. (2013) · Industrial and Organizational Psychology · doi:10.1111/iops.12045 · Cited by 124
Conceptualizing and evaluating replication across domains of behavioral research
Tackett, J. L., & McShane, B. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000882 · Cited by 124
Understanding replication of experiments in software engineering: A classification
Gómez, O. S., Juristo, N., & Vegas, S. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.04.004 · Cited by 125
“replication AND kind Lüdtke [44] Replication W/2 type Schmidt [61] Radder [56] U”
BackgroundMethodInfluentialGustafson, A., Ballew, M. T., Goldberg, M. H., Cutler, M., Rosenthal, S. A., & Leiserowitz, A. (2020) · Communication Reports · doi:10.1080/08934215.2020.1799049 · Cited by 155
Troyer, M., Kim, J. S., Hale, E., Wantchekon, K. A., & Asher, C. A. (2018) · Reading and Writing · doi:10.1007/s11145-018-9907-9 · Cited by 180
“The primary aim of this study is to conduct a conceptual replication (Schmidt, 2009)—a replication study that aims to confirm prior hypotheses using established measures and analytic models.”
BackgroundThe (mis)measurement of the Dark Triad Dirty Dozen: exploitation at the core of the scale
Kajonius, P. J., Persson, B. N., Rosenberg, P., & Garcia, D. (2016) · PeerJ · doi:10.7717/peerj.1748 · Cited by 115
“In short, replication should be at ‘‘the heart of science’’ (Schmidt, 2009).”
Ji, J. L., Holmes, E. A., & Blackwell, S. E. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2016.11.025 · Cited by 122
“This provides a direct replication (Schmidt, 2009) of Blackwell et al.”
Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli
Westfall, J., Judd, C. M., & Kenny, D. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691614564879 · Cited by 119
“Ultimately, the goal of direct replication is to confirm that a particular experimental procedure can reliably produce a particular empirical result (Schmidt, 2009).”
BackgroundInfluentialReplication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 126
“Conceptual replications purposefully alter factors such as participant demographics, operationalization of dependent variables, or study context (see Schmidt, 2009, Figure 1).”
BackgroundIncreasing clarity where it is needed most: articulating and evaluating theoretical contributions
DeAndrea, D. C., & Holbert, R. L. (2017) · Annals of the International Communication Association · doi:10.1080/23808985.2017.1304163 · Cited by 117
“Constructs can be measured and manipulated in many ways; the more we can establish that theoretical relationships are not dependent on very particular methods or procedures, the more confident we can be in the underlying theory (Schmidt, 2009; Stroebe & Strack, 2014).”
BackgroundMethodReplication, Communication, and the Population Dynamics of Scientific Discovery
McElreath, R., & Smaldino, P. E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0136088 · Cited by 117
“probability that an hypothesis with a given tally is correct? For any tally s, we can calculate: Pr(truejs) = pˆ T,s pˆ T,s + pˆ F,s , Pr(sjtrue) = pˆ T,s å i pˆ T,i , Pr(sjfalse) = pˆ F,s å i pˆ F,i (4) The precision of a tally s is Pr(truejs), the proportion of hypotheses with tally s that are true. The sensitivity,…”
BackgroundDietz, B., van Knippenberg, D., Hirst, G., & Restubog, S. L. D. (2015) · Journal of Applied Psychology · doi:10.1037/a0038888 · Cited by 136
Replication Research and Special Education
Travers, J. C., Cook, B. G., Therrien, W. J., & Coyne, M. D. (2016) · Remedial and Special Education · doi:10.1177/0741932516648462 · Cited by 125
“Relatedly, conceptual replication allows for the detection and understanding of the boundaries where a theory fails to account for results (Schmidt, 2009), which may stimulate new avenues of research and different theories with sufficient explanatory and predictive power.”
BackgroundMethodCoyne, M. D., Cook, B. G., & Therrien, W. J. (2016) · Remedial and Special Education · doi:10.1177/0741932516648463 · Cited by 128
“…role of more distal replications, as part of a carefully designed series of conceptual replications, is to systematically investigate the effects of an intervention under different conditions by purposely manipulating theoretically important instructional and contextual variables (Schmidt, 2009).”
BackgroundInfluentialReplicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Świątkowski, W., & Dompnier, B. (2017) · International Review of Social Psychology · doi:10.5334/irsp.66 · Cited by 114
“Until recently, replicating existing studies has been no more than just a rare phenomenon in psychological science (Schmidt, 2009).”
BackgroundLearning From Testimony on Quantitative Research in Management
King, A. A., Goldfarb, B. D., & Simcoe, T. S. (2019) · Academy of Management Review · doi:10.2139/ssrn.3816190 · Cited by 135
The NIH Toolbox Pattern Comparison Processing Speed Test: Normative Data
Carlozzi, N. E., Beaumont, J. L., Tulsky, D. S., & Gershon, R. C. (2015) · Archives of Clinical Neuropsychology · doi:10.1093/arclin/acv031 · Cited by 115
Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPreprint
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/k4bgq · Cited by 115
When Lightning Strikes Twice: Profoundly Gifted, Profoundly Accomplished
Makel, M. C., Kell, H. J., Lubinski, D., Putallaz, M., & Benbow, C. P. (2016) · Psychological Science · doi:10.1177/0956797616644735 · Cited by 113
“…this idea in two ways: collaterally, comparing corresponding criterion classes of the two samples, for the purposes of direct replication (S. Schmidt, 2009), and, sequentially, comparing the three criterion classes over time within each sample, for ascertaining the longitudinal potency of…”
MethodReconsidering the expectancy disconfirmation model. Three experimental replications
Grimmelikhuijsen, S., & Porumbescu, G. A. (2017) · Public Management Review · doi:10.1080/14719037.2017.1282000 · Cited by 131
“Replication is an important feature of scientific knowledge (e.g. Tsang and Kwan 1999; Schmidt 2009).”
BackgroundMethodFalsifiability is not optional.
LeBel, E. P., Berger, D., Campbell, L., & Loving, T. J. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000106 · Cited by 105
Devezer, B., Nardin, L. G., Baumgaertner, B., & Buzbas, E. O. (2019) · PLOS ONE · doi:10.1371/journal.pone.0216125 · Cited by 107
“Introduction If obtained independently of each other, multiple confirmations of a claim in a scientific inquiry lend credibility to that claim [1,2].”
BackgroundVickovic, S. G., & Morrow, W. J. (2019) · Criminal Justice Review · doi:10.1177/0734016819863099 · Cited by 146
Metadata
- Authors
- Schmidt, S.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Schmidt, Stefan
- ORCID
- Schmidt, Stefan
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 90-100
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2021458971
- Cited by
- 1,125
- References
- 55
- Citations by year
- 2026: 42 · 2025: 95 · 2024: 88 · 2023: 85 · 2022: 75 · 2021: 138 · 2020: 94 · 2019: 105 · 2018: 93 · 2017: 74 · 2016: 84 · 2015: 52 · 2014: 35 · 2013: 26 · 2012: 23
- FWCI
- 16.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of General Psychology (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, social sciences, fact-checking, publication practices, research methodology, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.