Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Replication is one of the most important tools for the verification of facts within the empirical sciences. A detailed examination of the notion of replication reveals that there are many different meanings to this concept and the relevant procedures, but hardly any systematic literature. This paper analyzes the concept of replication from a theoretical point of view. It demonstrates that the theoretical demands are scarcely met in everyday work within the social sciences. Some demands are just not feasible, whereas others are constricted by restrictions relating to publication. A new classification scheme based on a functional approach that distinguishes between different types of replication is proposed. Next, it will be argued that replication addresses the important connection between existing and new knowledge. To do so it has to be applied explicitly and systematically. The paper ends with a description of procedures how this could be done and a set of recommendations how to handle the concept of replication in the future to exploit its potential to the full.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“RRs provide a mechanism for publishing replication research”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“which are rarely published in the standard model 13,14 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule [48].”
From The natural selection of bad science (2016) · cites this paper as [48]
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,706
Design and Analysis: A Researcher's Handbook
Beaton, A. E., & Keppel, G. (1975) · American Educational Research Journal · doi:10.2307/1162588 · Cited by 5,822
Scientific Explanation and the Causal Structure of the World
Salmon, W. C. (1985) · Princeton University Press eBooks · doi:10.1515/9780691221489 · Cited by 2,459
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.2307/1869945 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.1086/ahr/92.3.658 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Traynham, J. G., Shapin, S., & Schaffer, S. (1987) · Journal of Interdisciplinary History · doi:10.2307/204299 · Cited by 2,045
Design and Analysis: A Researcher's Handbook
Aaker, D. A., & Keppel, G. (1976) · Journal of Marketing Research · doi:10.2307/3150757 · Cited by 2,011
Research works printed but not linked: 12
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- Beloff J., 1985, The repeatability problem in parapsychology, 1
- Collins H. M., 1985, Changing order
- CookT. D. (1990). Rersearch methodology: Strengthening causal interpretations of nonexperimental data. In SechrestL., PerrinE. & BunkerJ., US Department of health and human services (pp. 9–30). Public Health Service, Agency for Health Care Policy and Research.
- Edge H., 1985, The repeatability problem in parapsychology, 53
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 21
- Rao K. R., 1981, Journal of Parapsychology, 45, 311
- Rosenthal R., 1991, Replication research in the social sciences, 1
- Sargent C. L., 1981, European Journal of Parapsychology, 3, 423
- Schacter D. L., 2000, Wir sind Erinnerung. Gedächtnis und Persönlichkeit
- Walach H., 2000, Neuroendocrinology Letters, 21, 221
- Weizsäcker E. U. v., 1986, Offene Systeme I: Beiträge zur Zeitstruktur von Information, Entropie und Evolution, 82
Cited by
Items 101–200 of 1,001
Gender and Social‐Cognitive Development
Leaper, C. (2015) · Handbook of Child Psychology and Developmental Science · doi:10.1002/9781118963418.childpsy219 · Cited by 90
Nakagawa, S., & Parker, T. H. (2015) · BMC Biology · doi:10.1186/s12915-015-0196-3 · Cited by 110
“While a close replication attempts to duplicate experimental methods, a conceptual replication uses a distinctly different study designed to test the same hypothesis [17].”
BackgroundMiłkowski, M., Hensel, W. M., & Hohol, M. (2018) · Journal of Computational Neuroscience · doi:10.1007/s10827-018-0702-z · Cited by 112
“One distinction that has become influential is the contrast between direct and conceptual replications (Hüffmeier et al. 2016; Schmidt2009 ; Stroebe and Strack 2014; Zwaan et al. 2018).”
BackgroundSchmidt, S., Naranjo, J. R., Brenneisen, C., Gundlach, J., Schultz, C., Kaube, H., Hinterberger, T., & Jeanmonod, D. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031138 · Cited by 114
“more general sample [23].”
Self-citationBest Practice Recommendations for Replicating Experiments in Public Administration
Walker, R. M., Brewer, G. A., Lee, M. J., Petrovsky, N., & van Witteloostuijn, A. C. (2018) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muy047 · Cited by 118
Treadway, D. C., Breland, J. W., Williams, L. M., Cho, J., Yang, J., & Ferris, G. R. (2011) · Journal of Management · doi:10.1177/0149206311410887 · Cited by 139
“This constructive replication design offers the opportunity to contribute substantially to research knowledge by increasing confidence in the validity of hypothesized relationships being tested and, furthermore, to enrich and bolster theory (Eden, 2002; Schmidt, 2009).”
BackgroundResultInfluentialEmpirical research in requirements engineering: trends and opportunities
Ambreen, T., Ikram, N., Usman, M., & Niazi, M. (2016) · Requirements Engineering · doi:10.1007/s00766-016-0258-2 · Cited by 126
Couple Resilience to Economic Pressure Over Time and Across Generations
Masarik, A. S., Martin, M. J., Ferrer, E. E., Lorenz, F. O., Conger, K. J., & Conger, R. D. (2016) · Journal of Marriage and the Family · doi:10.1111/jomf.12284 · Cited by 125
“Furthermore, because systematic replication is essential for improved understanding in any empirical science (e.g., Schmidt, 2009), another aim of our study was to assess the degree of replication for the aforementioned stress and resilience processes in two generations of romantic couples.,…”
Methods for Heart Rate Variability Biofeedback (HRVB): A Systematic Review and Guidelines
Lalanza, J. F., Lorente, S., Bullich, R., García, C., Losilla, J.-M., & Capdevila, L. (2023) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-023-09582-6 · Cited by 117
“One of the fundamental pillars in science is replication, which consists in re-examining research findings independently to increase their validity (Schmidt, 2009).”
BackgroundHochwarter, W. A., & Thompson, K. W. (2012) · Human Relations · doi:10.1177/0018726711430003 · Cited by 98
“To address concerns associated with single-study designs, scholars have encouraged the development of replication studies (Schmidt, 2009).”
BackgroundThe fundamental principles of reproducibility
Gundersen, O. E. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0210 · Cited by 90
“ity is independent investigators subjecting the original data to their own analyses and interpretations. Drummond [23] "Reproducibility requires changes; replicability avoids them." Schmidt [24] Distinguishes between two fundamental levels of replication: 1) the narrow bounded notion that is a repetition of…”
BackgroundMethodInfluentialFrom law to folklore: work stress and the Yerkes-Dodson Law
Corbett, M. (2015) · Journal of Managerial Psychology · doi:10.1108/jmp-03-2013-0085 · Cited by 123
Replication of empirical studies in software engineering research: a systematic mapping study
da Silva, F. Q. B., Suassuna, M., França, A. C. C., Grubb, A. M., Gouveia, T. B., Monteiro, C. V. F., & dos Santos, I. E. (2012) · Empirical Software Engineering · doi:10.1007/s10664-012-9227-7 · Cited by 126
“is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained” (Schmidt 2009).”
BackgroundMethodResultInfluentialSng, C. Y., Carter, M., & Stephenson, J. R. (2018) · Autism & Developmental Language Impairments · doi:10.1177/2396941518803806 · Cited by 125
Metaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F., Chee, Y. E., Wintle, B. C., Burgman, M. A., McCarthy, M. A., & Gordon, A. (2017) · BioScience · doi:10.1093/biosci/biw159 · Cited by 94
A Call for Examining Replication and Bias in Special Education Research
Cook, B. G. (2014) · Remedial and Special Education · doi:10.1177/0741932514528995 · Cited by 108
““The lack of interest in replication is striking given its centrality to science” (Nosek et al., 2012, p. 618; see also Schmidt, 2009).”
BackgroundReplication of Special Education Research: Necessary but Far Too Rare
Makel, M. C., Plucker, J. A., Freeman, J., Lombardi, A. R., Simonsen, B. M., & Coyne, M. D. (2016) · Remedial and Special Education · doi:10.1177/0741932516646083 · Cited by 109
“We most often think of replication as direct replication, duplicating exactly the methods and procedures of an original study (Schmidt, 2009).”
MethodJin, H., Mikeska, J. N., Hokayem, H., & Mavronikolas, E. (2019) · Science Education · doi:10.1002/sce.21525 · Cited by 131
Replications types in experimental disciplines
Gómez, O. S., Juristo, N., & Vegas, S. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852790 · Cited by 94
Replication of Software Engineering Experiments
Juristo, N., & Gómez, O. S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-25231-0_2 · Cited by 113
Machery, É. (2020) · Philosophy of Science · doi:10.1086/709701 · Cited by 110
Replication in criminology: A necessary practice
McNeeley, S., & Warner, J. J. (2015) · European Journal of Criminology · doi:10.1177/1477370815578197 · Cited by 99
“An additional benefit of replication, according to Schmidt (2009), is its ability to address possible issues with internal and external validity.”
BackgroundSex differences in the right tail of cognitive abilities: An update and cross cultural extension
Makel, M. C., Wai, J., Peairs, K. F., & Putallaz, M. (2016) · Intelligence · doi:10.1016/j.intell.2016.09.003 · Cited by 95
Jiang, L., & Probst, T. M. (2017) · Journal of Applied Psychology · doi:10.1037/apl0000179 · Cited by 91
“In an attempt to constructively replicate (Schmidt, 2009) the results from Study 1, Study 2 collected data from 402 employees residing in 48 states in the United States, which provided converging evidence of the exacerbating effect of income inequality operating at the state level on the JI-burnout (i.”
BackgroundDoes Intuition Cause Cooperation?
Verkoeijen, P. P. J. L., & Bouwmeester, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096654 · Cited by 124
“These replications are important because they allow us to assess the reliability of the intuitivecooperation effect with Mechanical Turk participants (see [2], [3], [4], [5], [6] and [7] for papers on replication in psychological research).”
BackgroundRETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 97
“Science progressively learns about the world through the discovery of replicable findings 1,2 .”
BackgroundRetractedRetrieval attempts enhance learning, but retrieval success (versus failure) does not matter.
Kornell, N., Klein, P. J., & Rawson, K. A. (2015) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0037850 · Cited by 92
“Experiments 4–6 were close replications of Experiments 1–3, respectively, based on recent emphasis on the importance of replicating novel findings (e.g., Ledgerwood & Sherman, 2012; Pashler & Harris, 2012; Roediger, 2012; Schmidt, 2009).”
MethodMaking Replication MainstreamPreprint
Zwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, B. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4tg9c · Cited by 92
Estimating the Reproducibility of Psychological Science
Nosek, B. A., Cohoon, J., Kidwell, M., & Spies, J. R. (2016) · doi:10.31219/osf.io/447b3 · Cited by 88
“Direct replication is the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding (7, 8) and is the means of establishing reproducibility of a finding with new data.”
BackgroundWe need more replication research – A case for test-retest reliability
Leppink, J., & Pérez-Fuster, P. (2017) · Perspectives on Medical Education · doi:10.1007/s40037-017-0347-z · Cited by 110
Koletić, G. (2017) · Journal of Adolescence · doi:10.1016/j.adolescence.2017.04.006 · Cited by 106
“…will strongly depend on the availability of replication studies to minimize the risk of erroneous conclusions that may have occurred due to sampling errors, sample characteristics, lack of internal validity, fraud, contextual background of the research or data collection procedures (Schmidt, 2009).”
BackgroundIsager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner‐Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., Veer, A. V. '., et al. (2021) · Psychological Methods · doi:10.1037/met0000438 · Cited by 105
Maniadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 105
“12 To add more detail, in the empirical part, we use the similar – but extended – definitions of direct and conceptual replications of Schmidt (2009), summarised by Makel et al. (2012):”
InfluentialA conceptual replication of ambidextrous leadership theory: An experimental approach
Klonek, F. E., Gerpott, F. H., & Parker, S. K. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101473 · Cited by 72
Gupta, V. K., Mortal, S. C., & Guo, X. (2018) · Strategic Management Journal · doi:10.1002/smj.2905 · Cited by 95
Scientific inbreeding and same-team replication: Type D personality as an example
Ioannidis, J. P. A. (2012) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2012.09.014 · Cited by 93
“The story may be compounded by efforts at conceptual replication [5] exploring diverse but conceptually related associations done by the same team or by followers thereof.”
BackgroundTwo Different Communication Genres and Implications for Vocabulary Development and Learning to Read
Massaro, D. W. (2015) · Journal of Literacy Research · doi:10.1177/1086296x15627528 · Cited by 95
“In the present constructive or conceptual replication (Makel & Plucker, 2014; Schmidt, 2009) and analysis, the written language sample consisted of the complete text of 112 picture books for a total of more than 57,000 words.”
BackgroundPostretrieval new learning does not reliably induce human memory updating via reconsolidation
Hardwicke, T. E., Taqi, M., & Shanks, D. R. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1601440113 · Cited by 84
“We conducted a replication battery (31) consisting of both “direct replications” (32) that followed the methodology of the original study as closely as possible, and “conceptual replications” (33) that manipulated key task parameters to explore the broader validity of the reconsolidation-updating theory.”
MethodMackey, J. D., Ellen, B. P., Hochwarter, W. A., & Ferris, G. R. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2013.07.003 · Cited by 98
Ivanova, I. V., Tasca, G. A., Hammond, N., Balfour, L., Ritchie, K., Koszycki, D., & Bissada, H. (2015) · European Eating Disorders Review · doi:10.1002/erv.2344 · Cited by 83
“The limited number of replications of research in psychology has been criticized (Schmidt, 2009; Tsang & Kwan, 1999), particularly after a number of high-profile failures to replicate results in social psychological studies (Carpenter, 2012).”
BackgroundExamining retail customer experience and the moderation effect of loyalty programmes
Lin, Z., & Bennett, D. R. (2014) · International Journal of Retail & Distribution Management · doi:10.1108/ijrdm-11-2013-0208 · Cited by 110
“Such studies, based on single sets of data are rarely re-tested or even confirmed (Ehrenberg, 1990, Schmidt, 2009) and therefore little useful knowledge is accumulated.”
BackgroundMethodReplications of software engineering experiments
Carver, J. C., Juristo, N., Baldassarre, M. T., & Vegas, S. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9290-8 · Cited by 88
“Schmidt recently noted for the social sciences that “The word replication has been used as a collective term to describe various meanings in different contexts” (Schmidt 2009).”
BackgroundSecurity Smells in Ansible and Chef Scripts: A Replication Study
Rahman, A., Rahman, M. R., Parnin, C., & Williams, L. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3408897 · Cited by 80
“We hypothesize through a replication [45] of our prior work, we can systematically identify security smells for other languages namely, Ansible and Chef.”
MethodExpanding STEM opportunities through inclusive STEM‐focused high schools
Means, B., Wang, H., Wei, X., Lynch, S. J., Peters, V. L., Young, V. M., & Allen, C. D. (2017) · Science Education · doi:10.1002/sce.21281 · Cited by 92
Token Economy: A Systematic Review of Procedural Descriptions
Ivy, J. W., Meindl, J. N., Overley, E., & Robson, K. M. (2017) · Behavior Modification · doi:10.1177/0145445517699559 · Cited by 96
“Replication is a hallmark feature of the scientific process (Schmidt, 2009).”
BackgroundPhysiology is hard: a replication study of students’ perceived learning difficulties
Slominski, T., Grindberg, S., & Momsen, J. L. (2019) · AJP Advances in Physiology Education · doi:10.1152/advan.00040.2018 · Cited by 90
“we can generalize results to different populations (19).”
ResultFrom Repeatability to Reproducibility and Corroboration
Feitelson, D. G. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723875 · Cited by 94
“So if the program is a simulator or even a complete system, and you have all the needed inputs and get the same result, this does not verify the result — it just verifies that you managed to repeat the original execution [53].”
BackgroundMethodResultInfluentialTowards reproducibility in recommender-systems research
Beel, J., Breitinger, C., Langer, S., Lommatzsch, A., & Gipp, B. (2016) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-016-9174-x · Cited by 108
“In the medical sciences, social sciences, and natural sciences the issue of reproducibility has already been discussed and analyzed in depth (Open Science Collaboration and others 2015; Deyo et al. 1991; Downing 2004; Flyvbjerg 2001; Hoeymans et al. 1997; McNutt 2014; Rothwell and Martyn 2000; Schmidt 2009).”
BackgroundKeating, D. M., & Totzkay, D. (2019) · Annals of the International Communication Association · doi:10.1080/23808985.2019.1632218 · Cited by 77
“Yet, the ultimate goal of replication is to be informative (Earp & Trafimow, 2015), and how informative replication attempts are depends on the goals and methods of such attempts (Schmidt, 2009).”
BackgroundMethodHendriks, F., Kienhues, D., & Bromme, R. (2020) · Public Understanding of Science · doi:10.1177/0963662520902383 · Cited by 73
McAllister, C. P., Harris, J. N., Hochwarter, W. A., Perrewé, P. L., & Ferris, G. R. (2016) · Journal of Business and Psychology · doi:10.1007/s10869-016-9441-1 · Cited by 87
“Our constructive replication (e.g., differing measures for outcomes, multiple rating sources) across the four studies helps establish a strong empirical foundation for future research in this area by adding greater confidence in the validity of the obtained results (Schmidt 2009).”
BackgroundMethodKapoutsis, I., Papalexandris, A., Nikolopoulos, A. G., Hochwarter, W. A., & Ferris, G. R. (2010) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2010.09.009 · Cited by 92
Campbell, R., Bybee, D., Townsend, S. M., Shaw, J., Karim, N., & Markowitz, J. (2014) · Violence Against Women · doi:10.1177/1077801214536286 · Cited by 72
“(2012), we conducted a conceptual replication, whereby we retained the same focal research question, but deliberately changed both the study’s settings and some elements of the methodology (Schmidt, 2009).”
BackgroundMethodThe replication crisis: How might philosophy and theory of psychology be of use?
Morawski, J. G. (2019) · Journal of Theoretical and Philosophical Psychology · doi:10.1037/teo0000129 · Cited by 78
“Among the unresolved matters is how to treat “context and background,” one of the variables held to be relevant to replication (Schmidt, 2009).”
BackgroundInfluentialAndresen, F. J., Monteith, L. L., Kugler, J., Cruz, R. A., & Blais, R. K. (2019) · Journal of Clinical Psychology · doi:10.1002/jclp.22773 · Cited by 90
“Rather than investigate the validity of the original data by matching the experimental methodology, conceptual replication seeks to test the rigor of the overarching construct or underlying hypothesis (Schmidt, 2009).”
BackgroundMostafa, A. M. S., & Bottomley, P. A. (2018) · Journal of Business Ethics · doi:10.1007/s10551-018-3964-5 · Cited by 74
“This constructive replication helps add confidence regarding the validity of the findings (Lykken, 1968; Schmidt, 2009).”
BackgroundSolutions to the Credibility Crisis in Management Science
Byington, E. K., & Felps, W. (2016) · Academy of Management Learning and Education · doi:10.5465/amle.2015.0035 · Cited by 77
Replication studies in international business
Dau, L. A., Santangelo, G. D., & van Witteloostuijn, A. C. (2021) · Journal of International Business Studies · doi:10.1057/s41267-021-00471-w · Cited by 79
“From the perspective of the Popperian philosophy of science, neither reproduction nor re-analysis of data can be seen as ‘real’ replication, as the underlying data – and hence the analyzed observations – are identical (Schmidt, 2009).”
BackgroundInfluentialOn the Method of Evolutionary Psychology and Its Applicability to Consumer Research
Saad, G. (2017) · Journal of Marketing Research · doi:10.1509/jmr.14.0645 · Cited by 77
“By definition, the building of nomological networks of cumulative evidence, as practiced by evolutionary behavioral scientists, engenders direct and conceptual replications (for a detailed discussion of various forms of replication, see Schmidt 2009).”
BackgroundReplication and Reproducibility in Cross-Cultural Psychology
Milfont, T. L., & Klein, R. A. (2018) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117744892 · Cited by 73
“…(lack of internal validity), (c) to encourage transparent reporting of results and potentially deter fraud, (d) to generalize results to a larger or to a different population, and (e) to test the underlying hypothesis of the earlier experiment (Schmidt, 2009; see also Lykken, 1968; Roediger, 2012).”
BackgroundResultInfluentialToste, J. R., Capin, P., Williams, K. J., Cho, E., & Vaughn, S. R. (2018) · Journal of Learning Disabilities · doi:10.1177/0022219418775114 · Cited by 81
“Conceptual replications vary in one or more features from the original study (Schmidt, 2009), they support the accumulation of scientific knowledge (G.”
BackgroundStatistical analyses for studying replication: Meta-analytic perspectives.
Hedges, L. V., & Schauer, J. M. (2019) · Psychological Methods · doi:10.1037/met0000189 · Cited by 69
“This is not to say that there is no literature on replication; however, much of it (e.g., Lykken, 1968) focuses on the definition and functions of replication, not on the analysis of replications (see Schmidt, 2009).”
BackgroundLeichtmann, B., Nitsch, V., & Mara, M. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.838116 · Cited by 66
“Replication has thus the functions to control for sampling error, to control for artifacts, to control for fraud, to generalize results to different populations and situations, and to verify the underlying hypothesis (Schmidt, 2009).”
BackgroundThe effect of priming learning vs. performance goals on a complex task
Chen, X., & Latham, G. P. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2014.06.004 · Cited by 89
“This is because the phenomenon has been shown not to hinge on a particular set of operations, but to generalize to a larger area of application (Schmidt, 2009; Stroebe & Strack, 2014).”
BackgroundLermer, E., Streicher, B., Sachs, R., Raue, M., & Frey, D. (2015) · Risk Analysis · doi:10.1111/risa.12445 · Cited by 78
“Moreover, conceptual replications are of theoretical relevance because the replications’ confirmatory power increases with every variation from the previous study.(38) Our three studies followed these recommendations in that we used the same manipulation in all three studies, but changed both risk perspective (Studies…”
MethodHensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 78
Wilkie, S., & Clouston, L. (2015) · Urban forestry & urban greening · doi:10.1016/j.ufug.2015.03.002 · Cited by 76
Creative destruction in science
Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., et al. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.07.002 · Cited by 66
“Thus, while scholars continue to generate new theory at an accelerated pace, their propositions typically enjoy preliminary rather than definitive support, and are rarely subjected to attempts at direct replication (Schmidt, 2009; Simons, 2014) or placed in competition against adjacent (and sometimes contradictory) th…”
BackgroundMethodInfluentialHuang, Y., & Snedeker, J. (2020) · Cognition · doi:10.1016/j.cognition.2020.104251 · Cited by 67
Rawson, K. A., Vaughn, K. E., & Carpenter, S. K. (2014) · Memory & Cognition · doi:10.3758/s13421-014-0477-z · Cited by 77
“Given these promising initial findings and the likely possibility that statistical power issues could have obscured the emergence of effects in Experiment 1, an important goal of Experiment 2 was to replicate these key outcomes with a larger sample (for recent emphasis on the importance of replicating novel findings,…”
BackgroundA decade of replication study in education? A mapping review (2011–2020)
Perry, T., Morris, R., & Lea, R. G. (2022) · Educational Research and Evaluation · doi:10.1080/13803611.2021.2022315 · Cited by 70
“…the Discussion section we return to the question of whether replication is more prevalent than suggested by our data due to studies not using the term, or preferring alternatives such as “follow-on” (Schmidt, 2009), or including replication elements within an extension study or as a scale-up study.”
MethodInfluentialNovelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61
“Also, for statistician R. A. Fisher, experimentally established facts are those that rarely fail to replicate (1926, 504). purportedly causally responsible for the effect (Campbell and Jackson 1979; Schmidt 2009).8 Using this concept, we can state a replicability principle.”
BackgroundMore Than One Replication Study Is Needed for Unambiguous Tests of Replication
Hedges, L. V., & Schauer, J. M. (2019) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998619852953 · Cited by 74
Rate and success of study replication in ecology and evolution
Kelly, C. D. (2019) · PeerJ · doi:10.7717/peerj.7654 · Cited by 71
“The ‘‘replication crisis’’ of the past decade has not only stimulated considerable reflection and discussion within scientific disciplines (Parker et al., 2016; Brandt et al., 2014; Kelly, 2006; Schmidt, 2009; Zwaan et al., 2017) but it has also produced large-scale, systematic efforts to”
BackgroundInfluentialGüttinger, S. (2020) · European Journal for Philosophy of Science · doi:10.1007/s13194-019-0269-1 · Cited by 84
“The debate about the replication crisis is complicated by the fact that there is still no firm consensus on how exactly to define the terms used in the debate (Schmidt 2009; Goodman et al. 2016; Nosek and Errington 2017).”
BackgroundA Critical Review of Sexual Violence Prevention on College Campuses
Newlands, R. T. (2016) · Acta Psychopathologica · doi:10.4172/2469-6676.100040 · Cited by 59
Multi-Study Packages in Organizational Science Research
Hochwarter, W. A., Ferris, G. R., & Hanes, T. J. (2011) · Research methodology in strategy and management · doi:10.1108/s1479-8387(2011)0000006005 · Cited by 61
“analytic strategy, and countless more (Evanschitzky & Armstrong, 2010; Schmidt, 2009; Tsang & Kwan, 1999).”
BackgroundResultThe Effect of Self–Awareness on the Identification of Goal–Related Obstacles
Kreibich, A., Hennecke, M., & Brandstätter, V. (2020) · European Journal of Personality · doi:10.1002/per.2234 · Cited by 78
“An important component of cumulative science is replication (Asendorpf et al., 2013; Brandt et al., 2014; Jasny, Chin, Chong, & Vignieri, 2011; Nosek, 2013; Schmidt, 2009).”
BackgroundCriminalization of scientific misconduct
Bülow, W., & Helgesson, G. (2018) · Medicine Health Care and Philosophy · doi:10.1007/s11019-018-9865-7 · Cited by 72
“We suggest that this aspect is relevant to the ongoing debate on the difficulties to replicate results (Schmidt 2009; Francis 2012; Open Science Collaboration 2015; Anderson et al.”
BackgroundOpen Science Promotes Diverse, Just, and Sustainable Research and Educational Outcomes
Grahe, J. E., Cuccolo, K., Leighton, D. C., & Cramblet Alvarez, L. D. (2020) · Psychology Learning & Teaching · doi:10.1177/1475725719869164 · Cited by 56
Chen, X., Affourtit, J., Ryskin, R., Regev, T. I., Norman-Haignere, S., Jouravlev, O., Malik-Moraleda, S., Kean, H., Varley, R., & Fedorenko, E. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhad087 · Cited by 57
“This task thus serves as a conceptual replication (Schmidt 2009).”
BackgroundKlein, P., Küchemann, S., Brückner, S., Zlatkin‐Troitschanskaia, O., & Kühn, J. (2019) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.15.020116 · Cited by 59
From ‘publish or perish’ to ‘publish for purpose’
Elbanna, S., & Child, J. (2023) · European Management Review · doi:10.1111/emre.12618 · Cited by 77
Do I Fit In? Perceptions of Organizational Fit as a Resource in the Workplace Stress Process
Mackey, J. D., Perrewé, P. L., & McAllister, C. P. (2016) · Group & Organization Management · doi:10.1177/1059601115625155 · Cited by 67
“Constructive replications can utilize changes in sampling procedures, rating sources, measures, and/or subjects to provide stronger evidence for the validity of results than is possible with single-study designs (Schmidt, 2009; Singh, Ang, & Leong, 2003; Tsang & Kwan, 1999) that can make substantive and meaningful con…”
BackgroundThe Role of Replication Studies in Theory Building
Irvine, E. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620970558 · Cited by 64
“As Fabrigar and Wegener (2016) pointed out (see also S. Schmidt, 2009; Stroebe & Strack, 2014), the original materials or operationalizations used in a study may be developed for a specific population or context, so they may not generate the same psychological phenomenon when used with other…”
BackgroundRelationships between sleep duration and adolescent depression: a conceptual replication
Berger, A., Wahlstrom, K., & Widome, R. (2019) · Sleep Health · doi:10.1016/j.sleh.2018.12.003 · Cited by 53
“A broader conception of replication not only allows for control of sampling error and/or fraud but also allows one to generalize from results and to verify the underlying hypotheses of previous research.(25) Although START (Study 2) and Study 1 differ in the selection of participating schools, the students recruited,…”
ResultOn the merits of student‐recruited sampling: Opinions a decade in the making
Hochwarter, W. A. (2013) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12043 · Cited by 57
The moderating role of cultural factors in the adoption of mobile banking in Brazil
Goularte, A. D. C., & Zilber, S. N. (2018) · International Journal of Innovation Science · doi:10.1108/ijis-11-2017-0119 · Cited by 61
Science is not a signal detection problem
Wilson, B. M., Harris, C. R., & Wixted, J. T. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914237117 · Cited by 56
Better together: Advancing life course research through multi-cohort analytic approaches
O’Connor, M., Spry, E., Patton, G. C., Moreno‐Betancur, M., Arnup, S. J., Downes, M., Goldfeld, S. R., Burgner, D. P., & Olsson, C. A. (2022) · Advances in Life Course Research · doi:10.1016/j.alcr.2022.100499 · Cited by 58
Mathis, F., Vaniea, K. E., & Khamis, M. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445478 · Cited by 62
“Human test subjects consciously or sub-consciously remember previous experiences that can impact their thoughts, behaviour, and performance; thus, experiments can result in different results due to the non-uniformity of nature [35, 90].”
BackgroundThe Role of Replication in Engineering Education Research
Benson, L., & Borrego, M. J. (2015) · Journal of Engineering Education · doi:10.1002/jee.20082 · Cited by 54
Can the behavioral sciences self-correct? A social epistemic study
Romero, F. (2016) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2016.10.002 · Cited by 57
The standardization–generalization dilemma: a way out
van der Staay, F. J., Arndt, S. S., & Nordquist, R. E. (2010) · Genes Brain & Behavior · doi:10.1111/j.1601-183x.2010.00628.x · Cited by 61
“Close, exact or direct replications are based on a narrow notion of replication, namely the repetition of an experimental procedure (Schmidt 2009).”
BackgroundInfluentialAn Experimental Field Study on the Effects of Coaching: The Mediating Role of Psychological Capital
Fontes, A., & Russo, S. D. (2020) · Applied Psychology · doi:10.1111/apps.12260 · Cited by 67
A Systematic Investigation of Replications in Computing Education Research
Hao, Q., Smith IV, D. H., Iriumi, N., Tsikerdekis, M., & Ko, A. J. (2019) · ACM Transactions on Computing Education · doi:10.1145/3345328 · Cited by 51
“To be consistent with studies investigating replication in other fields [e.g., 9, 10], we adopted the replication classification of Schmidt [7] in this study.”
MethodResultInfluentialCan We Help Just by Good Intentions? A Meta-Analysis of Experiments on Distant Intention Effects
Schmidt, S. (2012) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2011.0321 · Cited by 51
Self-citationLanivich, S. E., Brees, J. R., Hochwarter, W. A., & Ferris, G. R. (2010) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2010.05.004 · Cited by 59
Davis-Kean, P. E., & Ellis, A. (2019) · Infant Behavior and Development · doi:10.1016/j.infbeh.2019.101339 · Cited by 61
Lee, Y., Zhuang, Y., Joo, M., & Bae, T. J. (2019) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2019.e00144 · Cited by 65
Metadata
- Authors
- Schmidt, S.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Schmidt, Stefan
- ORCID
- Schmidt, Stefan
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 90-100
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2021458971
- Cited by
- 1,125
- References
- 55
- Citations by year
- 2026: 42 · 2025: 95 · 2024: 88 · 2023: 85 · 2022: 75 · 2021: 138 · 2020: 94 · 2019: 105 · 2018: 93 · 2017: 74 · 2016: 84 · 2015: 52 · 2014: 35 · 2013: 26 · 2012: 23
- FWCI
- 16.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of General Psychology (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, social sciences, fact-checking, publication practices, research methodology, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.