Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Replication is one of the most important tools for the verification of facts within the empirical sciences. A detailed examination of the notion of replication reveals that there are many different meanings to this concept and the relevant procedures, but hardly any systematic literature. This paper analyzes the concept of replication from a theoretical point of view. It demonstrates that the theoretical demands are scarcely met in everyday work within the social sciences. Some demands are just not feasible, whereas others are constricted by restrictions relating to publication. A new classification scheme based on a functional approach that distinguishes between different types of replication is proposed. Next, it will be argued that replication addresses the important connection between existing and new knowledge. To do so it has to be applied explicitly and systematically. The paper ends with a description of procedures how this could be done and a set of recommendations how to handle the concept of replication in the future to exploit its potential to the full.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“RRs provide a mechanism for publishing replication research”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“which are rarely published in the standard model 13,14 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule [48].”
From The natural selection of bad science (2016) · cites this paper as [48]
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,706
Design and Analysis: A Researcher's Handbook
Beaton, A. E., & Keppel, G. (1975) · American Educational Research Journal · doi:10.2307/1162588 · Cited by 5,822
Scientific Explanation and the Causal Structure of the World
Salmon, W. C. (1985) · Princeton University Press eBooks · doi:10.1515/9780691221489 · Cited by 2,459
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.2307/1869945 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.1086/ahr/92.3.658 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Traynham, J. G., Shapin, S., & Schaffer, S. (1987) · Journal of Interdisciplinary History · doi:10.2307/204299 · Cited by 2,045
Design and Analysis: A Researcher's Handbook
Aaker, D. A., & Keppel, G. (1976) · Journal of Marketing Research · doi:10.2307/3150757 · Cited by 2,011
Research works printed but not linked: 12
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- Beloff J., 1985, The repeatability problem in parapsychology, 1
- Collins H. M., 1985, Changing order
- CookT. D. (1990). Rersearch methodology: Strengthening causal interpretations of nonexperimental data. In SechrestL., PerrinE. & BunkerJ., US Department of health and human services (pp. 9–30). Public Health Service, Agency for Health Care Policy and Research.
- Edge H., 1985, The repeatability problem in parapsychology, 53
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 21
- Rao K. R., 1981, Journal of Parapsychology, 45, 311
- Rosenthal R., 1991, Replication research in the social sciences, 1
- Sargent C. L., 1981, European Journal of Parapsychology, 3, 423
- Schacter D. L., 2000, Wir sind Erinnerung. Gedächtnis und Persönlichkeit
- Walach H., 2000, Neuroendocrinology Letters, 21, 221
- Weizsäcker E. U. v., 1986, Offene Systeme I: Beiträge zur Zeitstruktur von Information, Entropie und Evolution, 82
Cited by
Items 201–300 of 1,001
The science of leadership: A journey from survey research to field experimentation
Eden, D. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101472 · Cited by 61
Leading empowered teams: An examination of the role of external team leaders and team coaches
Rapp, T. L., Gilson, L. L., Mathieu, J. E., & Ruddy, T. M. (2015) · The Leadership Quarterly · doi:10.1016/j.leaqua.2015.08.005 · Cited by 71
Kelly, A. B., Mason, W. A., Chmelka, M. B., Herrenkohl, T. I., Kim, M. J., Patton, G. C., Hemphill, S. A., Toumbourou, J. W., & Catalano, R. F. (2016) · Journal of Youth and Adolescence · doi:10.1007/s10964-016-0433-2 · Cited by 64
“Replication is a keystone of psychological research and it is relatively rare in this specific area of research (Schmidt 2009).”
BackgroundTesting the Efficacy of a Tier 2 Mathematics Intervention: A Conceptual Replication Study
Doabler, C. T., Clarke, B., Kosty, D. B., Kurtz‐Nelson, E., Fien, H., Smolkowski, K., & Baker, S. K. (2016) · Exceptional Children · doi:10.1177/0014402916660084 · Cited by 56
“Replication is a fundamental principle of scientific research (Flay et al., 2005; Gottfredson et al., 2015; Schmidt, 2009; Valentine et al., 2011).”
BackgroundInfluentialDaugherty, T., Hoffman, E. L., Kennedy, K. E., & Nolan, M. (2018) · European Journal of Marketing · doi:10.1108/ejm-10-2017-0657 · Cited by 64
Economic stress and well-being: Does population health context matter?
Probst, T. M., Sinclair, R. R., Sears, L. E., Gailey, N. J., Black, K. J., & Cheung, J. H. (2018) · Journal of Applied Psychology · doi:10.1037/apl0000309 · Cited by 58
A Concise Set of Core Recommendations to Improve the Dependability of Psychological Research
Lishner, D. A. (2015) · Review of General Psychology · doi:10.1037/gpr0000028 · Cited by 77
“Indeed, replication is widely advocated in both positivepsychology and negative-psychology literatures and the newer research dependability literature (Asendorpf et al., 2013; Bakker, Van Dijk, & Wicherts, 2012; Braver, Thoemmes, & Rosenthal, 2014; Cohen, 1994; Cumming, 2014; Frank & Saxe, 2012; Ioannidis, 2012; Kerr,…”
BackgroundInfluentialOpportunistic biases: Their origins, effects, and an integrated solution.
DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015) · American Psychologist · doi:10.1037/a0039191 · Cited by 51
“…the importance of replication to ensure that observed findings are not the result of random error (e.g., Shaver & Norton, 1980; Cohen, 1994; Schmidt, 2009), and some have specifically discussed its importance for protecting the field from experimenter biases (e.g., Valentine et al., 2011).”
BackgroundLotz, S. (2015) · PLoS ONE · doi:10.1371/journal.pone.0131562 · Cited by 60
A Replication by Any Other Name: A Systematic Review of Replicative Intervention Studies
Cook, B. G., Collins, L. W., Cook, S. C., & Cook, L. (2016) · Remedial and Special Education · doi:10.1177/0741932516637198 · Cited by 56
“(p. 90) Despite a common understanding among researchers as to the general meaning and importance of replication, definitions and classifications of replication are varied (Schmidt, 2009).”
BackgroundInfluentialReplication in computing education research: researcher attitudes and experiences
Ahadi, A., Hellas, A., Ihantola, P., Korhonen, A., & Petersen, A. (2016) · Proceedings of the 16th Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999554 · Cited by 48
“Schmidt presented a comprehensive summary of the issues that contributed to varying definitions of reproducibility in 2009 [42].”
BackgroundOn the bilingualism effect in task switching
Branzi, F. M., Calabria, M., Gade, M., Fuentes, L. J., & Costa, A. (2016) · Bilingualism Language and Cognition · doi:10.1017/s136672891600119x · Cited by 51
“Recent evidence suggests that inhibition that is measured through the n-2 repetition cost cannot be generalised 1 It is important to distinguish between direct (or exact) and conceptual replications in experimental science (see Schmidt, 2009).”
BackgroundFoster, N. L., Rawson, K. A., & Dunlosky, J. T. (2017) · Learning and Instruction · doi:10.1016/j.learninstruc.2017.10.002 · Cited by 58
Pornography and Sexual Behavior: Do Sexual Attitudes Mediate or Confound?
Wright, P. J. (2018) · Communication Research · doi:10.1177/0093650218796363 · Cited by 53
“By including two measures of pornography consumption, two measures of sexual attitudes, two measures of sexual behavior, and multiple samples, the generalizability and confidence that can be placed in the results is enhanced (Schmidt, 2009; Slater & Gleason, 2012).”
BackgroundTime orientation mediates the link between hunger and hedonic choices across domains
Otterbring, T. (2019) · Food Research International · doi:10.1016/j.foodres.2019.02.032 · Cited by 52
Chhin, C. S., Taylor, K. A., & Wei, W. S. (2018) · Educational Researcher · doi:10.3102/0013189x18788047 · Cited by 54
“Using the definitions of direct and conceptual replication described by Schmidt (2009) and Coyne DECEmbER 2018 597 et al. (2016), we examined Goal Three and Goal Four grant applications funded under the Education and Special Education Research Grants programs to address the following research…”
BackgroundMethodInfluentialLeichtmann, B., Hinterreiter, A., Humer, C., Streit, M., & Mara, M. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2221605 · Cited by 58
“In the subsequent reform movement, it was emphasized that replications test the robustness of effects, verify ideas and theories, control for sampling errors, control for fraud, and control for artifacts (Schmidt, 2009).”
Kong, P.-R., & Han, K.-T. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.172197 · Cited by 51
Brenner, L. A., Forster, J. E., Hoffberg, A. S., Matarazzo, B. B., Hostetter, T. A., Signoracci, G., & Simpson, G. K. (2018) · Journal of Head Trauma Rehabilitation · doi:10.1097/htr.0000000000000351 · Cited by 39
The moral behavior of ethics professors: A replication-extension in German-speaking countries
Schönegger, P., & Wagner, J. (2019) · Philosophical Psychology · doi:10.1080/09515089.2019.1587912 · Cited by 50
“crisis in social science fields (Schmidt, 2009) like moral psychology and adjacent”
BackgroundMethodBrand love: Corroborating evidence across four continents
Sajtos, L., Cao, J. T., Espinosa, J. A., Phau, I., Rossi, P., Sung, B., & Voyer, B. G. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.02.040 · Cited by 56
Gordon, M., Viganola, D., Bishop, M., Chen, Y., Dreber, A., Goldfedder, B., Holzmeister, F., Johannesson, M., Liu, Y., Twardy, C. R., et al. (2020) · Royal Society Open Science · doi:10.1098/rsos.200566 · Cited by 54
“Moreover, replications allow us to test if results generalize to a different or larger population and help to verify the underlying theory [2,4,5] and its scope [6].”
BackgroundInfluentialProposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science
Murphy, J., Mesquida, C., Caldwell, A. R., Earp, B. D., & Warne, J. P. (2022) · Sports Medicine · doi:10.1007/s40279-022-01749-1 · Cited by 46
“Although replication is most valuable when the existing understanding for a theory is weakest [5], it can also update boundaries of a claim to further develop theories [7, 8].”
BackgroundReassessing the relationship between general intelligence and self-control in childhood
Meldrum, R. C., Petkovsek, M. A., Boutwell, B. B., & Young, J. T. N. (2016) · Intelligence · doi:10.1016/j.intell.2016.10.005 · Cited by 48
“…of the need to replicate prior work in the psychological sciences in recent years (Open Science Collaboration, 2015; Ritchie, Wiseman, & French, 2012; Schmidt, 2009), the opportunity to further examine extant findings in a more rigorous manner with new data is a critically important enterprise.”
BackgroundA Causal Replication Framework for Designing and Assessing Replication Efforts
Steiner, P. M., Wong, V. C., & Anglin, K. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000385 · Cited by 52
“The Causal Replication Framework begins with the premise that replication is for “establishing a fact, truth, or piece of knowledge” (Schmidt, 2009).”
BackgroundMethodRyan, J. C., & Tipu, S. A. A. (2021) · Research Policy · doi:10.1016/j.respol.2021.104408 · Cited by 49
Radical Solutions and Open Science: An Open Approach to Boost Higher Education
Burgos, D. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3 · Cited by 51
Misconduct, Marginality and Editorial Practices in Management, Business and Economics Journals
Karabağ, S. F., & Berggren, C. (2016) · PLoS ONE · doi:10.1371/journal.pone.0159492 · Cited by 45
“This second survey asked specific questions on a number of practices discussed in the broader science community: the application of screening software to detect plagiarism [14]; the use of replications [27, 28]; measures to specify co-author contribution [29, 30]; methods to assess and publicly reward reviewers [31];…”
BackgroundChapman, C. M., Masser, B., & Louis, W. R. (2018) · Nonprofit and Voluntary Sector Quarterly · doi:10.1177/0899764018805196 · Cited by 46
“To increase confidence in the results obtained (see Asendorpf et al., 2013; Brandt et al., 2014; Schmidt, 2009), analyses were conducted on survey responses from peer-topeer fundraisers at two separate timepoints.”
ResultReproducibility of amygdala activation in facial emotion processing at 7T
Geissberger, N., Tik, M., Sladky, R., Woletz, M., Schuler, A., Willinger, D., & Windischberger, C. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116585 · Cited by 46
“…as an umbrella term and would like to determine further terminology in this manuscript: When varying parameters (i.e. task instructions or acquisition protocol) and comparing group results, we are using the term replicability, in the sense of systematic replication as proposed by Schmidt (2009).”
Stracke, C. M. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3_2 · Cited by 54
“The situation in (social) science has slowly enhanced since the rhetoric questioning of replications by Schmidt (2009).”
BackgroundWard, L., Nault, D., Cramer, H., & Moonaz, S. H. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-054585 · Cited by 42
An Investigation of the Prevalence of Replication Research in Human Factors
Jones, K. S., Derby, P. L., & Schmidlin, E. A. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810384394 · Cited by 41
“, psychology) suggest that replications are also uncommon in that domain (Bozarth & Roberts, 1972; Mahoney, 1985; Schmidt, 2009; Sterling, 1959; cf. Neuliep & Crandall, 1993a).”
BackgroundHurtado‐Parrado, C., & López‐López, W. (2015) · Integrative Psychological and Behavioral Science · doi:10.1007/s12124-014-9290-2 · Cited by 44
Replication Studies in Mathematics Education: What Kind of Questions Would Be Productive to Explore?
Aguilar, M. S. (2020) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-020-10069-7 · Cited by 40
Turner, M. J., Miller, A. J., Youngs, H., Barber, N., Brick, N., Chadha, N. J., Chandler, C., Coyle, M., Didymus, F. F., Evans, A. L., et al. (2022) · Journal of Sports Sciences · doi:10.1080/02640414.2022.2042124 · Cited by 51
“Schmidt (2009) posited that in order to demonstrate the same result as study 1 with a different sample (i.e., student-athletes), a modified procedure is required.”
ResultAcute alcohol response phenotype in heavy social drinkers is robust and reproducible
Roche, D. J. O., Palmeri, M. D., & King, A. C. (2014) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.12280 · Cited by 39
“As observed in other fields (Ioannidis, 2005; Schmidt, 2009), most alcohol challenge studies have focused on conceptual From the Department of Psychiatry and Biobehavioral Sciences (DJOR), University of California, Los Angeles, California; and Department of Psychiatry and Behavioral Sciences (MDP, ACK), University of…”
BackgroundSingle-case experimental research designs.
Kazdin, A. E. (2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-029 · Cited by 39
The role of replication in psychological science
Fletcher, S. C. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-020-00329-2 · Cited by 49
“I describe these in more detail in Section 2, drawing on the functional classification of replication efforts by Schmidt (2009, 2017).3 Besides setting the terms of the debate more precisely, this also makes explicit the variety of types of underdetermination that replication can address in…”
BackgroundMethodInfluentialExplicating Exact versus Conceptual Replication
Hudson, R. G. S. (2021) · Erkenntnis · doi:10.1007/s10670-021-00464-z · Cited by 48
“Apart from exact, experimental and conceptual replications, there are a variety of other notions of replication one might consider (Schmidt, 2009, 91, has an extensive list).”
BackgroundInfluentialEconomic stress, emotional contagion and safety outcomes: A cross-country study
Петитта, Л., Probst, T. M., Ghezzi, V., & Barbaranelli, C. (2020) · Work · doi:10.3233/wor-203182 · Cited by 43
“In doing so, we sought to enhance 53 the conceptual replication [6, 7] of our hypothe54 ses, and increase the generalizability of our findings.”
Parallel age‐related cognitive effects in autism: A cross‐sectional replication study
Torenvliet, C., Groenman, A. P., Radhoe, T. A., van Rentergem, J. A. A., van der Putten, W., & Geurts, H. M. (2021) · Autism Research · doi:10.1002/aur.2650 · Cited by 44
“The current study is a direct replication (for definition see: Schmidt, 2009) of a previous study (Lever et al., 2015; Lever & Geurts, 2016).”
MethodOtterbring, T., & Folwarczny, M. (2023) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2023.103612 · Cited by 40
Replication Unreliability in Psychology: Elusive Phenomena or “Elusive” Statistical Power?
Tressoldi, P. E. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00218 · Cited by 41
“INTRODUCTION ARE THERE ELUSIVE PHENOMENA OR IS THERE AN “ELUSIVE” POWER TO DETECT THEM? When may a phenomenon be considered to be real or very probable, following the rules of current scientific methodology? Among the many requirements, there is a substantial consensus that replication is one of the more fundamental (…”
BackgroundZamary, A., Rawson, K. A., & Dunlosky, J. T. (2016) · Learning and Instruction · doi:10.1016/j.learninstruc.2016.08.002 · Cited by 41
Creative self-efficacy and teacher ratings of student creativity in Chinese elementary classrooms
Hartley, K. A., Plucker, J. A., & Long, H. (2016) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2016.10.001 · Cited by 40
Framing Unauthorized Immigrants: The Effects of Labels on Evaluations
Ommundsen, R., Larsen, K. S., van der Veer, C. G., & Eilertsen, D. (2014) · Psychological Reports · doi:10.2466/17.pr0.114k20w0 · Cited by 49
“By increasing our knowledge of the effect of framing labels, the present study answers a recent call for replication studies to evaluate the robustness and generality of research findings (see e.g. Schmidt, 2009; Koole & Lakens, 2012).”
MethodKleiman, T., Sher, N., Elster, A., & Mayo, R. (2015) · Cognition · doi:10.1016/j.cognition.2015.06.001 · Cited by 43
“Together, our studies have a strong confirmatory power, beyond a specific procedure or measure (Schmidt, 2009; Stroebe & Strack, 2014).”
BackgroundReplication is relevant to qualitative research
Makel, M. C., Meyer, M. S., Simonsen, M. A., Roberts, A. M., & Plucker, J. A. (2022) · Educational Research and Evaluation · doi:10.1080/13803611.2021.2022310 · Cited by 79
“For example, the purpose of replication is often framed as verifying results or assessing generalisability (Schmidt, 2009).”
BackgroundWhat Crowdsourcing Can Offer to Cross-Cultural Psychological Science
Cuccolo, K., Irgens, M. S., Zlokovich, M. S., Grahe, J. E., & Edlund, J. E. (2020) · Cross-Cultural Research · doi:10.1177/1069397120950628 · Cited by 39
“Being able to easily and accurately conduct replications or to further investigate particular effects helps flush out false positives, false negatives, and potentially even fraud (Schmidt, 2009).”
BackgroundAn editorial on replication studies and scaling up efforts
Spector, J. M., Johnson, T. E., & Young, P. A. (2014) · Educational Technology Research and Development · doi:10.1007/s11423-014-9364-3 · Cited by 41
Estimating the Reproducibility of Experimental PhilosophyPreprint
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/sxdah · Cited by 38
Never let an academic crisis go to waste: Leadership Studies in the wake of journal retractions
Spoelstra, S., Butler, N., & Delaney, H. (2016) · Leadership · doi:10.1177/1742715016658215 · Cited by 48
Investigating and explaining the effects of successive relearning on long-term retention.
Rawson, K. A., Vaughn, K. E., Walsh, M., & Dunlosky, J. (2018) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000146 · Cited by 36
“Experiment 2 Given recent emphasis in the field on the importance of replication (e.g., Maner, 2014; Pashler & Harris, 2012; Schmidt, 2009; Simons, 2014), one purpose of Experiment 2 was to replicate the key outcomes of Experiment 1.”
BackgroundBareket, O., Shnabel, N., Abeles, D., Gervais, S. J., & Yuval‐Greenberg, S. (2018) · Sex Roles · doi:10.1007/s11199-018-0983-8 · Cited by 47
“In line with the importance of conducting conceptual replications (i.e., testing the same hypothesis with different methods or materials; see Schmidt 2009), we tested our hypothesis using two photograph sets of women, with either standard or realistic natures.”
MethodHow has forking changed in the last 20 years?: a study of hard forks on GitHub
Zhou, S., Vasilescu, B., & Kästner, C. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380412 · Cited by 36
“, regarding motivations and outcomes of hard forks) and can be considered a conceptual replication [24, 43].”
BackgroundMethodSaving Science Through Replication Studies
Edlund, J. E., Cuccolo, K., Irgens, M. S., Wagge, J., & Zlokovich, M. S. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620984385 · Cited by 43
“Replications by other labs allow for increased credibility, generalizability of results, room for identification of moderators, and trust that the finding was not a Type I error (Schmidt, 2009; Simons, 2014).”
BackgroundDimensionality of the Chinese CES-D: Is It Stable across Gender, Time, and Samples?
Dou, D., Shek, D. T. L., Zhu, X., & Zhao, L. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182211818 · Cited by 38
“However, despite the importance of replication, there are very few replication studies, particularly in social sciences [24].”
BackgroundTransparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
“Journals have been criticized for traditional publication models that focus on novelty rather than reproducibility, results rather than methods, and narrative rather than data and analysis (Fidler & Wilcox, 2018; Neuliep & Crandall, 1990; Schmidt, 2009).”
Dudschig, C., Maienborn, C., & Kaup, B. (2016) · Brain and Cognition · doi:10.1016/j.bandc.2016.01.001 · Cited by 42
“…we used the N400-complex (Kutas & Hillyard, 1980) as our main measurement and implemented a conceptual replication of the original study with minor – but potentially important - changes in the experimental setup (for discussion on the value of such replication studies see: Schmidt, 2009).”
MethodPower, S. A., & Vélez, G. (2020) · Review of General Psychology · doi:10.1177/1089268020915841 · Cited by 39
“…issues concerning status and reputation of experiments and (failed) replications of them; debates about the importance of direct or conceptual replications; and disagreements over deciding when a study has or has not been replicated (Rodgers & Shrout, 2017; Schmidt, 2009; Zwaan et al., 2018).”
BackgroundInfluentialCritical Essay: Building new management theories on sound data? The case of neuroscience
Lindebaum, D. (2015) · Human Relations · doi:10.1177/0018726715599831 · Cited by 42
Do Neuroscience Journals Accept Replications? A Survey of Literature
Yeung, A. W. K. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00468 · Cited by 37
“Whereas direct replication provides an insight into the consistency of the presence or absence of an effect, conceptual replication can take a step further to generalize results to a larger or different population, verify the hypotheses of previous works and even evaluate data from new perspectives by improved methods…”
MethodResultPedersen, M. J., & Stritch, J. M. (2018) · Public Administration Review · doi:10.1111/puar.12910 · Cited by 38
Grünbaum, T., & Christensen, M. S. (2020) · Neuroscience of Consciousness · doi:10.1093/nc/niaa019 · Cited by 46
“…the construct validation of experimental procedures, we will understand construct validation of experimental procedures as conceptual replication: a test of a hypothesis with a given procedure or an effect demonstrated using a given procedure repeated using different procedures (Schmidt 2009, 91).”
MethodCarolan, B. V. (2015) · British Journal of Sociology of Education · doi:10.1080/01425692.2015.1013085 · Cited by 45
Replication and Open Science in Applied Linguistics ResearchPreprint
McManus, K. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/bqr9w · Cited by 36
Carter, E. C., & McCullough, M. E. (2013) · BMC Psychology · doi:10.1186/2050-7283-1-22 · Cited by 39
“aFollowing Schmidt (2009), we use the term conceptual replication to refer to any attempt at replicating a previous test via different methods.”
MethodExamining the interactive effects of accountability, politics, and voice
Hochwarter, W. A., Ellen, B. P., & Ferris, G. R. (2014) · Career Development International · doi:10.1108/cdi-01-2014-0012 · Cited by 38
“Such an approach increases confidence that findings, if consistent across settings and respondents, are valid rather than artifactual (Schmidt, 2009).”
BackgroundWhat Values and Traits Do Parents Teach to Their Children? New Data from 54 Countries
Minkov, M., Dutt, P., Schachner, M., Jandosova, J., Khassenbekov, Y., Morales, O., Sanchez, C. J., & Mudd, B. (2018) · Comparative Sociology · doi:10.1163/15691330-12341456 · Cited by 43
No Evidence for an Object Working Memory Capacity Benefit with Extended Viewing Time
Quirk, C., Adam, K. C. S., & Vogel, E. K. (2020) · eNeuro · doi:10.1523/eneuro.0150-20.2020 · Cited by 34
“Although no replication attempt can ever be truly identical, taken together, a range of replications from near-direct to more conceptual are useful for determining potential constraints on the generalizability of a given result (Schmidt, 2009; Simons et al., 2017).”
BackgroundHeckman, S. S., Carver, J. C., Sherriff, M. S., & Al-Zubidy, A. (2021) · ACM Transactions on Computing Education · doi:10.1145/3470652 · Cited by 38
“Specifically, the rubric was focused on aspects of a research paper that are essential for others to understand and potentially replicate the experiment, as evidenced by prior work [56, 72] and supported by guidelines in educational research [12, 19, 59, 60].”
InfluentialBeyond Random Effects: When Small-Study Findings Are More Heterogeneous
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221120427 · Cited by 41
“Several researchers have classified replications as “conceptual” versus “direct” (or “close”; Hedges & Schauer, 2019; Linden & Hönekopp, 2021; Schauer & Hedges, 2020; S. Schmidt, 2009).”
BackgroundMuradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201697 · Cited by 38
The empirical march: Making science better at self-correction.
Makel, M. C. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035803 · Cited by 39
“Replication is the duplication of research procedures to corroborate previous results (Lykken, 1968; Schmidt, 2009).”
BackgroundResultLovato, N., Short, M. A., Micic, G., Hiller, R. M., & Gradisar, M. (2017) · Nature and Science of Sleep · doi:10.2147/nss.s111521 · Cited by 38
“Although prospective studies are practically more difficult to perform, replicating them, and disseminating their findings, even if no effects are found, are important issues in this field.(12,44,45) It is important to acknowledge that the use of self-”
From victimhood to peace activism: The potential role of personal loss and inclusive victim beliefs
Shnabel, N., Belhassen, Y., & Mor, S. (2017) · Group Processes & Intergroup Relations · doi:10.1177/1368430217699463 · Cited by 35
“In light of current understanding of the importance of replication studies (e.g., Schmidt, 2009), we conducted a second survey following the 2016 memorial ceremony.”
MethodBrederoo, S., Nieuwenstein, M. R., Cornelissen, F. W., & Lorist, M. M. (2018) · Cortex · doi:10.1016/j.cortex.2018.10.021 · Cited by 40
Living in a Changing World: The Change of Culture and Psychology
Cai, H., Huang, Z., & Jing, Y. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780190679743.003.0023 · Cited by 46
Assessing heterogeneity and power in replications of psychological experiments.
Schauer, J. M., & Hedges, L. V. (2020) · Psychological Bulletin · doi:10.1037/bul0000232 · Cited by 38
Understanding Replication in a Way That Is True to Science
Haig, B. D. (2021) · Review of General Psychology · doi:10.1177/10892680211046514 · Cited by 38
“In an influential article, S. Schmidt (2009) pared down an array of replication possibilities to just two, which he named direct replication and conceptual replication.”
BackgroundHow Product-Plot Integration and Cognitive Load Affect Brand Attitude: A Replication
Pantoja, F., Rossi, P., & Borges, A. A., Jr. (2015) · Journal of Advertising · doi:10.1080/00913367.2015.1085818 · Cited by 36
Bernecker, S. L., Constantino, M. J., Atkinson, L. R., Bagby, R. M., Ravitz, P., & McBride, C. (2016) · Psychotherapy · doi:10.1037/pst0000036 · Cited by 37
Tunç, D. U., & Tunç, M. N. (2023) · Meta-Psychology · doi:10.15626/mp.2021.2756 · Cited by 38
Feest, U. (2016) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2016.04.003 · Cited by 49
Richardson, A. J. (2018) · Accounting Auditing & Accountability Journal · doi:10.1108/aaaj-08-2014-1808 · Cited by 42
Improving Universal Classroom Practices Through Teacher Formative Assessment and Coaching
Dudek, C. M., Reddy, L. A., Lekwa, A. J., Hua, A. N., & Fabiano, G. A. (2018) · Assessment for Effective Intervention · doi:10.1177/1534508418772919 · Cited by 46
A Replication Agenda for Composition Studies
Raucci, J. (2021) · College Composition and Communication · doi:10.58680/ccc202131162 · Cited by 31
An Introduction to Replication Research in Gifted Education: Shiny and New Is Not the Same as Useful
Makel, M. C., & Plucker, J. A. (2015) · Gifted Child Quarterly · doi:10.1177/0016986215578747 · Cited by 32
“gifted education, replication, research methods First articulated by Lykken (1968), replication is now typically broken into two primary categories: direct and conceptual (Schmidt, 2009).”
BackgroundMethodInfluentialJiang, L., Bohle, S. L., & Roche, M. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9553-x · Cited by 43
“…research involves a l a ck o f r ep r oduc i b i l i t y t e s t s (Open Sc i e nc e Collaboration, 2015) and Breplication is one of the most important tools for the verification of facts within the empirical sciences^ (Schmidt, 2009, p. 90), we examined our model using two independent samples.”
MethodMartyniuk, U., & Štulhofer, A. (2018) · Journal of Adolescence · doi:10.1016/j.adolescence.2018.09.006 · Cited by 34
Walach, H., Horan, M., Hinterberger, T., & von Lucadou, W. (2019) · Psychology of Consciousness Theory Research and Practice · doi:10.1037/cns0000199 · Cited by 35
Egner, L. E., Klöckner, C. A., & Pellegrini‐Masini, G. (2021) · Energy and Buildings · doi:10.1016/j.enbuild.2021.111542 · Cited by 32
Replication and the search for the laws in the geographic sciences
Kedron, P., & Holler, J. (2022) · Annals of GIS · doi:10.1080/19475683.2022.2027011 · Cited by 33
“See Schmidt (2009) for a highly cited presentation of the importance of distinguishing between the different parts of a study for replication, chapter 5 of NASEM (2019) for a discussion of the fundamental role of variation in assessing replication, and Roesch and Rougier (2020) as an example of the…”
BackgroundPredicting and reasoning about replicability using structured groups
Wintle, B. C., Smith, E. T., Bush, M., Mody, F., Wilkinson, D. P., Hanea, A. M., Marcoci, A., Fraser, H. S., Hemming, V., Thorn, F. S., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221553 · Cited by 33
de Magalhães, C. V. C., da Silva, F. Q. B., Santos, R. E. S., & Suassuna, M. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.02.001 · Cited by 31
Teaching Good Research Practices: Protocol of a Research Master Course
Sarafoglou, A., Hoogeveen, S., Matzke, D., & Wagenmakers, E. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719858807 · Cited by 38
Sawanyawisuth, K., Schwartz, A., Wolpaw, T. M., & Bordage, G. (2014) · Medical Teacher · doi:10.3109/0142159x.2014.947942 · Cited by 30
“The purpose of this study was to explore whether the increased expression of clinical reasoning and uncertainties seen in American medical students who used the SNAPPS case presentation method, would generalize to medical students in other cultural and specialty settings (Schmidt 2009).”
BackgroundBurnett, H. J. (2017) · SAGE Open · doi:10.1177/2158244017730857 · Cited by 38
Exact replication: Foundation of science or game of chance?
Piper, S. K., Grittner, U., Rex, A., Riedel, N., Fischer, F. H., Nadon, R., Siegerink, B., & Dirnagl, U. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000188 · Cited by 36
Metadata
- Authors
- Schmidt, S.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Schmidt, Stefan
- ORCID
- Schmidt, Stefan
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 90-100
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2021458971
- Cited by
- 1,125
- References
- 55
- Citations by year
- 2026: 42 · 2025: 95 · 2024: 88 · 2023: 85 · 2022: 75 · 2021: 138 · 2020: 94 · 2019: 105 · 2018: 93 · 2017: 74 · 2016: 84 · 2015: 52 · 2014: 35 · 2013: 26 · 2012: 23
- FWCI
- 16.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of General Psychology (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, social sciences, fact-checking, publication practices, research methodology, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.