Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Replication is one of the most important tools for the verification of facts within the empirical sciences. A detailed examination of the notion of replication reveals that there are many different meanings to this concept and the relevant procedures, but hardly any systematic literature. This paper analyzes the concept of replication from a theoretical point of view. It demonstrates that the theoretical demands are scarcely met in everyday work within the social sciences. Some demands are just not feasible, whereas others are constricted by restrictions relating to publication. A new classification scheme based on a functional approach that distinguishes between different types of replication is proposed. Next, it will be argued that replication addresses the important connection between existing and new knowledge. To do so it has to be applied explicitly and systematically. The paper ends with a description of procedures how this could be done and a set of recommendations how to handle the concept of replication in the future to exploit its potential to the full.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“RRs provide a mechanism for publishing replication research”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“which are rarely published in the standard model 13,14 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule [48].”
From The natural selection of bad science (2016) · cites this paper as [48]
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,706
Design and Analysis: A Researcher's Handbook
Beaton, A. E., & Keppel, G. (1975) · American Educational Research Journal · doi:10.2307/1162588 · Cited by 5,822
Scientific Explanation and the Causal Structure of the World
Salmon, W. C. (1985) · Princeton University Press eBooks · doi:10.1515/9780691221489 · Cited by 2,459
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.2307/1869945 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.1086/ahr/92.3.658 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Traynham, J. G., Shapin, S., & Schaffer, S. (1987) · Journal of Interdisciplinary History · doi:10.2307/204299 · Cited by 2,045
Design and Analysis: A Researcher's Handbook
Aaker, D. A., & Keppel, G. (1976) · Journal of Marketing Research · doi:10.2307/3150757 · Cited by 2,011
Research works printed but not linked: 12
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- Beloff J., 1985, The repeatability problem in parapsychology, 1
- Collins H. M., 1985, Changing order
- CookT. D. (1990). Rersearch methodology: Strengthening causal interpretations of nonexperimental data. In SechrestL., PerrinE. & BunkerJ., US Department of health and human services (pp. 9–30). Public Health Service, Agency for Health Care Policy and Research.
- Edge H., 1985, The repeatability problem in parapsychology, 53
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 21
- Rao K. R., 1981, Journal of Parapsychology, 45, 311
- Rosenthal R., 1991, Replication research in the social sciences, 1
- Sargent C. L., 1981, European Journal of Parapsychology, 3, 423
- Schacter D. L., 2000, Wir sind Erinnerung. Gedächtnis und Persönlichkeit
- Walach H., 2000, Neuroendocrinology Letters, 21, 221
- Weizsäcker E. U. v., 1986, Offene Systeme I: Beiträge zur Zeitstruktur von Information, Entropie und Evolution, 82
Cited by
Items 301–400 of 1,001
Do pro-social students care more for the environment?
Bhattacharya, H. (2019) · International Journal of Sustainability in Higher Education · doi:10.1108/ijshe-11-2018-0223 · Cited by 31
Perspectives on scientific error
van Ravenzwaaij, D., Bakker, M., Heesen, R., Romero, F., van Dongen, N. N. N., Crüwell, S., Field, S. M., Held, L., Munafo, M. R., Pittelkow, M.-M., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.230448 · Cited by 33
“While direct replications remain valuable in many cases, meta-research should also move forward towards better understanding the phenomena in question and developing a richer theory of replication and reproducibility [33].”
BackgroundJilke, S., Petrovsky, N., Meuleman, B., & James, O. (2016) · Public Management Review · doi:10.1080/14719037.2016.1210906 · Cited by 30
Stieglitz, S., Meske, C., Roß, B., & Mirbabaie, M. (2018) · Information Systems Frontiers · doi:10.1007/s10796-018-9867-2 · Cited by 34
“…have been prominent calls for showing the replicability of results in several disciplines, from medicine (Ioannidis 2014) to the social sciences (Schmidt 2009), after several previously accepted findings were discovered to be spurious (perhaps most famously, the suggested link between MMR…”
McManus, K. (2021) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000838 · Cited by 31
“…study (“A replication study is defined as an empirical study that involves repeating the research procedure of a previous piece of work, with or without changes [Schmidt 2009]”) and a definition of L2 research (“L2 research includes any research that involves analyzing data from L2 speakers”).”
BackgroundMethodThompson, P. S., & Klotz, A. C. (2022) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2022.104170 · Cited by 39
A Duty to Describe: Better the Devil You Know Than the Devil You Don’t
Brown, S. D., Furrow, D. R., Hill, D. F., Gable, J. C., Porter, L. P., & Jacobs, W. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551749 · Cited by 34
“6) Many authors have made general recommendations for increased sharing of study details as a means of addressing problems of direct1 (Schmidt, 2009; Sidman, 1960), systematic2 (Sidman, 1960), and conceptual3 (Schmidt, 2009) replication (e.g., Asendorpf et al., 2013; Eich, 2014; Grant et al.,…”
Backgroundvan Aert, R. C. M., & van Assen, M. A. L. M. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0967-6 · Cited by 36
“Discrepancies may also be caused by findings that can only be replicated under specific conditions and that do not generalize to different settings or subjects, or that do not persist over time (Amir & Sharon, 1990; Henrich, Heine, & Norenzayan, 2010; Klein et al., 2014; S. Schmidt, 2009).”
Hensel, P. G. (2018) · European Management Journal · doi:10.1016/j.emj.2018.03.004 · Cited by 29
“Along the same lines, Schmidt (2009) differentiates between direct and conceptual replications.”
BackgroundThe Impact of Individual Differences on a Bilingual Vocabulary Approach for Latino Preschoolers
Méndez, L. I., Crais, E. R., & Kainz, K. (2018) · Journal of Speech Language and Hearing Research · doi:10.1044/2018_jslhr-l-17-0186 · Cited by 32
King, K. M., Pullmann, M. D., Lyon, A. R., Dorsey, S., & Lewis, C. C. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000417 · Cited by 39
“Variation in how data are processed and analyzed may influence the credibility of scientific conclusions and reduce the replicability of specific findings (Schmidt, 2009).”
BackgroundSiegelman, M., Blank, I., Mineroff, Z., & Fedorenko, E. G. (2017) · Neuroscience · doi:10.1016/j.neuroscience.2019.06.003 · Cited by 35
“Conceptual replications have been argued to be as important, if not more important in some cases, for establishing robust cumulative science (e.g., Schmidt, 2009). fMRI data acquisition and preprocessing Structural and functional data were collected on the wholebody 3-”
Open Science CollaborationPreprint
Nosek, B. A., Spies, J. R., Cohn, M., Bartmess, E. N., Lakens, D., Holman, D., Cohoon, J., Lewis, M., Gordon-McKeon, S., & IJzerman, H. R. (2012) · OSF Preprints (OSF Preprints) · Cited by 29
A Unified Framework to Quantify the Credibility of Scientific FindingsPreprint
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/uwmr8 · Cited by 29
Let’s Do It Again: A Call for Replications in Psi Chi Journal of Psychological Research
Edlund, J. E. (2016) · Psi Chi Journal of Psychological Research · doi:10.24839/2164-8204.jn21.1.59 · Cited by 28
Kinds of Replicability: Different Terms and Different Functions
Matarese, V. (2022) · Axiomathes · doi:10.1007/s10516-021-09610-2 · Cited by 35
Therrien, W. J., Mathews, H. M., Hirsch, S. E., & Solis, M. R. (2016) · Remedial and Special Education · doi:10.1177/0741932516646081 · Cited by 30
Volunteer Science: An Online Laboratory for Experiments in Social Psychology
Radford, J. P., Pilny, A., Reichelmann, A. V., Keegan, B., Welles, B. F., Hoye, J., Ognyanova, K., Meleis, W. M., & Lazer, D. M. J. (2016) · Social Psychology Quarterly · doi:10.1177/0190272516675866 · Cited by 39
Albarello, F., Crisp, R. J., & Rubini, M. (2017) · The Journal of Social Psychology · doi:10.1080/00224545.2017.1346581 · Cited by 32
“Given the importance of providing replications of previous findings in order to establish the robustness of gathered evidence (Funder et al., 2014; Schmidt, 2009), this contribution aimed to replicate the evidence by Albarello and Rubini (2012) that multiple and human categorization forms the most…”
BackgroundKeuning, T., van Geel, M. J. M., Visscher, A. J., & Fox, J. (2019) · Journal of Educational Measurement · doi:10.1111/jedm.12236 · Cited by 36
Brückner, S., Zlatkin‐Troitschanskaia, O., Küchemann, S., Klein, P., & Kühn, J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.02090 · Cited by 30
“To achieve a higher degree of (external) validity and generalizability, the replication of a study requires a comprehensive presentation of the control variables and can expand the original study in some aspects (Schmidt, 2009).”
BackgroundA New Account of Replication in the Experimental Life Sciences
Guttinger, S. (2019) · Philosophy of Science · doi:10.1086/703555 · Cited by 27
“…are replicated to confirm earlierfindings (Collins 1985; Schmidt 2009) and to ensure the reliability and robustness of experimental output (Soler et al. 2012).3 However, despite this theoretical consensus, in practice replication seems to be more an idea than a reality, in particular as…”
BackgroundExtrapolation of Experimental Results through Analogical Reasoning from Latent Classes
van Eersel, G. G., Koppenol‐Gonzalez, G. V., & Reiss, J. (2018) · Philosophy of Science · doi:10.1086/701956 · Cited by 28
Ideal Point Modeling of Non-cognitive Constructs: Review and Recommendations for Research
Tay, L., & Ng, V. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02423 · Cited by 28
“Psychology is also moving toward a greater emphasis on consolidating and replicating findings (Schmidt, 2009; Schooler, 2014; Maxwell et al., 2015), and we need to use methodologies appropriate for addressing issues of non-cognitive construct continua.”
MethodThe effect of cognitive load on economic decision-making: a replication attempt
Ball, S. B., Katz, B., Li, F., & Smith, A. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.03.018 · Cited by 29
Kreibig, S. D., Samson, A. C., & Gross, J. J. (2015) · Psychophysiology · doi:10.1111/psyp.12425 · Cited by 34
“Direct replication involves repetition of an experimental procedure, whereas conceptual replication involves repetition of a test of a hypothesis with different methods (Schmidt, 2009).”
MethodLikeableness and meaningfulness ratings of 555 (+487) person-descriptive words
Chandler, J. (2016) · Journal of Research in Personality · doi:10.1016/j.jrp.2016.07.005 · Cited by 34
Hakim, N., Simons, D. J., Zhao, H., & Wan, X. I. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616667613 · Cited by 32
“A direct replication attempts to reproduce the methods and procedures of the original study as precisely as possible in order to determine whether those procedures yield the same effect reliably (Schmidt, 2009).”
MethodNew developments in research methods
Ledgerwood, A. (2019) · eScholarship (California Digital Library) · Cited by 28
Hopp, C., Wentzel, D., & Rose, S. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101437 · Cited by 30
Addressing the sins of consumer psychology via the evolutionary lens
Saad, G. (2021) · Psychology and Marketing · doi:10.1002/mar.21446 · Cited by 31
Validating Automatic Concept-Based Explanations for AI-Based Digital Histopathology
Sauter, D., Lodde, G., Nensa, F., Schadendorf, D., Livingstone, E., & Kukuk, M. (2022) · Sensors · doi:10.3390/s22145346 · Cited by 26
“We apply two XAI methods to histopathology: ACE and Guided Grad-CAM as examples for explanations on model level and image level, respectively.”
MethodThe process of replication target selection in psychology: what to consider?
Pittelkow, M.-M., Field, S. M., Isager, P. M., Veer, A. V. '., Anderson, T., Cole, S. N., Dominik, T., Giner‐Sorolla, R., Gök, S., Heyman, T., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.210586 · Cited by 38
“Schmidt S. 2009 Shall we really do it again?”
BackgroundResultInfluentialMcManus, K. (2024) · Studies in Second Language Acquisition · doi:10.1017/s0272263124000652 · Cited by 38
“As a result, a replication study is a new empirical study with new data that purposefully addresses the same or similar research question and/or hypothesis as a previous study (McManus, 2023; National Academies of Sciences, Engineering, and Medicine, 2019; Schmidt, 2009).”
The role of identification and gratitude in motivating organization-serving intentions and behaviors
Madrigal, R. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.05.020 · Cited by 31
Byrd, J. L., Gailey, N. J., Probst, T. M., & Jiang, L. (2016) · Safety Science · doi:10.1016/j.ssci.2016.11.017 · Cited by 26
Rawson, K. A., & Zamary, A. (2019) · Journal of Memory and Language · doi:10.1016/j.jml.2019.01.002 · Cited by 32
Muthard, C., & Gilbertson, R. J. (2016) · Psi Chi Journal of Psychological Research · doi:10.24839/2164-8204.jn21.1.16 · Cited by 36
How open is innovation research? – An empirical analysis of data sharing among innovation scholars
Barczak, G. J., Hopp, C., Kaminski, J., Piller, F. T., & Pruschak, G. (2021) · Industry and Innovation · doi:10.1080/13662716.2021.1967727 · Cited by 33
“Replications capture innovation scholars’ attitudes towards both, direct and conceptual replications as well as on replications of their own work and the work of others (Schmidt 2009; Hopp and Hoover 2019).”
BackgroundInfluentialChen, N. C., & Petersen, F. E. (2022) · Journal of Macromarketing · doi:10.1177/02761467221111987 · Cited by 25
“Although it is encouraging that the results are consistent across the studies, these are conceptual and not exact replications (Easley, Madden, and Dunn 2000; Schmidt 2009), which limit the robustness and comparability of the results to a certain extent.”
ResultBuckley, J., Adams, L., Aribilola, I., Arshad, I., Azeem, M., Bracken, L., Breheny, C., Buckley, C., Chimello, I., Fagan, A., et al. (2021) · International Journal of Technology and Design Education · doi:10.1007/s10798-021-09695-1 · Cited by 37
“Where replication studies fail to replicate the results of original studies, it can be an indication of sampling error, measurement error, validity issues, or QRP’s in the original study (Schmidt, 2009).”
ResultWalach, H. (2013) · Wiley eBooks · doi:10.1002/9781118591277.ch4 · Cited by 29
“A host of failed replications and debates are testimony to this, and only through huge struggles and debates does science succeed to create a consensual understanding of the shared material world (Schmidt, 2009).”
BackgroundWhy, when, and how to Replicate Research
Abbuhl, R. (2011) · Wiley eBooks · doi:10.1002/9781444347340.ch15 · Cited by 31
Practices and attitudes toward replication in empirical translation and interpreting studies
Olalla-Soler, C. (2019) · Target International Journal of Translation Studies · doi:10.1075/target.18159.ola · Cited by 27
“The repetition of the methods that led to a reported finding (Schmidt 2009), that is, replication, is a key concept in the philosophy of science.”
BackgroundMethodInfluentialVolatility shocks and investment behavior
Huber, C., Huber, J., & Kirchler, M. (2021) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2021.12.007 · Cited by 30
Cobern, W. W., Adams, B. A., Pleasants, B. A.-S., Bentley, A., & Kagumba, R. (2022) · Science & Education · doi:10.1007/s11191-021-00292-1 · Cited by 30
“We quickly discovered, however, that “there are many different meanings to [replication] and the relevant procedures, but hardly any systematic literature” (Schmidt, 2009, p. 90).”
BackgroundRonimus, M., Tolvanen, A. J., & Hautala, J. (2022) · Learning and Individual Differences · doi:10.1016/j.lindif.2022.102197 · Cited by 35
Koepp, A. E., Watts, T. W., Gershoff, E. T., Ahmed, S. F., Davis-Kean, P. E., Duncan, G. J., Kuhfeld, M. R., & Vandell, D. L. (2023) · Developmental Psychology · doi:10.1037/dev0001533 · Cited by 45
Pope, J. P., & Wilson, P. M. (2014) · International Journal of Sport and Exercise Psychology · doi:10.1080/1612197x.2014.956325 · Cited by 38
“Specifically, we sought to conduct conceptual replication which replicates a hypothesis or results from previous research with varied methods (Schmidt, 2009).”
BackgroundMethodInformation Systems Foundations: Theory Building in Information Systems
Dennis Hart, N., & Gregor, S. D. (2012) · ANU Press eBooks · doi:10.22459/isf.10.2012 · Cited by 25
Zamary, A., & Rawson, K. A. (2016) · Educational Psychology Review · doi:10.1007/s10648-016-9396-9 · Cited by 29
“The first purpose of experiment 2 was to replicate these findings (for recent emphasis on the importance of replicating novel findings, see Lishner 2015; Maner 2014; Schmidt 2009; Simons 2014).”
BackgroundReplication and the Manufacture of Scientific Inferences: A Formal Approach
García, F. M. (2016) · International Studies Perspectives · doi:10.1093/isp/ekv011 · Cited by 23
Linder, C., & Farahbakhsh, S. (2020) · Business Ethics Quarterly · doi:10.1017/beq.2019.52 · Cited by 26
“This is why recommendations to make scientists more accountable by making data publicly available (Sijtsma, 2016), replicating data regularly (Schmidt, 2009), improving statistical knowledge (Wagenmakers, 2007), or creating open and collegial environment and training (Anderson, 2007) have not…”
BackgroundOn the role of creativity in the formation of new complex words
Kőrtvélyessy, L., Štekauer, P., & Kačmár, P. (2021) · Linguistics · doi:10.1515/ling-2020-0003 · Cited by 30
Nachtigall, V., & Rummel, N. (2021) · Instructional Science · doi:10.1007/s11251-021-09556-3 · Cited by 24
“Specifically, we pursued what Schmidt (2016) calls a conceptual replication.”
BackgroundResultVelez, M., Jamshidi, P., Siegmund, N., Apel, S., & Kästner, C. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510043 · Cited by 34
“Afterwards, we replicate the study, intentionally varying some aspects of the designs (theoretical replication [34, 63]), to evaluate to which extent our information providers generalize for a more complex task with an interaction in a larger system.”
MethodThompson, K. W., Sikora, D. M., Perrewé, P. L., & Ferris, G. R. (2015) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12112 · Cited by 30
A New Replication Norm for Psychology
LeBel, E. P. (2015) · Collabra · doi:10.1525/collabra.23 · Cited by 33
“A conceptual replication, on the other hand, intentionally alters some (or several) aspects of the methodology to test whether a finding generalizes to different experimental manipulations, measures, or contexts [3, 84].”
BackgroundGlöckner, A., Fiedler, S., & Renkewitz, F. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000384 · Cited by 28
Jitendra, A. K., Harwell, M. R., Im, S., Karl, S. R., & Slater, S. C. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000335 · Cited by 27
Rubin, M. (2019) · Synthese · doi:10.1007/s11229-019-02433-0 · Cited by 27
“…p. 137; Cumming 2008; De Ruiter 2018; Earp and Trafimow 2015; Hampel 2003, p. 3; Hansen 2011; Iso-Ahola 2017; Lindsay and Ehrenberg 1993; Macdonald 1997, p. 337; Nosek and Errington 2017, 2019;Nosek et al. 2012; Rubin 2017a; Schmidt 2009; Schneider 2015; Stroebe and Strack 2014; Zwaan et al. 2018).”
BackgroundMethodInfluentialWhen working memory mechanisms compete: Predicting cognitive flexibility versus mental set
Stockum, C. A. V., & DeCaro, M. S. (2020) · Cognition · doi:10.1016/j.cognition.2020.104313 · Cited by 33
“By demonstrating convergent, functionally opponent effects across problem types, Study 3 strengthens the validity of the underlying theory of functional opponency in WMC (Schmidt, 2009).”
BackgroundWhy does collaborative retrieval improve memory? Enhanced relational and item-specific processing
Wissman, K. T., & Rawson, K. A. (2015) · Journal of Memory and Language · doi:10.1016/j.jml.2015.05.003 · Cited by 25
What Has Replication Ever Done for Us? Insights from Neuroimaging of Speech Perception
Evans, S. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00041 · Cited by 25
“A broad distinction exists between direct replication, in which an identical procedure is repeated with the aim to recreate the previous experiment in its entirety, and conceptual replication, in which a previous result or hypothesis is tested with different methods (Schmidt, 2009).”
BackgroundAAN position statement: Ethical issues in clinical research in neurology
Tolchin, B. D., CONWIT, R. A., Epstein, L. G., & Russell, J. A. (2020) · Neurology · doi:10.1212/wnl.0000000000009241 · Cited by 24
“demonstrate statistically significant results that cannot be reproduced by other researchers.(39,40) Some failures of replication are likely due to intervening scientific and technological advances, failures to perfectly reproduce original study conditions, and the use of insufficient sample sizes in replicated studie…”
ResultMatanda, D. J., Mittelmark, M. B., Urke, H. B., & Amugsi, D. A. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005194 · Cited by 29
Meanings for Sex and Commitment Among First Semester College Men and Women: A Mixed-Methods Analysis
Olmstead, S. B., Anders, K. M., & Conrad, K. A. (2016) · Archives of Sexual Behavior · doi:10.1007/s10508-016-0777-4 · Cited by 29
Science: the slow march of accumulating evidence
Picho, K., Maggio, L. A., & Artino, A. R. (2016) · Perspectives on Medical Education · doi:10.1007/s40037-016-0305-1 · Cited by 24
Toward Application and Testing of Measurement Scales and an Example
Kuppelwieser, V. G., Putinas, A.-C., & Bastounis, M. (2017) · Sociological Methods & Research · doi:10.1177/0049124117701486 · Cited by 24
Unpredictability increases the error-related negativity in children and adolescents
Speed, B. C., Jackson, F. L., Nelson, B. D., Infantolino, Z. P., & Hajcak, G. (2017) · Brain and Cognition · doi:10.1016/j.bandc.2017.09.006 · Cited by 27
“Although uncommon in the published literature, reproducibility is a critical and basic component of empirical science that warrants prioritization in psychological science (Giner-Sorolla, 2012; Schmidt, 2009), particularly given the current replication ‘crisis’ suggested by recent replication efforts (Open Science Col…”
Thanheiser, E. (2018) · Journal for Research in Mathematics Education · doi:10.5951/jresematheduc.49.1.0039 · Cited by 29
Chick, G., Hsu, Y., Yeh, C.-K., Hsieh, C., Iarmolenko, S., Bae, S. Y., Xue, L., & Dong, E. (2018) · Leisure Sciences · doi:10.1080/01490400.2018.1506724 · Cited by 22
“Despite the claim that replication is “one of the central issues in any empirical science” (Schmidt, 2009, p. 90), it is rare in the social sciences (Kepes & McDaniel, 2013).”
BackgroundPsychologists’ psychologies of psychologists in a time of crisis.
Morawski, J. G. (2020) · History of Psychology · doi:10.1037/hop0000140 · Cited by 23
Coudevylle, G. R., Boulley, G. E., Finez, L., Eugène, K., & Robin, N. (2020) · Educational Psychology · doi:10.1080/01443410.2020.1746237 · Cited by 25
“Replicating an earlier study is interesting in itself as, over the past decade, replications have been highly encouraged and valued in the social sciences (see e.g., Open Science Collaboration, 2012, 2015; Schmidt, 2009).”
BackgroundVachon, B., Curran, J., Karunananthan, S., Brehaut, J., Graham, I. D., Moher, D., Sales, A. E., Straus, S. E., Fiander, M., Paprica, P. A., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.027 · Cited by 29
Quantify and control reproducibility in high-throughput experiments
Zhao, Y., Sampson, M. G., & Wen, X. (2020) · Nature Methods · doi:10.1038/s41592-020-00978-4 · Cited by 28
Wucherpfennig, F., Rubel, J. A., Hollon, S. D., & Lutz, W. (2016) · Behaviour Research and Therapy · doi:10.1016/j.brat.2016.11.003 · Cited by 29
Lebois, L. A. M., Wilson‐Mendenhall, C. D., & Barsalou, L. W. (2015) · Cognitive Science · doi:10.1111/cogs.12295 · Cited by 25
“Replicating previous research is of much current interest (e.g., Schmidt, 2009; Zwaan & Pecher, 2012).”
BackgroundMethodInfluentialStar, J. R. (2018) · Journal for Research in Mathematics Education · doi:10.5951/jresematheduc.49.1.0098 · Cited by 22
Charting the Constellation of Science Reform
Field, S. M. (2022) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.229114775 · Cited by 30
Doing better leadership science via replications and registered reports
Carsten, M. K., Clapp‐Smith, R., Haslam, S. A., Bastardoz, N., Gooty, J., Connelly, S., & Spain, S. M. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101712 · Cited by 24
Conroy, M. A., Sutherland, K. S., Granger, K. L., Marcoulides, K. M., Huang, K., & Montesion, A. (2021) · Journal of Early Intervention · doi:10.1177/10538151211018662 · Cited by 27
“Therefore, in this study, we proposed to conduct a conceptual replication (Schmidt, 2009; Travers et al., 2016) of BEST in CLASS to examine and compare the intervention effects when the professional development component was delivered over the web in comparison to on-site.”
MethodThe romance of leadership: Rekindling the fire through replication of Meindl and Ehrlich
Hammond, M. M., Schyns, B., Lester, G. V., Clapp‐Smith, R., & Thomas, J. S. (2021) · The Leadership Quarterly · doi:10.1016/j.leaqua.2021.101538 · Cited by 21
Walach, H., Kirmse, K. A., Sedlmeier, P., Vogt, H., Hinterberger, T., & von Lucadou, W. (2022) · Journal of Scientific Exploration · doi:10.31275/20212031 · Cited by 18
How Identity Impacts Bystander Responses to Workplace Mistreatment
Gloor, J. L. P., Okimoto, T. G., Li, X., Gazdag, B. A., & Ryan, M. K. (2023) · Journal of Management · doi:10.1177/01492063231177976 · Cited by 26
Single-case research designs and the scientist-practitioner ideal in applied psychology.
Blampied, N. M. (2012) · American Psychological Association eBooks · doi:10.1037/13937-008 · Cited by 29
Content and structure of laboratory packages for software engineering experiments
Solari, M., Vegas, S., & Juristo, N. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.12.016 · Cited by 28
“The use of such types of knowledge transfer mechanisms is widespread among researchers from other disciplines, such as physics, biology and the social sciences [11] [12] [13].”
BackgroundTyler, D. C., Hitt, B. D., Nterful, F., & Mettling, M. (2019) · College & Research Libraries · doi:10.5860/crl.80.4.525 · Cited by 24
von Thiele Schwarz, U., Förberg, U., Sundell, K., & Hasson, H. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0496-8 · Cited by 20
“with different methods or, more commonly, in different contexts, thus investigating the generalisability of the findings [5, 6].”
ResultReplication Success Under Questionable Research Practices—a Simulation Study
Freuli, F., Held, L., & Heyard, R. (2023) · Statistical Science · doi:10.1214/23-sts904 · Cited by 20
Impact of Video and Subtitle Speed on Subtitle Reading: An Eye-tracking Replication Study
Szarkowska, A., Ragni, V., Orrego-Carmona, D., Black, S., Szkriba, S., Kruger, J., Krejtz, K., & Silva, B. B. (2024) · Journal of Audiovisual Translation · doi:10.47476/jat.v7i1.2024.283 · Cited by 28
Jagged competencies: Measuring the reliability of generative AI in academic research
Thomas, L. D. W., Romasanta, A. K. S., & Priego, L. P. (2025) · Journal of Business Research · doi:10.1016/j.jbusres.2025.115804 · Cited by 28
The Rewards of Authority in the Workplace: Do Gender and Age Matter?
Schieman, S., Schafer, M. H., & McIvor, M. (2013) · Sociological Perspectives · doi:10.1525/sop.2012.56.1.75 · Cited by 22
“Replications of key findings such as these are critically important to the field (Freese 2007; Nosek, Spies, and Motyl in press; Schmidt 2009).”
BackgroundTrail and Ultrarunning: The Impact of Distance, Nature, and Personality on Flow and Well-Being
Martinez, C. T., & Scott, C. (2016) · Psi Chi Journal of Psychological Research · doi:10.24839/2164-8204.jn21.1.6 · Cited by 29
Cultural influences on the processing of social comparison feedback signals—an ERP study
Pfabigan, D. M., Wucherer, A. M., Wang, X., Pan, X., Lamm, C., & Han, S. (2018) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsy097 · Cited by 23
“This clearly emphasizes the importance of choosing appropriate analytical methods to minimize possible confounding variables in future cultural neuroscience research, in particular given the current replication efforts to establish reproducibility as indisputable component of empirical research (Schmidt, 2009; Pashler…”
MethodBernárdez, B., Durán, A., Parejo, J. A., & Ruiz–Cortés, A. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.06.104 · Cited by 21
The Joint Importance of Secure and Satisfying Work: Insights from Three Studies
Shoss, M. K., Brummel, B. J., Probst, T. M., & Jiang, L. (2019) · Journal of Business and Psychology · doi:10.1007/s10869-019-09627-w · Cited by 27
The need for open science practices and well-conducted replications in the field of gambling studies
Wohl, M. J. A., Tabri, N., & Zelenski, J. M. (2019) · International Gambling Studies · doi:10.1080/14459795.2019.1672769 · Cited by 23
“Instead, the best that a researcher can hope for is a replication study that only differs from the original study in terms of the participants who take part (see Schmidt, 2009).”
BackgroundThe New Reality of Reproducibility: The Role of Data Work in Scientific Research
Feinberg, M., Sutherland, W., Nelson, S. B., Jarrahi, M. H., & Rajasekar, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392840 · Cited by 28
“Schmidt [50] points out that if all aspects of the replication are the same as the original experiment then they are the same experiment and the replication does not serve as a confirmation.”
BackgroundMethodSex Education, Public Opinion, and Pornography: A Conditional Process Analysis
Wright, P. J. (2018) · Journal of Health Communication · doi:10.1080/10810730.2018.1472316 · Cited by 22
“As is the case with other nuanced theoretical formulations addressing a broad social phenomenon, attempts at falsification will have to proceed in a patterned manner, targeting particular elements of the model across a series of direct and conceptual replication studies (Schmidt, 2009).”
Background
Metadata
- Authors
- Schmidt, S.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Schmidt, Stefan
- ORCID
- Schmidt, Stefan
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 90-100
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2021458971
- Cited by
- 1,125
- References
- 55
- Citations by year
- 2026: 42 · 2025: 95 · 2024: 88 · 2023: 85 · 2022: 75 · 2021: 138 · 2020: 94 · 2019: 105 · 2018: 93 · 2017: 74 · 2016: 84 · 2015: 52 · 2014: 35 · 2013: 26 · 2012: 23
- FWCI
- 16.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of General Psychology (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, social sciences, fact-checking, publication practices, research methodology, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.