Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Replication is one of the most important tools for the verification of facts within the empirical sciences. A detailed examination of the notion of replication reveals that there are many different meanings to this concept and the relevant procedures, but hardly any systematic literature. This paper analyzes the concept of replication from a theoretical point of view. It demonstrates that the theoretical demands are scarcely met in everyday work within the social sciences. Some demands are just not feasible, whereas others are constricted by restrictions relating to publication. A new classification scheme based on a functional approach that distinguishes between different types of replication is proposed. Next, it will be argued that replication addresses the important connection between existing and new knowledge. To do so it has to be applied explicitly and systematically. The paper ends with a description of procedures how this could be done and a set of recommendations how to handle the concept of replication in the future to exploit its potential to the full.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“RRs provide a mechanism for publishing replication research”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“which are rarely published in the standard model 13,14 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule [48].”
From The natural selection of bad science (2016) · cites this paper as [48]
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,706
Design and Analysis: A Researcher's Handbook
Beaton, A. E., & Keppel, G. (1975) · American Educational Research Journal · doi:10.2307/1162588 · Cited by 5,822
Scientific Explanation and the Causal Structure of the World
Salmon, W. C. (1985) · Princeton University Press eBooks · doi:10.1515/9780691221489 · Cited by 2,459
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.2307/1869945 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.1086/ahr/92.3.658 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Traynham, J. G., Shapin, S., & Schaffer, S. (1987) · Journal of Interdisciplinary History · doi:10.2307/204299 · Cited by 2,045
Design and Analysis: A Researcher's Handbook
Aaker, D. A., & Keppel, G. (1976) · Journal of Marketing Research · doi:10.2307/3150757 · Cited by 2,011
Research works printed but not linked: 12
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- Beloff J., 1985, The repeatability problem in parapsychology, 1
- Collins H. M., 1985, Changing order
- CookT. D. (1990). Rersearch methodology: Strengthening causal interpretations of nonexperimental data. In SechrestL., PerrinE. & BunkerJ., US Department of health and human services (pp. 9–30). Public Health Service, Agency for Health Care Policy and Research.
- Edge H., 1985, The repeatability problem in parapsychology, 53
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 21
- Rao K. R., 1981, Journal of Parapsychology, 45, 311
- Rosenthal R., 1991, Replication research in the social sciences, 1
- Sargent C. L., 1981, European Journal of Parapsychology, 3, 423
- Schacter D. L., 2000, Wir sind Erinnerung. Gedächtnis und Persönlichkeit
- Walach H., 2000, Neuroendocrinology Letters, 21, 221
- Weizsäcker E. U. v., 1986, Offene Systeme I: Beiträge zur Zeitstruktur von Information, Entropie und Evolution, 82
Cited by
Items 901–1,000 of 1,001
Project M.O.T.A.R: Motivation Towards Athlete Recovery
Deck, S. (2016) · Brock University Digital Repository (Brock University)
The Battle over the Burden of Proof: A Report from the Trenches
Cicchini, M. D. (2018) · University of Pittsburgh Law Review · doi:10.5195/lawreview.2017.525
Westwood, S. J., Olson, A., Miall, R. C., & Romani, C. (2017) · Cortex · doi:10.1016/j.cortex.2017.08.012 · Cited by 1
“There is a difference between a conceptual replication and a direct replication (for discussion, see Cesario, 2014; Schmidt, 2009; Simons, 2014; Stroebe & Strack, 2014).”
Espinosa, E. G., & Ferreira, J. M. (2017) · RECIBE REVISTA ELECTRÓNICA DE COMPUTACIÓN INFORMÁTICA BIOMÉDICA Y ELECTRÓNICA · doi:10.32870/recibe.v5i3.62
Siddaway, A. P. (2017) · Stirling Online Research Repository (University of Stirling)
A Foot in Two Worlds: Exploring Organizational and Professional Dual Identification
Ostermeier, K. K. (2018) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc1157541
“hypothesized relationships being tested and, furthermore, to enrich and bolster theory (Eden, 2002; Schmidt, 2009).”
Kowitt, J. S. (2018) · OpenCommons at University of Connecticut (University of Connecticut)
Emotional callousness and vicarious emotional reactions to the misfortune of others
Steinert, S. W. (2017) · Minds at UW (University of Wisconsin)
Song, W. (2018) · University of Minnesota Digital Conservancy (University of Minnesota) · Cited by 2
Cooperación en Dilemas Sociales: Normas Sociales y Procesos Cognitivos
Ortiz, M. V., & Reyna, C. (2019) · Psykhe (Santiago) · doi:10.7764/psykhe.28.1.1044 · Cited by 1
“Only Stage 1 in both studies is similar to that implemented by the authors, so it can be considered a direct replication according to Schmidt (2009).”
MethodUnderstanding Replication: Trust But Verify
Suter, W. N., & Suter, P. M. (2019) · Home Health Care Management & Practice · doi:10.1177/1084822319850501 · Cited by 1
Motor sequence learning in older adults
Barnhoorn, J. (2018) · doi:10.3990/1.9789036545266 · Cited by 1
Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPreprint
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2016) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/k4bgq_v1
Functional opponency in working memory capacity predicts cognitive flexibility in problem solving.
Stockum, C. A. V. (2019) · ThinkIR: The University of Louisville's Institutional Repository (University of Louisville) · doi:10.18297/etd/3277
“By demonstrating convergent, functionally opponent effects across problem types, Study 3 strengthens the validity of the underlying theory of functional opponency in WMC (Schmidt, 2009).”
BackgroundJSE 35:4 Winter 2021 Whole Issue PDF
Erickson, K. E. (2022) · Journal of Scientific Exploration · doi:10.31275/20212405
Notes on the Fringes of the Replication Crisis in Management Science
Hensel, P. G. (2019) · Problemy Zarządzania - Management Issues · doi:10.7172/1644-9584.84.4
Quick, J. W. (2018) · Digital Commons - Gardner-Webb University (Gardner–Webb University)
Instructing jurors on detecting deception: is the jury still out?
Woskett, J. (2020) · Own your potential (DEAKIN)
Analiza empirycznych argumentów na rzecz tezy o zróżnicowaniu kulturowym intuicji epistemicznych
Sękowski, K. (2019) · Studia Philosophica Wratislaviensia · doi:10.19195/1895-8001.14.2.4
Healthy Eating in The Australian Defence Force: A Segmentation Study
Kitunen, A. (2020) · Griffith Research Online · doi:10.25904/1912/107
Hong, J., Hwang, M., & Tai, K. (2020) · IntechOpen eBooks · doi:10.5772/intechopen.93902
Independent replication of classic trials in neurosurgery: A missing validation practice
Kerr, N., Kerr, J., Sheridan, M. M. P., & Holcombe, A. O. (2020) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2020.10.028
JSE 34:4 Winter Whole Issue PDF
Erickson, K. E. (2020) · Journal of Scientific Exploration
Transparency and replication in alcohol research
Drax, K., & Munafo, M. R. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-816720-5.00005-0
Psychological resilience at work: A labyrinth worth navigating
IJntema, R. C. (2020) · Utrecht University Repository (Utrecht University) · doi:10.33540/159 · Cited by 6
Bausell, R. B. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197536537.003.0007
Krepel, N. (2021) · Research Publications (Maastricht University) · doi:10.26481/dis.20210618nk
A unified view of lateralized vision
Brederoo, S. (2021) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.166393133 · Cited by 2
Predicting relationship change across the transition to parenthood: Resources and risk factors
ter Kuile, H. (2021) · Utrecht University Repository (Utrecht University) · doi:10.33540/816
The Constructs of Antisocial Behavior and Crime: A Measurement View
Blanc, M. L. (2021) · Springer eBooks · doi:10.1007/978-3-030-68429-7_1
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.001
Construct and Data-Evaluation Validity
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.004
Single-Case Experimental Research Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.009
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.007
Cautions, Negative Effects, and Replication
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.016
Selecting Measures for Research
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.011
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.021
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.013
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.022
Ideas that Begin the Research Process
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.005
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.020
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.025
Presenting and Analyzing the Data
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.015
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.024
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.018
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.002
Assessment: Types of Measures and Their Use
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.012
Experimental Research Using Group Designs
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.006
Ethical Issues and Guidelines for Research
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.017
Communication of Research Findings
Kazdin, A. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108993647.019
Replication crisisPreprint
Quang-Loc, N. (2022) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/4wd5x
The Natural Selection of Bad Psychological ScientistsPreprint
Aarts, A. A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4176925
How physical cues in micro food environments influence consumption: a social norm account
Raghoebar, S. (2021) · doi:10.18174/533544 · Cited by 1
Processing the Word Red and Intellectual Performance: Four Replication AttemptsPreprint
Gnambs, T., Kovacs, C., & Stiglbauer, B. (2019) · doi:10.31234/osf.io/4ertj
Blouin‐Hudon, E. C. (2019) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2019-13813 · Cited by 1
Multiplicity Control vs Replication: Making an Obvious Choice Even More ObviousPreprint
Hunter, A., Beribisky, N., Farmus, L., & Cribbie, R. A. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/hbe48
Psychological Science Replicates Just Fine, ThanksPreprint
Aarts, A. A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.4176922
Mellenbergh, G. J. (2019) · Springer eBooks · doi:10.1007/978-3-030-12272-0_19
Harald Atmanspacher, A., & Maasen, S. (2016) · John Wiley & Sons, Inc. eBooks · doi:10.1002/9781118865064.ch0
Identifying Factors that Increase Sustainability of SWPBIS in US Middle Schools
Hoffert, H. L. (2022) · The Research Repository @ WVU (West Virginia University) · doi:10.33915/etd.11168 · Cited by 1
“, 2016), and show that results can be generalized to other settings (Schmidt, 2009).”
BackgroundSchool Social Relations and Child Development: Gene-Environment Interplay
Brendgen, M. R., & Kretschmer, T. (2022) · Springer eBooks · doi:10.1007/978-3-031-07109-6_4
Aarts, A. A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4176916
Aarts, A. A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.4176882
Cloudy, J., Banks, J., & Bowman, N. D. (2022) · Technology Mind and Behavior · doi:10.1037/tmb0000083.supp
Kim, I. K. (2022) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.21220838 · Cited by 1
Mason, E. N., & Lembke, E. S. (2022) · Assessment for Effective Intervention · doi:10.1177/15345084221133730
“Direct replications intend to exactly reproduce a study’s procedures; conceptual replications employ different methods but are aimed at testing the same hypotheses as a previous study (Schmidt, 2009).”
MethodA case against social networks as the only means for recruitment in scientific studies with humansPreprint
Pastor, A. (2022) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/84utm
Experimentalforschung in der politischen Kommunikation
Koch, T. (2022) · Springer eBooks · doi:10.1007/978-3-658-26233-4_49
Positive Psychological Coaching: An investigation on its efficacy in the workplace
Corbu, A., & Corbu, A. (2023) · Tesis Doctorals en Xarxa (Consorci de Serveis Universitaris de Catalunya) · doi:10.6035/14109.2023.486784
Syeda, U. H., South, L., Raynor, J., Panavas, L., Saffo, D., Morriss, T., Dunne, C., & Borkin, M. A. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/dferj
The multiversal methodology as a remedy of the replication crisisPreprint
Cantone, G. G. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/kuhmz
Titleman, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4457689
Cantone, G. G., & Tomaselli, V. (2023) · Proceedings e report · doi:10.36253/979-12-215-0106-3.10
Pittelkow, M.-M. (2023) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.762945985
Lachney, M., & Kuyenga, M. C. A. (2023) · History and Technology · doi:10.1080/07341512.2023.2251212
Los neuroderechos frente a la neuropredicción del delito y la reincidencia delictiva
Guarneros, A. I. R. (2023) · Revista Mexicana de Ciencias Penales · doi:10.57042/rmcp.v7i21.666
People with Hearing Loss: Effects of Reading Instruction on Vocal Verbal Repertoire
Verdu, A. C. M. A., Lucchesi, F. D. M., & Silva, L. T. N. (2023) · Springer eBooks · doi:10.1007/978-3-031-40868-7_9
Empirische Forschung und Theorie-Entwicklung
Eisend, M., & Kuß, A. (2023) · Springer eBooks · doi:10.1007/978-3-658-42690-3_9
Replicability and Faith in Science: Implications of Tacit Knowledge
Festré, A., & Østbye, S. (2023) · Historical-analytical studies on nature, mind and action · doi:10.1007/978-3-031-51228-5_2
Early Career Researchers Must Tell Stories About Their Work
Reyes, A. D. L. (2024) · Springer eBooks · doi:10.1007/978-3-031-54284-8_1
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0007
Why Theories About Rater Biases Fail to Explain Discrepant Results
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0006
The Ubiquity of Discrepant Results in Research
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0002
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0004
How Discrepant Results Reveal Blueprints for Discovery
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0004
Constructing Measurement Batteries
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0009
The Ubiquity of Discrepant Results in the Clinic
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0017
Distinguishing Domain-Relevant Information From Bias and Noise
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0010
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0022
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0008
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0021 · Cited by 1
Insights About Data Sources, Validation Testing, and Theory
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.003.0013
Reyes, A. D. L. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780197686607.002.0006
Metadata
- Authors
- Schmidt, S.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Schmidt, Stefan
- ORCID
- Schmidt, Stefan
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 90-100
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2021458971
- Cited by
- 1,125
- References
- 55
- Citations by year
- 2026: 42 · 2025: 95 · 2024: 88 · 2023: 85 · 2022: 75 · 2021: 138 · 2020: 94 · 2019: 105 · 2018: 93 · 2017: 74 · 2016: 84 · 2015: 52 · 2014: 35 · 2013: 26 · 2012: 23
- FWCI
- 16.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of General Psychology (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, social sciences, fact-checking, publication practices, research methodology, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.