Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Replication is one of the most important tools for the verification of facts within the empirical sciences. A detailed examination of the notion of replication reveals that there are many different meanings to this concept and the relevant procedures, but hardly any systematic literature. This paper analyzes the concept of replication from a theoretical point of view. It demonstrates that the theoretical demands are scarcely met in everyday work within the social sciences. Some demands are just not feasible, whereas others are constricted by restrictions relating to publication. A new classification scheme based on a functional approach that distinguishes between different types of replication is proposed. Next, it will be argued that replication addresses the important connection between existing and new knowledge. To do so it has to be applied explicitly and systematically. The paper ends with a description of procedures how this could be done and a set of recommendations how to handle the concept of replication in the future to exploit its potential to the full.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“RRs provide a mechanism for publishing replication research”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“which are rarely published in the standard model 13,14 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [14]
In the citing paper
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule [48].”
From The natural selection of bad science (2016) · cites this paper as [48]
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Journal of Consciousness Studies
(2022) · Journal of Consciousness Studies · doi:10.53765/20512201 · Cited by 6,832
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,706
Design and Analysis: A Researcher's Handbook
Beaton, A. E., & Keppel, G. (1975) · American Educational Research Journal · doi:10.2307/1162588 · Cited by 5,822
Scientific Explanation and the Causal Structure of the World
Salmon, W. C. (1985) · Princeton University Press eBooks · doi:10.1515/9780691221489 · Cited by 2,459
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.2307/1869945 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Cohen, I. B., Shapin, S., & Schaffer, S. (1987) · The American Historical Review · doi:10.1086/ahr/92.3.658 · Cited by 2,151
Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life
Traynham, J. G., Shapin, S., & Schaffer, S. (1987) · Journal of Interdisciplinary History · doi:10.2307/204299 · Cited by 2,045
Design and Analysis: A Researcher's Handbook
Aaker, D. A., & Keppel, G. (1976) · Journal of Marketing Research · doi:10.2307/3150757 · Cited by 2,011
Scientific Explanation and the Causal Structure of the World
Salmon, W. C. (2020) · Princeton University Press eBooks · doi:10.2307/j.ctv173f2gh · Cited by 2,007
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Publication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological Bulletin · doi:10.1037/h0026141 · Cited by 1,350
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
That's Interesting!: Towards a Phenomenology of Sociology and a Sociology of Phenomenology
Davis, M. S. (1971) · Philosophy of the Social Sciences · doi:10.1177/004839317100100211 · Cited by 1,219
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“…1993) 54% of the reviewers indicated they would prefer a study with new findings over a replication study.6 When asked what the problems with publishing replications were, comments such as “Not newsworthy” – “Waste of space” “. . . waste of resources” were given (see also Greenwald, 1978).”
BackgroundZuckerman, H., & Merton, R. K. (1971) · Minerva · doi:10.1007/bf01553188 · Cited by 789
The Logic of Scientific Discovery
Jeffrey, R. C., & Popper, K. R. (1960) · Econometrica · doi:10.2307/1907578 · Cited by 674
The golem: what everyone should know about science
(1994) · Choice Reviews Online · doi:10.5860/choice.31-2650 · Cited by 534
Dictionary of the Social Sciences
(2002) · Oxford University Press eBooks · doi:10.1093/acref/9780195123715.001.0001 · Cited by 507
The Design of Replicated Studies
Lindsay, R. M., & Ehrenberg, A. S. C. (1993) · The American Statistician · doi:10.1080/00031305.1993.10475983 · Cited by 490
The effect of experimenter bias on the performance of the albino rat
Rosenthal, R., & Fode, K. L. (2007) · Behavioral Science · doi:10.1002/bs.3830080302 · Cited by 384
Does psi exist? Replicable evidence for an anomalous process of information transfer.
Bem, D. J., & Honorton, C. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.4 · Cited by 330
A Dictionary of the Social Sciences
Pagès, R., Gould, J., Kolb, W. L., & Pages, R. (1965) · Revue Française de Sociologie · doi:10.2307/3318984 · Cited by 306
Ebbinghaus, H., & Psychologie, D. G. F. (1906) · Cited by 292
Replication in behavioral research.
Rosenthal, R. (1990) · Journal of social behavior and personality · Cited by 214
Replication Research in the Social Sciences
Madden, C. S., & Neuliep, J. W. (1992) · Journal of Marketing · doi:10.2307/1251993 · Cited by 195
Editorial bias against replication research.
Neuliep, J. W., & Crandall, R. (1990) · Journal of social behavior and personality · Cited by 139
How Journal Editors View Replication Research
Madden, C. S., Easley, R. W., & Dunn, M. G. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673490 · Cited by 126
Open exchange and epistemic progress.
Mahoney, M. J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.1.29 · Cited by 121
Reviewer bias against replication research.
Neuliep, J. W., & Crandall, R. (1993) · Journal of social behavior and personality · Cited by 113
The logic of generalization: Five principles common to experiments and ethnographies
Shadish, W. R. (1995) · American Journal of Community Psychology · doi:10.1007/bf02506951 · Cited by 89
Cantwell, Z. M. (1992) · Evaluation Practice · doi:10.1016/0886-1633(92)90030-f · Cited by 83
Psychologie, D. G. F. (1970) · Akademische Verlagsgesellschaft eBooks · Cited by 82
Experimental essays : Versuche zum Experiment
Heidelberger, M., & Steinle, F. (1998) · Cited by 79
In and About the World: Philosophical Studies of Science and Technology
Constant, E. W., & Radder, H. (1998) · Technology and Culture · doi:10.2307/1215908 · Cited by 70
Complementarity is a useful concept for consciousness studies. A Reminder.
Walach, H., & Römer, H. (2000) · PubMed · Cited by 58
ESP and Psychokinesis: A Philosophical Examination.
Giere, R. N., & Braude, S. E. (1982) · The Philosophical Review · doi:10.2307/2184638 · Cited by 55
Walach, H., Schmidt, S., Dirhold, T., & Nosch, S. (2002) · International Journal of Psychophysiology · doi:10.1016/s0167-8760(01)00188-x · Cited by 53
Self-citationDo We Really Need More Replications?
Hunt, K. P. (1975) · Psychological Reports · doi:10.2466/pr0.1975.36.2.587 · Cited by 53
The Metaphysics of Science: An Account of Modern Science in Terms of Principles, Laws and Theories
Maxwell, N. (2009) · International Studies in the Philosophy of Science · doi:10.1080/02698590903007220 · Cited by 43
Angelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C. J., Hoey, J., Horton, R., Kotzin, S., Laine, C. M., Marušić, A., Overbeke, A. J. P. M., et al. (2004) · PubMed · Cited by 37
(1937) · Science · doi:10.1126/science.85.2198.171.b · Cited by 37
Experimental Reproducibility and the Experimenters’ Regress
Radder, H. (1992) · PSA Proceedings of the Biennial Meeting of the Philosophy of Science Association · doi:10.1086/psaprocbienmeetp.1992.1.192744 · Cited by 36
Walach, H., & Schmidt, S. (2005) · Journal of Consciousness Studies · Cited by 31
Self-citationThe Metaphysics of Science: An Account of Modern Science in terms of Principles, Laws and Theories
Dilworth, C. (2014) · doi:10.1007/1-4020-3838-0 · Cited by 30
Walach, H., Schmidt, S., Bihr, Y.-M., & Wiesch, S. (2001) · European Psychologist · doi:10.1027//1016-9040.6.1.15 · Cited by 26
Self-citationSome methodological aspects of tacit knowledge elicitation
Ральфовна, Ч. К., & Wasserman, E. L. (2000) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/095281300146308 · Cited by 22
The ability to detect unseen staring: A literature review and empirical tests
Colwell, J., Schröder, S., & Sladen, D. (2000) · British Journal of Psychology · doi:10.1348/000712600161682 · Cited by 19
CIBA Foundation Symposium on Extrasensory Perception.
Wilkins, L. T., Wolstenholme, G. E. W., & Millar, E. C. P. (1957) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2342562 · Cited by 11
Wiseman, R. J., Smith, M. J., & Milton, J. (1998) · British Journal of Psychology · doi:10.1111/j.2044-8295.1998.tb02696.x · Cited by 10
Ciba Foundation Symposium ‐ Extrasensory Perception
(1956) · Novartis Foundation symposium · doi:10.1002/9780470718964 · Cited by 7
[Analysis of concepts, conditions and aims of replications].
SCHWEIZER, K. W. (1989) · PubMed · Cited by 5
Research works printed but not linked: 12
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- Beloff J., 1985, The repeatability problem in parapsychology, 1
- Collins H. M., 1985, Changing order
- CookT. D. (1990). Rersearch methodology: Strengthening causal interpretations of nonexperimental data. In SechrestL., PerrinE. & BunkerJ., US Department of health and human services (pp. 9–30). Public Health Service, Agency for Health Care Policy and Research.
- Edge H., 1985, The repeatability problem in parapsychology, 53
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 21
- Rao K. R., 1981, Journal of Parapsychology, 45, 311
- Rosenthal R., 1991, Replication research in the social sciences, 1
- Sargent C. L., 1981, European Journal of Parapsychology, 3, 423
- Schacter D. L., 2000, Wir sind Erinnerung. Gedächtnis und Persönlichkeit
- Walach H., 2000, Neuroendocrinology Letters, 21, 221
- Weizsäcker E. U. v., 1986, Offene Systeme I: Beiträge zur Zeitstruktur von Information, Entropie und Evolution, 82
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
“Although sometimes replication efforts can reveal poorly designed studies and irreproducible results, this is more the exception than the rule (Schmidt, 2009).”
Background
Metadata
- Authors
- Schmidt, S.
- Affiliations
- University Medical Center Freiburg (DE)
- Corresponding author
- Schmidt, Stefan
- ORCID
- Schmidt, Stefan
- Venue
- Review of General Psychology
- Publisher
- SAGE Publications
- Published
- 2009-06
- Volume
- 13
- Issue
- 2
- Pages
- 90-100
- ISSN
- 1089-2680 (ISSN-L)
- 1939-1552
- OpenAlex
- W2021458971
- Cited by
- 1,125
- References
- 55
- Citations by year
- 2026: 42 · 2025: 95 · 2024: 88 · 2023: 85 · 2022: 75 · 2021: 138 · 2020: 94 · 2019: 105 · 2018: 93 · 2017: 74 · 2016: 84 · 2015: 52 · 2014: 35 · 2013: 26 · 2012: 23
- FWCI
- 16.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of General Psychology (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replication, social sciences, fact-checking, publication practices, research methodology, epistemology of science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.