Replication in prevention science
Abstract
Replication research is essential for the advancement of any scientific field. In this paper, we argue that prevention science will be better positioned to help improve public health if (a) more replications are conducted; (b) those replications are systematic, thoughtful, and conducted with full knowledge of the trials that have preceded them; and (c) state-of-the art techniques are used to summarize the body of evidence on the effects of the interventions. Under real-world demands it is often not feasible to wait for multiple replications to accumulate before making decisions about intervention adoption. To help individuals and agencies make better decisions about intervention utility, we outline strategies that can be used to help understand the likely direction, size, and range of intervention effects as suggested by the current knowledge base. We also suggest structural changes that could increase the amount and quality of replication research, such as the provision of incentives and a more vigorous pursuit of prospective research registers. Finally, we discuss methods for integrating replications into the roll-out of a program and suggest that strong partnerships with local decision makers are a key component of success in replication research. Our hope is that this paper can highlight the importance of replication and stimulate more discussion of the important elements of the replication process. We are confident that, armed with more and better replications and state-of-the-art review methods, prevention science will be in a better position to positively impact public health.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is no single method for assessing the success or failure of replication attempts.
“There is no single method for assessing the success or failure of replication attempts”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Valentine et al., 2011
Cites
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977 · see citing works
Systematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693 · see citing works
Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination
Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Mościcki, E. K., Schinke, S., Valentine, J. C., & Ji, P. (2005) · Prevention Science · doi:10.1007/s11121-005-5553-y · Cited by 1,338 · see citing works
Self-citationHow Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294 · see citing works
Self-citationShall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · see citing works
Effects of psychotherapy for depression in children and adolescents: A meta-analysis
Weisz, J. R., McCarty, C. A., & Valeri, S. M. (2006) · Psychological Bulletin · doi:10.1037/0033-2909.132.1.132 · Cited by 942 · see citing works
Issues in Disseminating and Replicating Effective Prevention Programs
Elliott, D. S., & Mihalic, S. (2004) · Prevention Science · doi:10.1023/b:prev.0000013981.28071.52 · Cited by 794 · see citing works
Ennett, S. T., Tobler, N. S., Ringwalt, C. L., & Flewelling, R. L. (1994) · American Journal of Public Health · doi:10.2105/ajph.84.9.1394 · Cited by 655 · see citing works
Outcome selection bias in meta-analysis
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm415oa · Cited by 188 · cited 2× · see citing works
“If measured outcomes within studies are censored due to the lack of statistical significance, this can also lead to incorrect inferences (Williamson et al. 2005).”
BackgroundEditorial: Journal of Negative Results in Biomedicine
Pfeffer, C., & Olsen, B. R. (2002) · Journal of Negative Results in BioMedicine · doi:10.1186/1477-5751-1-2 · Cited by 62 · cited 2× · see citing works
“In medicine, one approach to this solving this problem has been to create new publication vehicles, such as the Journal of Negative Results in Biomedical Research (see Pfeffer and Olsen 2002 ).”
Background
Flagged first, then by citations
Cited by
Sorted by citations
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550 · see citing works
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742 · see citing works
Worldwide application of prevention science in adolescent health
Catalano, R. F., Fagan, A. A., Gavin, L. E., Greenberg, M. T., Irwin, C. E., Ross, D. A., & Shek, D. T. (2012) · The Lancet · doi:10.1016/s0140-6736(12)60238-4 · Cited by 674 · see citing works
Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015) · Prevention Science · doi:10.1007/s11121-015-0555-x · Cited by 627 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works
Society for Prevention Research Type 2 Translational Task Force Members and Contributing Authors, Spoth, R., Rohrbach, L. A., Greenberg, M., Leaf, P., Brown, C. H., Fagan, A., Catalano, R. F., Pentz, M. A., Sloboda, Z., et al. (2013) · Prevention Science · doi:10.1007/s11121-012-0362-6 · Cited by 283 · see citing works
Nosek, B. A., & Errington, T. M. (2017) · eLife · doi:10.7554/elife.23383 · Cited by 242 · see citing works
The SOS Suicide Prevention Program: Further Evidence of Efficacy and Effectiveness
Schilling, E. A., Aseltine, R. H., & James, A. (2016) · Prevention Science · doi:10.1007/s11121-015-0594-3 · Cited by 212 · see citing works
Valentine, J. C., & Thompson, S. G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1064 · Cited by 130 · see citing works
Self-citationNewton, N. C., Champion, K. E., Slade, T., Chapman, C., Stapinski, L., Koning, I., Tonks, Z., & Teesson, M. (2017) · Drug and Alcohol Review · doi:10.1111/dar.12497 · Cited by 111 · see citing works
- Authors
- Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P.
- Affiliations
- University of Louisville (US)
- Oregon Research Institute (US)
- University of Pennsylvania (US)
- Arizona State University (US)
- Pennsylvania State University (US)
- Oregon State University (US)
- American Institutes for Research (US)
- Columbia University (US)
- Corresponding author
- Valentine, Jeffrey C.
- ORCID
- Venue
- Prevention Science
- Published
- 2011-05-04
- Volume
- 12
- Issue
- 2
- Pages
- 103-117
- ISSN
- 1389-4986 (ISSN-L)
- 1573-6695
- OpenAlex
- W1985740357
- Cited by
- 166
- References
- 12
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 5 · 2023: 4 · 2022: 12 · 2021: 9 · 2020: 11 · 2019: 13 · 2018: 21 · 2017: 10 · 2016: 21 · 2015: 15 · 2014: 14 · 2013: 12 · 2012: 6
- FWCI
- 11.63
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Prevention Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Keywords
- replication study, intervention effectiveness, evidence synthesis, research register, research incentives
- MeSH terms
- Health Services Research, Preventive Medicine, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
