The role of method in treatment effectiveness research: Evidence from meta-analysis.
Abstract
A synthesis of 319 meta-analyses of psychological, behavioral, and educational treatment research was conducted to assess the influence of study method on observed effect sizes relative to that of substantive features of the interventions. An index was used to estimate the proportion of effect size variance associated with various study features. Study methods accounted for nearly as much variability in study outcomes as characteristics of the interventions. Type of research design and operationalization of the dependent variable were the method features associated with the largest proportion of variance. The variance as a result of sampling error was about as large as that associated with the features of the interventions studied. These results underscore the difficulty of detecting treatment outcomes, the importance of cautiously interpreting findings from a single study, and the importance of meta-analysis in summarizing results across studies.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Psychosocial interventions that target behavioral change often do not yield a significant effect.
“psychosocial interventions, that target behavioral change often do not yield a significant effect (48).”
Cites
The Jackknife, the Bootstrap and Other Resampling Plans
Efron, B. (1982) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970319 · Cited by 8,120 · see citing works
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · see citing works
Meta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635 · see citing works
Meta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · see citing works
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · see citing works
Self-citationMooney, C., & Duval, R. (1993) · doi:10.4135/9781412983532 · Cited by 893 · see citing works
Task complexity as a moderator of goal effects: a meta-analysis
Wood, R. E., Mento, A. J., & Locke, E. A. (1987) · Journal of Applied Psychology · doi:10.1037/0021-9010.72.3.416 · Cited by 598 · see citing works
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · see citing works
A Meta-analysis of the Effects of Advance Organizers on Learning and Retention
Luiten, J., Ames, W., & Ackerson, G. (1980) · American Educational Research Journal · doi:10.3102/00028312017002211 · Cited by 151 · see citing works
Kraemer, H. C., Gardner, C., Brooks, J. O., & Yesavage, J. A. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.1.23 · Cited by 141 · see citing works
Random versus nonrandom assignment in controlled experiments: Do you get the same answer?
Shadish, W. R., & Ragsdale, K. (1996) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.64.6.1290 · Cited by 126 · see citing works
Heinsman, D. T., & Shadish, W. R. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.154 · Cited by 125 · see citing works
A Meta-Analysis of Advance Organizer Studies
Stone, C. L. (1983) · The Journal of Experimental Education · doi:10.1080/00220973.1983.11011862 · Cited by 102 · cited 1× · see citing works
“*Stone, C. L. (1983). A meta-analysis of advance organizer”
BackgroundIntentional and Incidental Learning with Instructional Texts: A Meta-Analysis for 1970–1980
Klauer, K. J. (1984) · American Educational Research Journal · doi:10.3102/00028312021002323 · Cited by 42 · cited 1× · see citing works
“*Klauer, K. J. (1984). Intentional and incidental learning”
Background
Flagged first, then by citations
Cited by
Sorted by citations
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Irwin, M. R., Olmstead, R., & Carroll, J. E. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.05.014 · Cited by 2,085 · see citing works
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011) · Review of Educational Research · doi:10.3102/0034654310393361 · Cited by 1,029 · see citing works
Cook, D. A., & Reed, D. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000786 · Cited by 781 · see citing works
The effectiveness of correctional rehabilitation: a review of systematic reviews
Lipsey, M. W., & Cullen, F. T. (2007) · Annual Review of Law and Social Science · doi:10.1146/annurev.lawsocsci.3.081806.112833 · Cited by 763 · see citing works
Self-citationSample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542 · see citing works
Anglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534 · see citing works
Goldberg, S. B., Riordan, K. M., Sun, S., & Davidson, R. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620968771 · Cited by 520 · see citing works
“Although some previous reviews have conducted a second-order meta-analysis by quantitatively combining results across included meta-analyses (e.g., Wilson & Lipsey, 2001), this approach has the substantial limitation of overrepresenting studies published earlier (that may be more likely to appear…”
MethodMcCart, M. R., Priester, P. E., Davies, W. H., & Azen, R. (2006) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-006-9031-1 · Cited by 505 · see citing works
Menting, A. T. A., Orobio de Castro, B., & Matthys, W. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.07.006 · Cited by 417 · see citing works
Squires, J. E., Sullivan, K., Eccles, M. P., Worswick, J., & Grimshaw, J. M. (2014) · Implementation Science · doi:10.1186/s13012-014-0152-6 · Cited by 368 · see citing works
Steenbergen-Hu, S., & Cooper, H. (2014) · Journal of Educational Psychology · doi:10.1037/a0034752 · Cited by 355 · see citing works
“In cases for which only inference results were reported but no means and standard deviations were available, g was estimated from the inferential statistics, such as t, F, or p-values (Wilson & Lipsey, 2001).”
Busch, T., & Friede, G. (2018) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1480 · Cited by 345 · see citing works
Brewin, C. R., Kleiner, J. S., Vasterling, J. J., & Field, A. P. (2007) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.116.3.448 · Cited by 290 · see citing works
Murray, J., Farrington, D. P., & Eisner, M. P. (2009) · Journal of Experimental Criminology · doi:10.1007/s11292-008-9066-0 · Cited by 280 · see citing works
Quantitative Research Methods, Study Quality, and Outcomes: The Case of Interaction Research
Plonsky, L., & Gass, S. (2011) · Language Learning · doi:10.1111/j.1467-9922.2011.00640.x · Cited by 267 · see citing works
Study Designs for Effectiveness and Translation Research
Mercer, S. L., DeVinney, B. J., Fine, L. J., Green, L. W., & Dougherty, D. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.04.005 · Cited by 222 · see citing works
Cooper, H., & Koenka, A. C. (2012) · American Psychologist · doi:10.1037/a0027119 · Cited by 196 · see citing works
Donker, T., Bennett, K., Bennett, A., Mackinnon, A., van Straten, A., Cuijpers, P., Christensen, H., & Griffiths, K. M. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2307 · Cited by 154 · see citing works
Hall, J., Kellett, S., Berrios, R., Bains, M. K., & Scott, S. (2016) · The American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2016.06.006 · Cited by 157 · see citing works
Wilson, D. B. (2009) · Journal of Experimental Criminology · doi:10.1007/s11292-009-9085-5 · Cited by 86 · see citing works
Self-citationCurrent methods for meta-analysis
Schulze, R. (2007) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.215.2.90 · Cited by 79 · see citing works
Wilson, C. J., Deane, F. P., Marshall, K. L., & Dalley, A. (2008) · Journal of Youth and Adolescence · doi:10.1007/s10964-007-9225-z · Cited by 61 · see citing works
Toews, I., Anglemyer, A., Nyirenda, J. L., Alsaid, D., Balduzzi, S., Grummich, K., Schwingshackl, L., & Bero, L. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub3 · Cited by 60 · see citing works
Wolf, B., & Harbatkin, E. (2023) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2071364 · Cited by 45 · see citing works
Knight, L., & Samuel, V. (2022) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2022.06.006 · Cited by 37 · see citing works
Interaction of Theory and Practice to Assess External Validity
Leviton, L. C., & Trujillo, M. D. (2017) · Evaluation Review · doi:10.1177/0193841x15625289 · Cited by 35 · see citing works
Lasebikan, V. O., & Ola, B. A. (2016) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2016.00134 · Cited by 8 · see citing works
- Authors
- Wilson, D. B., & Lipsey, M. W.
- Affiliations
- George Mason University (US)
- Venue
- Psychological Methods
- Published
- 2001
- Volume
- 6
- Issue
- 4
- Pages
- 413-429
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2087570586
- Cited by
- 261
- References
- 16
- Citations by year
- 2026: 16 · 2025: 15 · 2024: 17 · 2023: 15 · 2022: 10 · 2021: 12 · 2020: 12 · 2019: 8 · 2018: 9 · 2017: 7 · 2016: 13 · 2015: 11 · 2014: 12 · 2013: 13 · 2012: 9
- FWCI
- 3.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (1r01mh051701-01a1)
- National Institute of Mental Health (R01-MH51701)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Keywords
- treatment effectiveness, research design, effect size heterogeneity, sampling error, intervention characteristics, psychological intervention, behavioral interventions, educational intervention
- MeSH terms
- Behavior Therapy, Clinical Trials as Topic, Data Interpretation, Statistical, Humans, Outcome and Process Assessment, Health Care, Psychology, Educational, Psychometrics, Psychotherapy, Research, Reproducibility of Results, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
