Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies.
Abstract
This study deals with some of the judgmental factors involved in selecting effect sizes from within the studies that enter a meta-analysis. Particular attention is paid to the conceptual redundancy rule that Smith, Glass, and Miller (1980) used in their study of the effectiveness of psychotherapy for deciding which effect sizes should and should not be counted in determining an overall effect size. Data from a random sample of 25 studies from Smith et al.'s (1980) population of psychotherapy outcome studies were first recoded and then reanalyzed meta-analytically. Using the conceptual redundancy rule, three coders independently coded effect sizes and identified more than twice as many of them per study as did Smith et al. Moreover, the treatment effect estimates associated with this larger sample of effects ranged between .30 and .50, about half the size claimed by Smith et al. Analyses of other rules for selecting effect sizes showed that average effect estimates also varied with these rules. Such results indicate that the average effect estimates derived from meta-analyses may depend heavily on judgmental factors that enter into how effect sizes are selected within each of the individual studies considered relevant to a meta-analysis.
Cites (2)
- Meta-analysis of psychotherapy outcome studies.Smith, Mary L., Glass, Gene V. · 1977 · American Psychologist
- Meta-analysis of comparative therapy outcome studies: a replication and refinement.Shapiro, David A., Shapiro, Diana · 1982 · Psychological Bulletin
Cited by (12)
- The Handbook of Research Synthesis and Meta-Analysis2019
- Identifying and developing empirically supported child and adolescent treatments.Kazdin, Alan E., Weisz, John R. · 1998 · Journal of Consulting and Clinical Psychology
- Principles Underlying the Use of Multiple Informants' ReportsDe Los Reyes, Andres, Thomas, Sarah A., Goodman, Kimberly L., Kundey, Shannon M.A. · 2013 · Annual Review of Clinical Psychology
- Effects of psychosocial interventions on quality of life in adult cancer patients: meta analysis of 37 published controlled outcome studiesRehse, Barbara, Pukrop, Ralf · 2003 · Patient Education and Counseling
- Psychotherapy for Children and AdolescentsKazdin, Alan E. · 2003 · Annual Review of Psychology
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. https://app.cochat.ai/papers/10.1037//0033-2909.105.1.106
- The paper
- Matt, G. E. (1989). Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. Psychological Bulletin. https://doi.org/10.1037//0033-2909.105.1.106
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.