Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies.

Matt, Georg E.·1989·Psychological Bulletin

Abstract

This study deals with some of the judgmental factors involved in selecting effect sizes from within the studies that enter a meta-analysis. Particular attention is paid to the conceptual redundancy rule that Smith, Glass, and Miller (1980) used in their study of the effectiveness of psychotherapy for deciding which effect sizes should and should not be counted in determining an overall effect size. Data from a random sample of 25 studies from Smith et al.'s (1980) population of psychotherapy outcome studies were first recoded and then reanalyzed meta-analytically. Using the conceptual redundancy rule, three coders independently coded effect sizes and identified more than twice as many of them per study as did Smith et al. Moreover, the treatment effect estimates associated with this larger sample of effects ranged between .30 and .50, about half the size claimed by Smith et al. Analyses of other rules for selecting effect sizes showed that average effect estimates also varied with these rules. Such results indicate that the average effect estimates derived from meta-analyses may depend heavily on judgmental factors that enter into how effect sizes are selected within each of the individual studies considered relevant to a meta-analysis.

Cites (2)

  1. Meta-analysis of psychotherapy outcome studies.
    Smith, Mary L., Glass, Gene V. · 1977 · American Psychologist
  2. Meta-analysis of comparative therapy outcome studies: a replication and refinement.
    Shapiro, David A., Shapiro, Diana · 1982 · Psychological Bulletin

Show all 2 references

Cited by (12)

  1. The Handbook of Research Synthesis and Meta-Analysis
    2019
  2. Identifying and developing empirically supported child and adolescent treatments.
    Kazdin, Alan E., Weisz, John R. · 1998 · Journal of Consulting and Clinical Psychology
  3. Principles Underlying the Use of Multiple Informants' Reports
    De Los Reyes, Andres, Thomas, Sarah A., Goodman, Kimberly L., Kundey, Shannon M.A. · 2013 · Annual Review of Clinical Psychology
  4. Effects of psychosocial interventions on quality of life in adult cancer patients: meta analysis of 37 published controlled outcome studies
    Rehse, Barbara, Pukrop, Ralf · 2003 · Patient Education and Counseling
  5. Psychotherapy for Children and Adolescents
    Kazdin, Alan E. · 2003 · Annual Review of Psychology

See all 53

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. https://app.cochat.ai/papers/10.1037//0033-2909.105.1.106
The paper
Matt, G. E. (1989). Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. Psychological Bulletin. https://doi.org/10.1037//0033-2909.105.1.106

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.