THE PROMISE AND PITFALLS OF SYSTEMATIC REVIEWS
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
▪ Abstract The systematic review “movement” that has transformed medical journal reports of clinical trials and reviews of clinical trials has taken hold in public health, with the most recent milestone, the publication of the first edition of The Guide to Community Health Services in 2005. In this paper we define and distinguish current terms, point out important resources for systematic reviews, describe the impact of systematic review on the quality of primary studies and summaries of the evidence, and provide perspectives on the promise of systematic reviews for shaping the agenda for public health research. Several pitfalls are discussed, including a false sense of rigor implied by the terms “systematic review” and “meta-analysis” and substantial variation in the validity of claims that a particular intervention is “evidence based,” and the difficulty of translating conclusions from systematic reviews into public health advocacy and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“For example, the problem presented by the exponential growth of research reports in the social sciences in the middle years of the twentieth century spurred Glass to coin the term “meta-analysis” in 1976 (37) for the work he and colleagues had undertaken on the relationship between class size and school achievement (3…”
BackgroundImproving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
Des Jarlais, D. C., Lyles, C., Crepaz, N., & the TREND Group (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.361 · Cited by 2,817
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545 · cited 1×
“Chief among them is CONSORT, Consolidated Standards of Reporting of Controlled Randomized Trials (10) and its update (71, 97), followed by a proposed extension to cluster randomized trials (35) and critiques of ethical and analytic issues in such trials (31, 77).”
BackgroundMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · cited 1×
“For example, the problem presented by the exponential growth of research reports in the social sciences in the middle years of the twentieth century spurred Glass to coin the term “meta-analysis” in 1976 (37) for the work he and colleagues had undertaken on the relationship between class size and school achievement (3…”
BackgroundThe Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · cited 2×
“Further, application of the various scoring schemes resulted in different conclusions (54).”
BackgroundMethodGlasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003) · American Journal of Public Health · doi:10.2105/ajph.93.8.1261 · Cited by 1,836
Systematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308 · cited 1×
“In the mid-1990s, a review of schema in use for scoring the quality of randomized clinical trials identified 25 scales and 9 checklists with diverse emphases, lack of empiric support for the scoring components, and general lack of psychometric assessment (68).”
Background
Cited by
Evidence-Based Public Health: A Fundamental Concept for Public Health Practice
Brownson, R. C., Fielding, J. E., & Maylahn, C. M. (2009) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.031308.100134 · Cited by 1,124
Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields
Borrego, M., Foster, M. J., & Froyd, J. E. (2014) · Journal of Engineering Education · doi:10.1002/jee.20038 · Cited by 620
Research collaboration in universities and academic entrepreneurship: the-state-of-the-art
Bozeman, B., Fay, D., & Slade, C. P. (2013) · The Journal of Technology Transfer · doi:10.1007/s10961-012-9281-8 · Cited by 582
External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)
Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001350 · Cited by 569
“The quality of published reviews can vary considerably, even when they try to answer the same question [2].”
BackgroundHarnack, L. J., & French, S. A. (2008) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-5-51 · Cited by 222
“Methods In accord with guidelines for conducting and reporting systematic reviews [15] a structured approach was utilized to identify, review, and draw conclusions from studies that have examined the effect of point-of-purchase calorie labeling on food choices.”
Metadata
- Authors
- Mullen, P. D., & Ramírez, G.
- Venue
- Annual Review of Public Health
- Publisher
- Annual Reviews
- Published
- 2006-04-01
- Volume
- 27
- Issue
- 1
- Pages
- 81-102
- ISSN
- 0163-7525 (ISSN-L)
- 1545-2093
- OpenAlex
- W2107764410
- Cited by
- 82
- References
- 25
- FWCI
- 2.81
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: THE PROMISE AND PITFALLS OF SYSTEMATIC REVIEWS. https://app.cochat.ai/papers/10.1146/annurev.publhealth.27.021405.102239
- The paper
- Mullen, P. D., & Ramírez, G. (2006). THE PROMISE AND PITFALLS OF SYSTEMATIC REVIEWS. Annual Review of Public Health. https://doi.org/10.1146/annurev.publhealth.27.021405.102239
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.