Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologies
The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
Abstract
STUDY QUESTION: Are there differences in the methodological quality of Cochrane systematic reviews (CRs) and non-Cochrane systematic reviews (NCRs) of assisted reproductive technologies? SUMMARY ANSWER: CRs on assisted reproduction are of higher methodological quality than similar reviews published in other journals. WHAT IS KNOWN ALREADY: The quality of systematic reviews varies. STUDY DESIGN, SIZE AND DURATION: This was a cross-sectional study of 30 CR and 30 NCR systematic reviews that were randomly selected from the eligible reviews identified from a literature search for the years 2007-2011. MATERIALS, SETTING AND METHODS: We extracted data on the reporting and methodological characteristics of the included systematic reviews. We assessed the methodological quality of the reviews using the 11-domain Measurement Tool to Assess the Methodological Quality of Systematic Reviews (AMSTAR) tool and subsequently compared CR and NCR systematic reviews. MAIN RESULTS AND THE ROLE OF CHANCE: The AMSTAR quality assessment found that CRs were superior to NCRs. For 10 of 11 AMSTAR domains, the requirements were met in >50% of CRs, but only 4 of 11 domains showed requirements being met in >50% of NCRs. The strengths of CRs are the a priori study design, comprehensive literature search, explicit lists of included and excluded studies and assessments of internal validity. Significant failings in the CRs were found in duplicate study selection and data extraction (67% meeting requirements), assessment for publication bias (53% meeting requirements) and reporting of conflicts of interest (47% meeting requirements). NCRs were more likely to contain methodological weaknesses as the majority of the domains showed <40% of reviews meeting requirements, e.g. a priori study design (17%), duplicate study selection and data extraction (17%), assessment of study quality (27%), study quality in the formulation of conclusions (23%) and reporting of conflict of interests (10%). LIMITATIONS, REASONS FOR CAUTION: The AMSTAR assessment can only judge what is reported by authors. Although two of the five authors are involved in the production of CRs, the risk of bias was reduced by not involving these authors in the assessment of the systematic review quality. WIDER IMPLICATIONS OF THE FINDINGS: Not all systematic reviews are equal. The reader needs to consider the quality of the systematic review when they consider the results and the conclusions of a systematic review. STUDY FUNDING/COMPETING INTEREST(S): There are no conflicts with any commercial organization. Funding was provided for the students by the summer studentship programme of the Faculty of Medical and Health Sciences of the University of Auckland.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The other two were recent systematic reviews and were selected from the Cochrane Database of Systematic Reviews as Cochrane Reviews tend to be of higher quality than those published in peer review journals [18, 19, 20, 21, 22, 23].”
From Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies (2025) · cites this paper as [19]
In the citing paper
“The rationale for prioritising inclusion of Cochrane reviews over non-Cochrane reviews was that Cochrane reviews are regularly updated and are of higher quality [52–55].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [55]
Citations
Cites
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.005 · Cited by 19,575
Cochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Self-citationPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147 · cited 1×
“Research that is funded by pharmaceutical companies is more likely to publish favourable outcomes supporting the company (Lexchin et al., 2003).”
BackgroundSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693 · cited 9×
“Systematic reviews attempt to provide the best available evidence by providing pooled results from studies that answer a specific research question and meet pre-specified eligibility criteria.”
BackgroundMethodInfluentialWhittington, C. J., Kendall, T., Fonagy, P., Cottrell, D., Cotgrove, A., & Boddington, E. (2004) · The Lancet · doi:10.1016/s0140-6736(04)16043-1 · Cited by 1,058 · cited 1×
“One study found that by including unpublished data with negative findings, the conclusion changed from being favourable to unfavourable for selective serotonin reuptake inhibitors (Whittington et al., 2004).”
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898 · cited 2×
“Publication bias (where trials that have positive findings are published more quickly and more often than those with negative findings; Hopewell et al., 2009) may result in different conclusions being drawn and recommendations being made by the review authors, and therefore may affect the practice of clinicians.”
BackgroundExternal Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)
Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001350 · Cited by 569 · cited 2×
“This tool was developed (Shea et al., 2007a) and validated (Shea et al., 2007b) in 2007 and assesses 11 aspects of methodological quality with the aim of critically appraising systematic reviews (Table I).”
MethodSelf-citation
Research works printed but not linked: 2
- Green, 2011, Chapter 1: Introduction, 3
- Higgins, 2011, Chapter 4: Guide to the contents of a Cochrane protocol and review, 51
Cited by
Systematic review adherence to methodological or reporting quality
Pussegoda, K., Turner, L., Garritty, C., Mayhew, A., Skidmore, B., Stevens, A., Boutron, I., Sarkis-Onofre, R., Bjerre, L. M., Hróbjartsson, A., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0527-2 · Cited by 322
Lunny, C., Pieper, D., Thabet, P., & Kanji, S. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01269-y · Cited by 252
Kelly, S. E., Moher, D., & Clifford, T. J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0258-9 · Cited by 137
“Any or all of these components may be tailored or even omitted as part of the streamlining processes designed to expedite systematic review completion times [3] so were unable to ascertain whether the reporting omissions were attributable to the rapid review approach or simply poor reporting [88].”
BackgroundProbiotics and gastrointestinal conditions: An overview of evidence from the Cochrane Collaboration
Parker, E. A., Roy, T., D'Adamo, C. R., & Wieland, L. S. (2018) · Nutrition · doi:10.1016/j.nut.2017.06.024 · Cited by 147
“[29] Windsor B, Popovich I, Jordan V, Showell M, Shea B, Farquhar C.”
BackgroundPieper, D., Puljak, L., González‐Lorenzo, M., & Minozzi, S. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.004 · Cited by 130
Metadata
- Authors
- Windsor, B., Popovich, I., Jordan, V., Showell, M., Shea, B., & Farquhar, C.
- Affiliations
- University of Auckland (NZ)
- University of Ottawa (CA)
- Corresponding author
- Farquhar, C.
- Venue
- Human Reproduction
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-10-02
- Volume
- 27
- Issue
- 12
- Pages
- 3460-3466
- ISSN
- 0268-1161 (ISSN-L)
- 1460-2350
- OpenAlex
- W2167473079
- Cited by
- 46
- References
- 11
- Citations by year
- 2025: 4 · 2024: 5 · 2023: 5 · 2022: 2 · 2021: 4 · 2020: 7 · 2019: 1 · 2018: 4 · 2017: 6 · 2016: 3 · 2015: 3 · 2014: 1 · 2013: 1
- FWCI
- 0.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Human Reproduction (publishedVersion)
- PubMed (publishedVersion)
- ResearchSpace (University of Auckland) (submittedVersion)
- Funders
- University of Auckland
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- methodological quality, systematic reviews, assisted reproduction, subfertility, AMSTAR, reporting quality, publication bias, conflicts of interest
- MeSH terms
- Cross-Sectional Studies, Female, Humans, Infertility, Male, Pregnancy, Publishing, Research Design, Review Literature as Topic, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.