Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologies
The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
Abstract
STUDY QUESTION: Are there differences in the methodological quality of Cochrane systematic reviews (CRs) and non-Cochrane systematic reviews (NCRs) of assisted reproductive technologies? SUMMARY ANSWER: CRs on assisted reproduction are of higher methodological quality than similar reviews published in other journals. WHAT IS KNOWN ALREADY: The quality of systematic reviews varies. STUDY DESIGN, SIZE AND DURATION: This was a cross-sectional study of 30 CR and 30 NCR systematic reviews that were randomly selected from the eligible reviews identified from a literature search for the years 2007-2011. MATERIALS, SETTING AND METHODS: We extracted data on the reporting and methodological characteristics of the included systematic reviews. We assessed the methodological quality of the reviews using the 11-domain Measurement Tool to Assess the Methodological Quality of Systematic Reviews (AMSTAR) tool and subsequently compared CR and NCR systematic reviews. MAIN RESULTS AND THE ROLE OF CHANCE: The AMSTAR quality assessment found that CRs were superior to NCRs. For 10 of 11 AMSTAR domains, the requirements were met in >50% of CRs, but only 4 of 11 domains showed requirements being met in >50% of NCRs. The strengths of CRs are the a priori study design, comprehensive literature search, explicit lists of included and excluded studies and assessments of internal validity. Significant failings in the CRs were found in duplicate study selection and data extraction (67% meeting requirements), assessment for publication bias (53% meeting requirements) and reporting of conflicts of interest (47% meeting requirements). NCRs were more likely to contain methodological weaknesses as the majority of the domains showed <40% of reviews meeting requirements, e.g. a priori study design (17%), duplicate study selection and data extraction (17%), assessment of study quality (27%), study quality in the formulation of conclusions (23%) and reporting of conflict of interests (10%). LIMITATIONS, REASONS FOR CAUTION: The AMSTAR assessment can only judge what is reported by authors. Although two of the five authors are involved in the production of CRs, the risk of bias was reduced by not involving these authors in the assessment of the systematic review quality. WIDER IMPLICATIONS OF THE FINDINGS: Not all systematic reviews are equal. The reader needs to consider the quality of the systematic review when they consider the results and the conclusions of a systematic review. STUDY FUNDING/COMPETING INTEREST(S): There are no conflicts with any commercial organization. Funding was provided for the students by the summer studentship programme of the Faculty of Medical and Health Sciences of the University of Auckland.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The other two were recent systematic reviews and were selected from the Cochrane Database of Systematic Reviews as Cochrane Reviews tend to be of higher quality than those published in peer review journals [18, 19, 20, 21, 22, 23].”
From Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies (2025) · cites this paper as [19]
In the citing paper
“The rationale for prioritising inclusion of Cochrane reviews over non-Cochrane reviews was that Cochrane reviews are regularly updated and are of higher quality [52–55].”
From Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews (2021) · cites this paper as [55]
Citations
Cites
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.005 · Cited by 19,575
Cochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Self-citationPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147 · cited 1×
“Research that is funded by pharmaceutical companies is more likely to publish favourable outcomes supporting the company (Lexchin et al., 2003).”
BackgroundSystematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693 · cited 9×
“Systematic reviews attempt to provide the best available evidence by providing pooled results from studies that answer a specific research question and meet pre-specified eligibility criteria.”
BackgroundMethodInfluentialWhittington, C. J., Kendall, T., Fonagy, P., Cottrell, D., Cotgrove, A., & Boddington, E. (2004) · The Lancet · doi:10.1016/s0140-6736(04)16043-1 · Cited by 1,058 · cited 1×
“One study found that by including unpublished data with negative findings, the conclusion changed from being favourable to unfavourable for selective serotonin reuptake inhibitors (Whittington et al., 2004).”
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898 · cited 2×
“Publication bias (where trials that have positive findings are published more quickly and more often than those with negative findings; Hopewell et al., 2009) may result in different conclusions being drawn and recommendations being made by the review authors, and therefore may affect the practice of clinicians.”
BackgroundExternal Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)
Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001350 · Cited by 569 · cited 2×
“This tool was developed (Shea et al., 2007a) and validated (Shea et al., 2007b) in 2007 and assesses 11 aspects of methodological quality with the aim of critically appraising systematic reviews (Table I).”
MethodSelf-citation
Research works printed but not linked: 2
- Green, 2011, Chapter 1: Introduction, 3
- Higgins, 2011, Chapter 4: Guide to the contents of a Cochrane protocol and review, 51
Cited by
Systematic review adherence to methodological or reporting quality
Pussegoda, K., Turner, L., Garritty, C., Mayhew, A., Skidmore, B., Stevens, A., Boutron, I., Sarkis-Onofre, R., Bjerre, L. M., Hróbjartsson, A., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0527-2 · Cited by 322
Lunny, C., Pieper, D., Thabet, P., & Kanji, S. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01269-y · Cited by 252
Kelly, S. E., Moher, D., & Clifford, T. J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0258-9 · Cited by 137
“Any or all of these components may be tailored or even omitted as part of the streamlining processes designed to expedite systematic review completion times [3] so were unable to ascertain whether the reporting omissions were attributable to the rapid review approach or simply poor reporting [88].”
BackgroundProbiotics and gastrointestinal conditions: An overview of evidence from the Cochrane Collaboration
Parker, E. A., Roy, T., D'Adamo, C. R., & Wieland, L. S. (2018) · Nutrition · doi:10.1016/j.nut.2017.06.024 · Cited by 147
“[29] Windsor B, Popovich I, Jordan V, Showell M, Shea B, Farquhar C.”
BackgroundPieper, D., Puljak, L., González‐Lorenzo, M., & Minozzi, S. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.004 · Cited by 130
Caiado, R. G. G., Scavarda, L. F., Azevedo, B. D., Nascimento, D. L. D. M., & Quelhas, O. L. G. (2022) · Sustainability · doi:10.3390/su14020830 · Cited by 125
Interventions for postoperative pain in children: An overview of systematic reviews
Borić, K., Došenović, S., Kadić, A. J., Batinić, M., Ćavar, M., Urlić, M., Markovina, N., & Puljak, L. (2017) · Pediatric Anesthesia · doi:10.1111/pan.13203 · Cited by 102
Goldkuhle, M., Narayan, V. M., Weigl, A., Dahm, P., & Skoetz, N. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-020869 · Cited by 88
Pussegoda, K., Turner, L., Garritty, C., Mayhew, A., Skidmore, B., Stevens, A., Boutron, I., Sarkis-Onofre, R., Bjerre, L. M., Hróbjartsson, A., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0507-6 · Cited by 75
Kruly, P. D. C., Giannini, M., Pascotto, R. C., Tokubo, L. M., Suga, U. S. G., Marques, A. D. C. R., & Terada, R. S. S. (2018) · PLoS ONE · doi:10.1371/journal.pone.0191942 · Cited by 76
Propadalo, I., Tranfic, M., Vuka, I., Barcot, O., Pericic, T. P., & Puljak, L. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.10.002 · Cited by 67
Puljak, L. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12175 · Cited by 48
Reactive Oxygen Species and Male Fertility
O’Flaherty, C. (2020) · Antioxidants · doi:10.3390/antiox9040287 · Cited by 45
“Antioxidant therapy is a common clinical strategy to improve sperm quality and function in infertile men [66,67].”
BackgroundMethodological quality of systematic reviews on influenza vaccination
Remschmidt, C., Wichmann, O., & Harder, T. (2014) · Vaccine · doi:10.1016/j.vaccine.2014.01.060 · Cited by 44
“reviews was superior to non-Cochrane reviews using the AMSTAR tool (15).”
BackgroundMethodGómez-García, F., Ruano, J., Aguilar-Luque, M., Gay-Mimbrera, J., Maestre-Lopez, B., Sanz-Cabanillas, J. L., Carmona-Fernández, P. J., González-Padilla, M., Vélez García-Nieto, A., & Isla-Tejera, B. (2017) · British Journal of Dermatology · doi:10.1111/bjd.15380 · Cited by 39
Half of Cochrane reviews were published more than 2 years after the protocol
Andersen, M. Z., Gülen, S., Fonnes, S., Andresen, K., & Rosenberg, J. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.05.011 · Cited by 40
Garcia-Alamino, J. M., Bankhead, C. R., Heneghan, C. J., Pidduck, N., & Perera, R. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-015888 · Cited by 35
ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods studyPreprint
Pitre, T., Jassal, T., Talukdar, J. R., Shahab, M., Ling, M., & Zeraatkar, D. (2023) · medRxiv · doi:10.1101/2023.11.19.23298727 · Cited by 27
Comparative analysis of Cochrane and non-Cochrane reviews over three decades
Andersen, M. Z., Zeinert, P., Rosenberg, J., & Fonnes, S. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02531-2 · Cited by 21
Interventions for Age-Related Macular Degeneration
Lindsley, K. B., Li, T., Ssemanda, E. N., Virgili, G., & Dickersin, K. (2016) · Ophthalmology · doi:10.1016/j.ophtha.2015.12.004 · Cited by 24
Interventions for tic disorders: An updated overview of systematic reviews and meta analyses
Yang, C., Cheng, X., Zhang, Q., Yu, D., Li, J., & Zhang, L. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.112905 · Cited by 20
Riberholt, C. G., Olsen, M. H., Milan, J. B., Hafliðadóttir, S. H., Svanholm, J. H., Pedersen, E. B., Lew, C. C. H., Asante, M. A., Ribeiro, J. P., Wagner, V., et al. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02318-y · Cited by 18
Cavalcante, M. B., da Silva, P. H. A., Sampaio, O. G. M., Câmara, F. E. A., Cavalcante, C. T. D. M. B., & Barini, R. (2023) · Journal of Reproductive Immunology · doi:10.1016/j.jri.2023.103986 · Cited by 17
Reporting of Cochrane systematic review protocols with network meta‐analyses—A scoping review
Wang, R., Dwan, K. M., Showell, M. G., van Wely, M., Mol, B. W., Askie, L. M., & Seidler, A. L. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1531 · Cited by 13
“Comparison-adjusted funnel plot(27) 19 (42) 15 (54) 4 (24)”
Self-citationLau, O., & Golder, S. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70050 · Cited by 29
Yerokhin, V., Carr, B., Sneed, G., & Vassar, M. B. (2016) · BMC Research Notes · doi:10.1186/s13104-016-2280-3 · Cited by 9
Faggion, C. M., Listl, S., & Alarcón, M. A. (2015) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.12394 · Cited by 8
Twenty years of Cochrane reviews in menstrual disorders and subfertility
Farquhar, C. M., Moore, V., Bhattacharya, S., Blake, D., Vail, A., Thomas, J., Cheong, Y., Showell, M. G., Nagels, H. E., Marjoribanks, J., et al. (2013) · Human Reproduction · doi:10.1093/humrep/det334 · Cited by 9
“A comparative analysis of 30 Cochrane reviews and 30 nonCochrane reviews of ART was undertaken for the years 2007–2011 (Windsor et al., 2012).”
BackgroundSelf-citationGroup authorships in Cochrane had low compliance with Cochrane recommendations
Andersen, M. Z., Fonnes, S., Andresen, K., & Rosenberg, J. (2020) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12396 · Cited by 6
Vireque, A. A., Stolakis, V., Berteli, T. S., Bertero, M. T., & Kofinas, J. D. (2025) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-025-03498-2 · Cited by 4
“The quality of SRs varied between Cochrane and non-Cochrane reviews across several AMSTAR domains, including the search strategies, with non-Cochrane reviews showing more methodological flaws [22].”
Alper, B. S., Fedorowicz, Z., & van Zuuren, E. J. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12161 · Cited by 3
Wieland, L. S., Brassington, R., & Macdonald, G. M. (2019) · European Journal of Integrative Medicine · doi:10.1016/j.eujim.2019.101008 · Cited by 3
Lau, O., & Golder, S. (2025) · medRxiv · doi:10.1101/2025.06.17.25329772 · Cited by 7
Liu, M., Gao, Y., Yang, K., Cai, Y., Xu, J., Dai, D., Wu, S., Zhang, J., & Tian, J. (2023) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12521 · Cited by 2
Amanollahi, A., Moradi‐Lakeh, M., Shokraneh, F., Bashiri, Y., & Mahmudi, L. (2020) · PubMed Central · doi:10.34171/mjiri.34.30 · Cited by 2
“However, preregistration and the published articles in Cochrane journals were also better than the non-Cochrane journals with regards to the quality of meta-analysis and systematic review (13, 17, 20).”
BackgroundSentiment Analysis in Digital Spaces: An Overview of Reviews
Ayravainen, L. E. M., Hinds, J. M., & Davidson, B. I. (2026) · ACM Computing Surveys · doi:10.1145/3842380 · Cited by 2
The 22nd Anniversary of the Cochrane Back and Neck Group
Furlan, A. D., Chou, R., Harbin, S., & Pardo, J. P. (2020) · Spine · doi:10.1097/brs.0000000000003626 · Cited by 1
Amanollahi, A., Moradi‐Lakeh, M., Shokraneh, F., Bashiri, Y., & Mahmudi, L. (2020) · Medical Journal of the Islamic Republic of Iran · doi:10.47176/mjiri.34.30 · Cited by 1
Reda, S. A., Hussein, Y. F., & Riad, M. (2023) · International Journal of Membrane Science and Technology · doi:10.15379/ijmst.v10i5.2561 · Cited by 1
A simple form could prevent authorship issues in Cochrane manuscripts: A cohort study
Fonnes, S., Andresen, K., Öberg, S., Baker, J. J., & Rosenberg, J. (2024) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.12053 · Cited by 1
A review of common evidence review designs published in JAN; a 10‐year review
Noble, H. R., & Smith, J. (2021) · Journal of Advanced Nursing · doi:10.1111/jan.14838
The applicability and utilization of systematic reviews for clinical practice
Lindsley, K. B. (2022) · Utrecht University Repository (Utrecht University) · doi:10.33540/1513
Berghella, V. (2023) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2023.100904
Khedkar, P., Balasubramanian, H., Pai, A. S., Ananthan, A., Anne, R. P., Mohan, D., Kabra, N. S., Rao, S. C., & Patole, S. (2025) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2025.114871
Additional file 1 of Comparative analysis of Cochrane and non-Cochrane reviews over three decades
Andersen, M. Z., Zeinert, P., Rosenberg, J., & Fonnes, S. (2024) · Figshare · doi:10.6084/m9.figshare.26715010
Riberholt, C. G., Olsen, M. H., Milan, J. B., Hafliðadóttir, S. H., Svanholm, J. H., Pedersen, E. B., Lew, C. C. H., Asante, M. A., Pereira Ribeiro, J., Wagner, V., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26974432
Lunny, C., Pieper, D., Thabet, P., & Kanji, S. (2021) · Figshare · doi:10.6084/m9.figshare.14926906
Metadata
- Authors
- Windsor, B., Popovich, I., Jordan, V., Showell, M., Shea, B., & Farquhar, C.
- Affiliations
- University of Auckland (NZ)
- University of Ottawa (CA)
- Corresponding author
- Farquhar, C.
- Venue
- Human Reproduction
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-10-02
- Volume
- 27
- Issue
- 12
- Pages
- 3460-3466
- ISSN
- 0268-1161 (ISSN-L)
- 1460-2350
- OpenAlex
- W2167473079
- Cited by
- 46
- References
- 11
- Citations by year
- 2025: 4 · 2024: 5 · 2023: 5 · 2022: 2 · 2021: 4 · 2020: 7 · 2019: 1 · 2018: 4 · 2017: 6 · 2016: 3 · 2015: 3 · 2014: 1 · 2013: 1
- FWCI
- 0.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Human Reproduction (publishedVersion)
- PubMed (publishedVersion)
- ResearchSpace (University of Auckland) (submittedVersion)
- Funders
- University of Auckland
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- methodological quality, systematic reviews, assisted reproduction, subfertility, AMSTAR, reporting quality, publication bias, conflicts of interest
- MeSH terms
- Cross-Sectional Studies, Female, Humans, Infertility, Male, Pregnancy, Publishing, Research Design, Review Literature as Topic, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.