Outcomes in Cochrane Systematic Reviews Addressing Four Common Eye Conditions: An Evaluation of Completeness and Comparability
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
INTRODUCTION: Choice of outcomes is critical for clinical trialists and systematic reviewers. It is currently unclear how systematic reviewers choose and pre-specify outcomes for systematic reviews. Our objective was to assess the completeness of pre-specification and comparability of outcomes in all Cochrane reviews addressing four common eye conditions. METHODS: We examined protocols for all Cochrane reviews as of June 2013 that addressed glaucoma, cataract, age-related macular degeneration (AMD), and diabetic retinopathy (DR). We assessed completeness and comparability for each outcome that was named in ≥ 25% of protocols on those topics. We defined a completely-specified outcome as including information about five elements: domain, specific measurement, specific metric, method of aggregation, and time-points. For each domain, we assessed comparability in how individual elements were specified across protocols. RESULTS: We identified 57 protocols addressing glaucoma (22), cataract (16), AMD (15), and DR (4). We assessed completeness and comparability for five outcome domains: quality-of-life, visual acuity, intraocular pressure, disease progression, and contrast sensitivity. Overall, these five outcome domains appeared 145 times (instances). Only 15/145 instances (10.3%) were completely specified (all five elements) (median = three elements per outcome). Primary outcomes were more completely specified than non-primary (median = four versus two elements). Quality-of-life was least completely specified (median = one element). Due to largely incomplete outcome pre-specification, conclusive assessment of comparability in outcome usage across the various protocols per condition was not possible. DISCUSSION: Outcome pre-specification was largely incomplete; we encourage systematic reviewers to consider all five elements. This will indicate the importance of complete specification to clinical trialists, on whose work systematic reviewers depend, and will indirectly encourage comparable outcome choice to reviewers undertaking related research questions. Complete pre-specification could improve efficiency and reduce bias in data abstraction and analysis during a systematic review. Ultimately, more completely specified and comparable outcomes could make systematic reviews more useful to decision-makers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Core outcome measures for chronic pain clinical trials: IMMPACT recommendations
Dworkin, R. H., Turk, D. C., Farrar, J. T., Haythornthwaite, J. A., Jensen, M. P., Katz, N. P., Kerns, R. D., Stucki, G., Allen, R. R., Bellamy, N., et al. (2005) · Pain · doi:10.1016/j.pain.2004.09.012 · Cited by 3,732
Core outcome domains for chronic pain clinical trials: IMMPACT recommendations
Turk, D. C., Dworkin, R. H., Allen, R. R., Bellamy, N., Brandenburg, N., Carr, D. B., Cleeland, C., Dionne, R., Farrar, J. T., Galer, B. S., et al. (2003) · Pain · doi:10.1016/j.pain.2003.08.001 · Cited by 3,510
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 1×
“using a cohort of Cochrane reviews [36] and by various investigators studying this issue in clinical trials [14], [17], [37]–[38].”
MethodDeveloping core outcome sets for clinical trials: issues to consider
Williamson, P. R., Altman, D. G., Blazeby, J. M., Clarke, M., Devane, D., Gargon, E., & Tugwell, P. (2012) · Trials · doi:10.1186/1745-6215-13-132 · Cited by 1,661
Simultaneous comparison of multiple treatments: Combining direct and indirect evidence.
Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005) · BMJ · doi:10.1136/bmj.331.7521.897 · Cited by 1,600 · cited 1×
“Cochrane ‘‘overviews’’ (Cochrane reviews which compile evidence from related reviews of interventions into a single accessible and usable document) [6], and network metaanalyses (analyses of three or more interventions for a given condition in one meta-analysis) [22]–[23] are examples of formal comparisons across syst…”
BackgroundThe nuts and bolts of PROSPERO: an international prospective register of systematic reviews
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-2 · Cited by 1,433 · cited 1×
“This practice will facilitate Cochrane’s contribution of its protocols and updates to PROSPERO [32]–[33], an international database of prospectively registered systematic reviews.”
MethodKirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133 · cited 2×
“using a cohort of Cochrane reviews [36] and by various investigators studying this issue in clinical trials [14], [17], [37]–[38].”
BackgroundMethodThe ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838
Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu, S. (2009) · JAMA · doi:10.1001/jama.2009.1242 · Cited by 730
Chan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613 · cited 1×
“using a cohort of Cochrane reviews [36] and by various investigators studying this issue in clinical trials [14], [17], [37]–[38].”
Method
Cited by
Registration of systematic reviews in PROSPERO: 30,000 records and counting
Page, M. J., Shamseer, L., & Tricco, A. C. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0699-4 · Cited by 509
Mayo-Wilson, E., Fusco, N., Li, T., Hong, H., Canner, J. K., & Dickersin, K. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.007 · Cited by 131
“For example, protocols for systematic reviews usually specify the outcome domains and the time-points of interest [41]; however, few define their methods for handling multiple measures or multiple analyses [13, 6, 35].”
BackgroundSelf-citationLi, T., Higgins, J. P., & Deeks, J. J. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch5 · Cited by 127
Self-citationMayo‐Wilson, E., Li, T., Fusco, N., Dickersin, K., & for the MUDS investigators (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1277 · Cited by 114
Self-citationCherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy
Mayo-Wilson, E., Li, T., Fusco, N., Bertizzolo, L., Canner, J. K., Cowley, T., Doshi, P., Ehmsen, J., Gresham, G., Guo, N., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.014 · Cited by 108
Self-citationHarms are assessed inconsistently and reported inadequately part 1: systematic adverse events
Mayo-Wilson, E., Fusco, N., Li, T., Hong, H., Canner, J. K., Dickersin, K., Bertizzolo, L., Cowley, T., Doshi, P., Ehmsen, J., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.04.022 · Cited by 63
Self-citationYoung, A. E., Brookes, S. T., Avery, K. N. L., Davies, A., Metcalfe, C., & Blazeby, J. M. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.016 · Cited by 77
“These include outcome, measurement, metric, aggregation of outcomes, and timing of measurement [13,14].”
BackgroundPandis, N., Fleming, P. S., Worthington, H., Dwan, K., & Salanti, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137667 · Cited by 30
“The importance of setting core outcomes common to individual trials and reviews in order to enhance usability and transparency of reports and to reduce variability in outcome specification [29] can also be considered as an important step in reducing discrepancies [30,31].”
BackgroundWeston, J., Dwan, K., Altman, D., Clarke, M., Gamble, C., Schroter, S., Williamson, P., & Kirkham, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010075 · Cited by 16
Metadata
- Authors
- Saldanha, I. J., Dickersin, K., Wang, X., & Li, T.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-10-16
- Volume
- 9
- Issue
- 10
- Pages
- e109400
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2150186451
- Cited by
- 119
- References
- 16
- FWCI
- 2.44
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.