Automating risk of bias assessment for clinical trials
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
In medicine, the publication of clinical trials now far out-paces clinicians' ability to read them. Systematic reviews, which aim to summarize the entirety of the available evidence on a specific clinical question, have therefore become the linchpin of evidence-based decision making. A key task in systematic reviews is determining whether the results of included studies may be affected by biases, e.g., poor randomization or blinding. This is called risk of bias assessment and is now standard practice. Standardized tools are used to perform these assessments; a notable example being the Cochrane risk of bias tool, which covers seven different types of potential biases and involves researchers extracting sentences from articles to support their bias assessments. These assessments are crucial in interpreting published evidence, but due to the exponential growth of the biomedical literature base, manually assessing the risk of bias in clinical trials has grown burdensome for clinical researchers. Aiming to mitigate this workload, we explore automating risk of bias assessment. We demonstrate that systematic reviews may be used to distantly supervise text mining models, obviating the need for manually annotated clinical trial reports. Specifically, we leverage data from the Cochrane Database of Systematic Reviews (a large repository of systematic reviews), and link clinical trial reports to structured data from the same studies found in CDSR to produce a pseudo-annotated labeled corpus. We then develop a joint model which, using (the PDF of) a clinical trial report as input, predicts the risks of bias in each of the aforementioned seven areas while simultaneously extracting the text fragments supporting these assessments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“Already the generation of primary evidence is outpacing our ability to synthesize it given pragmatic resource constraints [1, 15].”
BackgroundSavović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“has previously been recognized that machine learning tools can assist abstract screening [15]–[18], data extraction [19]–[23], summarization [24], [25], and scoping [26].”
BackgroundRisk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Krebs Seida, J., & Klassen, T. P. (2009) · BMJ · doi:10.1136/bmj.b4012 · Cited by 348 · cited 1×
“Indeed, a validation study of the Cochrane risk of bias tool found wide variations in judgements by different researchers in all domains, with the selective reporting domain showing the least agreement ( κ =0.13, 95% CI -0.05 to 0.31) [4].”
MethodInfluentialAutomatic classification of sentences to support Evidence Based Medicine.
Kim, S. N., Martinez, D., Cavedon, L., & Yencken, L. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s2-s5 · Cited by 214 · cited 1×
“has previously been recognized that machine learning tools can assist abstract screening [15]–[18], data extraction [19]–[23], summarization [24], [25], and scoping [26].”
BackgroundExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188
Shemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131
The automation of systematic reviews
Tsafnat, G., Dunn, A., Glasziou, P., & Coiera, E. (2013) · BMJ · doi:10.1136/bmj.f139 · Cited by 131
Active learning for biomedical citation screening
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835829 · Cited by 123
Self-citation
Research works printed but not linked: 12
- S. Brody and N. Elhadad . An unsupervised aspect-sentiment model for online reviews . In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 804 -- 812 . Association for Computational Linguistics , 2010 . S. Brody and N. Elhadad. An unsupervised aspect-sentiment model for online reviews. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 804--812. Association for Computational Linguistics, 2010.
- Evidence-Based Medicine Working Group . Evidence-based medicine. A new approach to teaching the practice of medicine. 268(17):2420--2425 , 1992 . Evidence-Based Medicine Working Group. Evidence-based medicine. A new approach to teaching the practice of medicine. 268(17):2420--2425, 1992.
- J. Higgins , D. Altman , P. Gotzsche , P. Juni , D. Moher , A. Oxman , J. Savovic , K. Schulz , L. Weeks , and J. Sterne . The Cochrane Collaboration's tool for assessing risk of bias in randomised trials . BMJ , 343 , 2011 . J. Higgins, D. Altman, P. Gotzsche, P. Juni, D. Moher, A. Oxman, J. Savovic, K. Schulz, L. Weeks, and J. Sterne. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ, 343, 2011.
- 10.1007/978-1-4614-3223-4_13 · DOI · Paper page
- 10.5555/1690219.1690287 · doi:10.5555/1690219.1690287
- T. Nguyen and A. Moschitti . End-to-end relation extraction using distant supervision from external semantic repositories . In Proceedings of the Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers , volume 2 , pages 277 -- 282 . Association for Computational Linguistics , 2011 . T. Nguyen and A. Moschitti. End-to-end relation extraction using distant supervision from external semantic repositories. In Proceedings of the Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers, volume 2, pages 277--282. Association for Computational Linguistics, 2011.
- 10.1561/1500000011 · DOI · Paper page
- The Cochrane Collaboration . The Cochrane Database of Systematic Reviews , 2014 . The Cochrane Collaboration. The Cochrane Database of Systematic Reviews, 2014.
- I. Titov and R. McDonald . A joint model of text and aspect ratings for sentiment summarization . In Proceedings of the Association for Computational Linguistics (ACL) , volume 51 , page 61801 , 2008 . I. Titov and R. McDonald. A joint model of text and aspect ratings for sentiment summarization. In Proceedings of the Association for Computational Linguistics (ACL), volume 51, page 61801, 2008.
- 10.1186/1472-6947-12-95 · DOI
- 10.1007/978-1-4757-3264-1 · DOI · Paper page
- 10.2217/cer.13.17 · DOI · Paper page
Cited by
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“The use of automation to assist in study selection is possibly the most advanced of all the areas where automation in systematic reviews is being developed; but others range from writing sections of the report, formulating the review question and automated data extraction and quality assessment [90-93].”
BackgroundAssessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
Searching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
“Marshall et al. (2014) [27] Risk of bias concerning sequence generation, allocation concealment and blinding 2200 clinical trial reports Soft-margin SVM for a joint model of risk of bias prediction and supporting sentence extraction Sentence Full text For sentence identification: F-score of 0.56,…”
BackgroundInfluentialRecommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221
Metadata
- Authors
- Marshall, I. J., Kuiper, J., & Wallace, B. C.
- Venue
- Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics
- Publisher
- ACM
- Published
- 2014-09-20
- Pages
- 88-95
- OpenAlex
- W1993701005
- Cited by
- 28
- References
- 12
- FWCI
- 9.60
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automating risk of bias assessment for clinical trials. https://app.cochat.ai/papers/10.1145/2649387.2649406
- The paper
- Marshall, I. J., Kuiper, J., & Wallace, B. C. (2014). Automating risk of bias assessment for clinical trials. Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. https://doi.org/10.1145/2649387.2649406
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.