Automating Risk of Bias Assessment for Clinical Trials
Open accessAbstract
Systematic reviews, which summarize the entirety of the evidence pertaining to a specific clinical question, have become critical for evidence-based decision making in healthcare. But such reviews have become increasingly onerous to produce due to the exponentially expanding biomedical literature base. This study proposes a step toward mitigating this problem by automating risk of bias assessment in systematic reviews, in which reviewers determine whether study results may be affected by biases (e.g., poor randomization or blinding). Conducting risk of bias assessment is an important but onerous task. We thus describe a machine learning approach to automate this assessment, using the standard Cochrane Risk of Bias Tool which assesses seven common types of bias. Training such a system would typically require a large labeled corpus, which would be prohibitively expensive to collect here. Instead, we use distant supervision, using data from the Cochrane Database of Systematic Reviews (a large repository of systematic reviews), to pseudoannotate a corpus of 2200 clinical trial reports in PDF format. We then develop a joint model which, using the full text of a clinical trial report as input, predicts the risks of bias while simultaneously extracting the text fragments supporting these assessments. This study represents a step toward automating or semiautomating extraction of data necessary for the synthesis of clinical trials.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data extraction systems have been created that can automatically retrieve unstructured information on the trial PICO elements and risk of bias.
“Data extraction systems have been created that can automatically retrieve unstructured information on the trial PICO elements,10–12 and risk of bias.13–16”
From Trialstreamer: A living, automatically updated database of clinical trial reports (2020) · cites this paper as [13]
The prior work predicted whether studies' results might be at risk of bias using the first version of the Cochrane Risk of Bias tool.
“In this prior work, we predicted whether studies’ results might be at risk of bias using the first version of the Cochrane Risk of Bias tool”
From Trialstreamer: A living, automatically updated database of clinical trial reports (2020) · cites this paper as [13]
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138 · see citing works
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582 · see citing works
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,957 · see citing works
Opinion mining and sentiment analysis
Pang, B., & Lee, L. (2008) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000011 · Cited by 6,372 · see citing works
Distant supervision for relation extraction without labeled data
Mintz, M. D., Bills, S., Snow, R., & Jurafsky, D. (2009) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/1690219.1690287 · Cited by 2,799 · see citing works
The Nature of Statistical Learning
Vapnik, V. N. (1995) · Medical Entomology and Zoology · Cited by 2,295 · see citing works
How to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436 · see citing works
A Survey of Opinion Mining and Sentiment Analysis
Liu, B., & Zhang, L. (2012) · Springer eBooks · doi:10.1007/978-1-4614-3223-4_13 · Cited by 1,356 · see citing works
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · see citing works
A survey of current work in biomedical text mining
Cohen, A. M. (2005) · Briefings in Bioinformatics · doi:10.1093/bib/6.1.57 · Cited by 767 · see citing works
Savović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668 · see citing works
A Joint Model of Text and Aspect Ratings for Sentiment Summarization
Titov, I., & McDonald, R. (2008) · Cited by 630 · see citing works
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · see citing works
Self-citationAn Unsupervised Aspect-Sentiment Model for Online Reviews
Brody, S., & Elhadad, N. (2010) · Cited by 501 · see citing works
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · see citing works
Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Krebs Seida, J., & Klassen, T. P. (2009) · BMJ · doi:10.1136/bmj.b4012 · Cited by 348 · see citing works
Systematic Reviews: Critical Links in the Great Chain of Evidence
Mulrow, C. D., Cook, D. J., & Davidoff, F. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-126-5-199703010-00008 · Cited by 262 · see citing works
Automatic classification of sentences to support Evidence Based Medicine.
Kim, S. N., Martinez, D., Cavedon, L., & Yencken, L. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s2-s5 · Cited by 214 · see citing works
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · see citing works
Integrated Annotation for Biomedical Information Extraction
Seth Kulick, A., Ann Bies, A., Liberman, M., Mandel, M. A., McDonald, R., Palmer, M. S., Schein, A. I., Ungar, L., Winters, S., & White, P. (2004) · North American Chapter of the Association for Computational Linguistics · Cited by 186 · see citing works
Frustratingly Easy Semi-Supervised Domain Adaptation
Daumé, H., Kumar, A., & Saha, A. (2010) · Cited by 170 · see citing works
Shemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170 · see citing works
Czoski‐Murray, C., Jones, M. L., McCabe, C. J., Claxton, K., Oluboyede, Y., Roberts, J., Nicholl, J., Rees, A., Reilly, C., Young, D., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16500 · Cited by 161 · see citing works
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131 · see citing works
Active learning for biomedical citation screening
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835829 · Cited by 123 · see citing works
Self-citationEnd-to-End Relation Extraction Using Distant Supervision from External Semantic Repositories
Nguyen, T. V. T., & Moschitti, A. (2011) · Institutional Research Information System (Università degli Studi di Trento) · Cited by 74 · see citing works
A Web-based archive of systematic review data
Ip, S., Hadar, N., Keefe, S., Parkin, C., Iovin, R. C., Balk, E. M., & Lau, J. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-15 · Cited by 66 · see citing works
Hopewell, S., Boutron, I., Altman, D. G., & Ravaud, P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003342 · Cited by 66 · see citing works
Towards Automating the Initial Screening Phase of a Systematic Review
Bekhuis, T., & Demner‐Fushman, D. (2010) · Studies in health technology and informatics · doi:10.3233/978-1-60750-588-4-146 · Cited by 61 · see citing works
Automated recognition of malignancy mentions in biomedical literature
Jin, Y., McDonald, R., Lerman, K., Mandel, M. A., Carroll, S., Liberman, M., Pereira, F. C., Winters, R. S., & White, P. S. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-492 · Cited by 61 · see citing works
Automated information extraction of key trial design elements from clinical trial publications.
(2008) · PubMed Central · Cited by 54 · see citing works
Modernizing the systematic review process to inform comparative effectiveness: tools and methods
Wallace, B., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.17 · Cited by 51 · see citing works
Self-citationAutomatic Summarization of Results from Clinical Trials
Summerscales, R. L., Argamon, S., Bai, S., Hupert, J., & Schwartz, A. (2011) · 2011 IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm.2011.72 · Cited by 49 · see citing works
Mining multi-faceted overviews of arbitrary topics in a text collection
Ling, X., Mei, Q., Zhai, C., & Schatz, B. R. (2008) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1401890.1401952 · Cited by 41 · see citing works
Generating gene summaries from biomedical literature: A study of semi-structured summarization
Ling, X., Jiang, J., He, X., Mei, Q., Zhai, C., & Schatz, B. R. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.01.018 · Cited by 40 · see citing works
Automating risk of bias assessment for clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. (2014) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/2649387.2649406 · Cited by 28 · see citing works
Self-citationvan Valkenhoef, G., Tervonen, T., de Brock, B., & Hillege, H. L. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-95 · Cited by 23 · see citing works
#CochraneTech: Technology and the Future of Systematic Reviews
Elliott, J., Sim, I., Thomas, J. A., Owens, N., Dooley, G., Riis, J., Wallace, B., Thomas, J., Noel-Storr, A. H., Gabriel Rada, G., et al. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000091 · Cited by 21 · see citing works
Self-citationAUTOMATIC SUMMARIZATION OF CLINICAL ABSTRACTS FOR EVIDENCE-BASED MEDICINE
Summerscales, R. (2013) · Digital Commons - Andrews University (Andrews University) · Cited by 10 · see citing works
Spá: A Web-Based Viewer for Text Mining in Evidence Based Medicine
Kuiper, J., Marshall, I. J., Wallace, B., & Swertz, M. A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44845-8_33 · Cited by 10 · see citing works
Self-citationCrowdsourcing a Comprehensive Clinical Trial Repository
van Valkenhoef, G., & Kuiper, J. (2014) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
1 of the 43 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · see citing works
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 351 · see citing works
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317 · see citing works
Self-citationAutomating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287 · see citing works
Rationale-Augmented Convolutional Neural Networks for Text Classification
Zhang, Y., Marshall, I., & Wallace, B. C. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1076 · Cited by 249 · see citing works
Self-citationBarbosa, A. D. S., Crispim, M. C., da Silva, L. B., Morioka, S. N., & de Souza, V. F. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01919-0 · Cited by 206 · see citing works
(2016) · PubMed · Cited by 134 · see citing works
Self-citationCréquit, P., Trinquart, L., Yavchitz, A., & Ravaud, P. (2016) · BMC Medicine · doi:10.1186/s12916-016-0555-0 · Cited by 125 · see citing works
Frampton, G. K., Whaley, P., Bennett, M. G., Bilotta, G. S., Dorne, J. L. C. M., Eales, J. F., James, K. L., Kohl, C., Land, M., Livoreil, B., et al. (2022) · Environmental Evidence · doi:10.1186/s13750-022-00264-0 · Cited by 103 · see citing works
Trialstreamer: A living, automatically updated database of clinical trial reports
Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa163 · Cited by 88 · see citing works
Self-citationImproving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62 · see citing works
Machine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65 · see citing works
Artificial intelligence bias in medical system designs: a systematic review
Kumar, A., Aelgani, V., Vohra, R., Gupta, S. K., Bhagawati, M., Paul, S., Saba, L., Suri, N., Khanna, N. N., Laird, J. R., et al. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16029-x · Cited by 65 · see citing works
dos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 65 · see citing works
Automating Biomedical Evidence Synthesis: RobotReviewer
Marshall, I., Kuiper, J., Banner, E., & Wallace, B. C. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4002 · Cited by 96 · see citing works
Self-citationSoboczenski, F., Trikalinos, T. A., Kuiper, J., Bias, R. G., Wallace, B. C., & Marshall, I. J. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0814-z · Cited by 53 · see citing works
Self-citationExtractive summarization of clinical trial descriptions
Gulden, C., Kirchner, M., Schüttler, C., Hinderer, M., Kampf, M., Prokosch, H., & Toddenroth, D. (2019) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2019.05.019 · Cited by 41 · see citing works
Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Jaspers, S., De Troyer, E., & Aerts, M. (2018) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2018.en-1427 · Cited by 36 · see citing works
Arroyave, W., Mehta, S., Guha, N., Schwingl, P., Taylor, K. W., Glenn, B. S., Radke, E., Vilahur, N., Carreón, T., Nachman, R., et al. (2020) · Journal of Exposure Science and Environmental Epidemiology · doi:10.1038/s41370-020-0228-0 · Cited by 31 · see citing works
Sundaram, G., & Berleant, D. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-3243-6_7 · Cited by 34 · see citing works
Šuster, S., Baldwin, T., & Verspoor, K. M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1749 · Cited by 30 · see citing works
Making research articles fit for purpose: structured reporting of key methods and findings
Altman, D. G. (2015) · Trials · doi:10.1186/s13063-015-0575-7 · Cited by 28 · see citing works
Créquit, P., Trinquart, L., & Ravaud, P. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011841 · Cited by 28 · see citing works
Digital Tools to Support the Systematic Review Process: An Introduction
Schmidt, L., Cree, I. A., Campbell, F., & WCT EVI MAP group (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70100 · Cited by 22 · see citing works
Yamada, T., Yoneoka, D., Hiraike, Y., Hino, K., Toyoshiba, H., Shishido, A., Noma, H., Shojima, N., & Yamauchi, T. (2020) · Journal of Medical Internet Research · doi:10.2196/22422 · Cited by 24 · see citing works
Query-oriented evidence extraction to support evidence-based medicine practice
Sarker, A., Mollá, D., & Paris, C. L. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.11.010 · Cited by 21 · see citing works
Šuster, S., Baldwin, T., Lau, J. H., Yepes, A. J., Martínez, D., Otmakhova, Y., & Verspoor, K. M. (2023) · Journal of Medical Internet Research · doi:10.2196/35568 · Cited by 19 · see citing works
Is it time for computable evidence synthesis?
Dunn, A. G., & Bourgeois, F. T. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa035 · Cited by 16 · see citing works
Determining the biases and consistencies in the evidence for conservation
Christie, A. P. (2021) · Apollo (University of Cambridge) · doi:10.17863/cam.68583 · Cited by 15 · see citing works
Šuster, S., Baldwin, T., & Verspoor, K. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.04.006 · Cited by 14 · see citing works
Mishra, A. K. (2020) · Wiley eBooks · doi:10.1002/9781119602927.ch2 · Cited by 12 · see citing works
A Neural Candidate-Selector Architecture for Automatic Structured Clinical Text Annotation.
Singh, G., Marshall, I. J., Thomas, J., Shawe-Taylor, J., & Wallace, B. C. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132989 · Cited by 11 · see citing works
Self-citationA distantly supervised dataset for automated data extraction from diagnostic studies.
Norman, C., Leeflang, M., Spijker, R., Kanoulas, E., & Névéol, A. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5012 · Cited by 9 · see citing works
Liu, S., Bourgeois, F. T., & Dunn, A. G. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1545 · Cited by 9 · see citing works
MS2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · arXiv · doi:10.48550/arxiv.2104.06486 · Cited by 10 · see citing works
Human-annotated rationales and explainable text classification: a survey
Herrewijnen, E., Nguyen, D., Bex, F., & van Deemter, K. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1260952 · Cited by 8 · see citing works
Zhou, F., Parrish, R., Afzal, M., Saha, A., Haynes, R. B., Iorio, A., & Lokker, C. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104825 · Cited by 8 · see citing works
Machine Learning Methods for Systematic Reviews:: A Rapid Scoping Review.
Roth, S. C., & Wermer‐Colan, A. (2023) · PubMed Central · doi:10.32481/djph.2023.11.008 · Cited by 7 · see citing works
Appraising UMLS Coverage for Summarizing Medical Evidence.
ShafieiBavani, E., Ebrahimi, M., Wong, R. K., & Chen, F. (2016) · UNSWorks (UNSW Sydney) · Cited by 6 · see citing works
RoBIn: A Transformer-based model for risk of bias inference with machine reading comprehension
Dias, A. C., Moreira, V. P., & Comba, J. L. D. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104819 · Cited by 6 · see citing works
Christie, A. P., Amano, T., Martin, P. A., Shackelford, G. E., Simmons, B. I., & Sutherland, W. J. (2022) · Journal of Applied Ecology · doi:10.1111/1365-2664.14154 · Cited by 5 · see citing works
Pereira, R. G., de Castro, G. Z., Azevedo, P. S., Torres, L., Zuppo, I. D. F., Rocha, T., & Guerra, A. A. (2020) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms49503.2020.00008 · Cited by 4 · see citing works
Automating Biomedical Evidence Synthesis: Recent Work and Directions Forward.
Wallace, B. (2018) · Cited by 4 · see citing works
Self-citationChen, D., Li, P., Khoshkish, E., Lee, S., Ning, T., Tahir, U., Wong, H. C. Y., Lee, M. S. F., & Raman, S. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf223 · Cited by 6 · see citing works
Extracting experimental parameter entities from scientific articles
Farnsworth, S., Gurdin, G., Vargas, J., Mulyar, A., Lewinski, N. A., & McInnes, B. T. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103970 · Cited by 4 · see citing works
What Does the Evidence Say? Models to Help Make Sense of the Biomedical Literature
Wallace, B. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/899 · Cited by 3 · see citing works
Self-citationTeperikidis, E., Karakasis, P., Patoulias, D., Boulmpou, A., & Kouzoukas, D. E. (2024) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2024.10.017 · Cited by 2 · see citing works
Strategies to Reduce Pain After Laparoscopic Gynecologic Surgery
Asgari, Z., Hosseini, R., Rastad, H., & Hosseini, L. (2018) · Surgical Laparoscopy Endoscopy & Percutaneous Techniques · doi:10.1097/sle.0000000000000576 · Cited by 2 · see citing works
Zero-shot interpretable biomedical literature appraisal with generative large language models
Zhou, F., Afzal, M., Saha, A., Parrish, R., Haynes, R. B., Iorio, A., & Lokker, C. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag043 · Cited by 1 · see citing works
First Steps Towards a Risk of Bias Corpus of Randomized Controlled Trials
Dhrangadhariya, A., Hilfiker, R., Sattelmayer, M., Giacomino, K., Caliesch, R., Elsig, S., Naderi, N., & Müller, H. (2023) · Studies in health technology and informatics · doi:10.3233/shti230210 · Cited by 1 · see citing works
Biondi‐Zoccai, G., Torre, G. L., Roever, L., & D’Ascenzo, F. (2016) · Springer eBooks · doi:10.1007/978-3-319-25655-9_8 · Cited by 1 · see citing works
Testa, L., & Bollati, M. (2018) · Springer eBooks · doi:10.1007/978-3-319-78966-8_8
Classifying Positive Results in Clinical Psychology Using Natural Language Processing. Preprint.Preprint
Schiekiera, L., Niemeyer, H., & Diederichs, J. (2024) · doi:10.31234/osf.io/uxyzh
Facilitating critical appraisal of evidence using automation tools
Bannach‐Brown, A., Hijlkema, N., van Beuningen, N., Alkema, W. B. L., Hooijmans, C. R., Wever, K. E., Macleod, M. R., Maksym, O., Novotny, T., Niccolai, L., et al. (2025) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2025.en-9515
Assessment of Bias in Clinical Trials with LLMs Using ROBUST-RCT: A Feasibility StudyPreprint
Vidor, P. R., Casiraghi, Y., de Souza, A. M., & Schmidt, M. I. (2025) · medRxiv · doi:10.1101/2025.08.12.25333520
Beyond the p < 0.05 trap: the adaptive integrity model for preventing and detecting P-hacking
Affognon, D. A. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1675991
Vidor, P. R., Casiraghi, Y., de Souza, A. M., & Schmidt, M. I. (2026) · Scientific Reports · doi:10.1038/s41598-026-44303-z · Cited by 1 · see citing works
Huang, Y. T., Chang, K.-C., Kuo, Y., Tai, C., & Yang, C. (2026) · Journal of orthopaedic surgery · doi:10.1177/10225536261459518
Automated tools for evidence quality assessment: a scoping review
Huang, J., Deng, X., Zhou, L., Tao, J., Liang, C., Yang, K., & Li, X. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02868-3
The silence of the labs: Ethical oversight in metabolomics research
Badia, J., Llambrich, M., Rubio, R. L., Saggion, H., & Cumeras, R. (2026) · PLoS ONE · doi:10.1371/journal.pone.0346284
- Authors
- Marshall, I. J., Kuiper, J., & Wallace, B. C.
- Affiliations
- King's College London (GB)
- The University of Texas at Austin (US)
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2015-07
- Volume
- 19
- Issue
- 4
- Pages
- 1406-1412
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W2116485318
- Cited by
- 59
- References
- 43
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 4 · 2023: 8 · 2022: 2 · 2021: 4 · 2020: 6 · 2019: 4 · 2018: 4 · 2017: 2 · 2016: 6 · 2015: 6
- FWCI
- 5.47
- Type
- journal article
- Language
- en
- Open access
- green
- kclpure.kcl.ac.uk
- Locations
- IEEE Journal of Biomedical and Health Informatics (publishedVersion)
- PubMed (publishedVersion)
- Research Portal (King's College London) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- risk of bias assessment, clinical trials, systematic reviews, distant supervision, Cochrane risk of bias tool, clinical trial reports, text extraction, evidence synthesis
- MeSH terms
- Clinical Trials as Topic, Humans, Medical Informatics, Natural Language Processing, Bias, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
