RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Objective To develop and evaluate RobotReviewer, a machine learning (ML) system that automatically assesses bias in clinical trials. From a (PDF-formatted) trial report, the system should determine risks of bias for the domains defined by the Cochrane Risk of Bias (RoB) tool, and extract supporting text for these judgments. Methods We algorithmically annotated 12,808 trial PDFs using data from the Cochrane Database of Systematic Reviews (CDSR). Trials were labeled as being at low or high/unclear risk of bias for each domain, and sentences were labeled as being informative or not. This dataset was used to train a multi-task ML model. We estimated the accuracy of ML judgments versus humans by comparing trials with two or more independent RoB assessments in the CDSR. Twenty blinded experienced reviewers rated the relevance of supporting text, comparing ML output with equivalent (human-extracted) text from the CDSR. Results By retrieving the top 3 candidate sentences per document (top3 recall), the best ML text was rated more relevant than text from the CDSR, but not significantly (60.4% ML text rated ‘highly relevant' v 56.5% of text from reviews; difference +3.9%, [−3.2% to +10.9%]). Model RoB judgments were less accurate than those from published reviews, though the difference was <10% (overall accuracy 71.0% with ML v 78.3% with CDSR). Conclusion Risk of bias assessment may be automated with reasonable accuracy. Automatically identified text supporting bias assessment is of equal quality to the manually identified text in the CDSR. This technology could substantially reduce reviewer workload and expedite evidence syntheses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent advances in natural language processing, text mining and machine learning have produced new algorithms that can accurately mimic human endeavour in systematic review activity, faster and more cheaply [5–8].”
From Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR) (2018) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Deep learning and machine learning have been used for assisting with the more complicated and time-intensive tasks of extracting and normalizing data that usually need to be hand curated [12].”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While other tools included in this review do support the manual extraction and appraisal of evidence, RobotReviewer, and Trialstreamer have gone furthest with the integration of automation and in making their tool and code accessible by using a transparent, well-reported, and open-source approach for validation [12,13].”
From A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses (2023) · cites this paper as [12]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Validation studies of RobotReviewer have found that the article bias classifications (i.e. ‘low’ versus ‘high/unclear’ risk of bias) are reasonable but less accurate than those in published Cochrane reviews [12, 15].”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“For systematic reviews of RCTs, there exist only a few prototype platforms that make such technologies available (ExaCT [33] and RobotReviewer [12, 34, 35] being among these).”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [12]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992 · cited 2×
“We combine this with a squared L2 penalty term on the model parameters to form our objective, which describes a linear-kernel soft-margin SVM.14 This objective is shown in Equation 1.”
(2008) · Cited by 3,790
Distant supervision for relation extraction without labeled data
Mintz, M., Bills, S., Snow, R., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690287 · Cited by 2,752 · cited 1×
“Here we take a different approach: to obtain a large corpus of labeled data we use distant supervision, a machine learning methodology that exploits structured data in existing databases in place of direct human supervision.(10,11) Distant supervision involves automatically deriving labels for unlabeled data from exis…”
Frustratingly Easy Domain Adaptation
Daumé, H. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0907.1815 · Cited by 1,383 · cited 1×
“Note that this can be viewed as an instantiation of Daumé’s frustratingly easy domain adaptation approach.(17) Our sentence model is thus trained jointly across all risk of bias domains; the shared component enables information sharing between them.”
Caruana, R. (1998) · Cited by 1,110
Feature hashing for large scale multitask learning
Weinberger, K., Dasgupta, A., Langford, J., Smola, A., & Attenberg, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553516 · Cited by 906 · cited 1×
“This has been shown to work well for large-scale multi-task text classification.(18) To summarize, we use a novel model for risk of bias prediction that 1) jointly makes article- and sentence-level predictions, and 2) borrows strength across the related risk of bias assessment tasks via multi-task learning, for both a…”
Evaluation Measures for Multi-class Subgroup Discovery
Abudawood, T., & Flach, P. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04180-8_20 · Cited by 778
Scaling to very very large corpora for natural language disambiguation
Banko, M., & Brill, E. (2001) · Proceedings of the 39th Annual Meeting on Association for Computational Linguistics - ACL '01 · doi:10.3115/1073012.1073017 · Cited by 675
Cited by
Items 101–200 of 321
Gates, A., Gates, M., Sim, S., Elliott, S. A., Pillay, J., & Hartling, L. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01354-2 · Cited by 28
Systematic Literature Review Langchain Proposed
Asyrofi, R., Dewi, M. R., Luthfi, M. I., & Wibowo, P. A. (2023) · International Electronics Symposium (IES) · doi:10.1109/ies59143.2023.10242497 · Cited by 28
“According to Marshall [5], automated procedures save time and effort for researchers and review teams.”
MethodBilal, J., Riaz, I. B., Kamal, M. U., Elyan, M., Sudano, D. G., & Khan, M. A. (2017) · JCR Journal of Clinical Rheumatology · doi:10.1097/rhu.0000000000000583 · Cited by 24
“Risk-of-Bias Assessment Risk of bias for the included trials was assessed using the RobotReviewer, a Cochrane Collaboration’s tool.(18) Strategy for Data Synthesis We calculated the risk ratio (RR) as an effect measure to compare efficacy and safety between treatment and control arms.”
MethodPanayi, A., Ward, K., Benhadji-Schaff, A., Ibanez-Lopez, A. S., Xia, A., & Barzilay, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02351-w · Cited by 23
“…of SLR development lend themselves to the use of artificial intelligence, such as search-string development, classification of study type, title and abstract screening, extraction of text describing PICO information, risk of bias analysis, data extraction, and data synthesis [7, 9, 11, 35–38].”
MethodAssessing risk of bias in toxicological studies in the era of artificial intelligence
Härtung, T., Hoffmann, S., & Whaley, P. (2025) · Archives of Toxicology · doi:10.1007/s00204-025-03978-5 · Cited by 33
“Several pio-neering applications have demonstrated the feasibility and efficacy of AI in identifying risk of bias in scientific research: – RobotReviewer : RobotReviewer 24 (Marshall et al. 2016) is In general, AI is most capable of identifying bias, 2627 (Soboczenski et al. 2019).”
Burns, J. K., Etherington, C., Cheng‐Boivin, O., & Boet, S. (2021) · Health Information & Libraries Journal · doi:10.1111/hir.12413 · Cited by 35
Martenot, V., Masdeu, V., Cupe, J., Gehin, F., Blanchon, M., Dauriat, J., Horst, A., Renaudin, M., Girard, P., & Zucker, J. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-02085-0 · Cited by 25
“This type of evaluation is common in other systems for assisted literature [32].”
BackgroundDigital Tools to Support the Systematic Review Process: An Introduction
Schmidt, L., Cree, I. A., Campbell, F., & WCT EVI MAP group (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70100 · Cited by 27
Yamada, T., Yoneoka, D., Hiraike, Y., Hino, K., Toyoshiba, H., Shishido, A., Noma, H., Shojima, N., & Yamauchi, T. (2020) · Journal of Medical Internet Research · doi:10.2196/22422 · Cited by 22
“These authors also evaluated the performance of the RobotReviewer in another paper [37] and showed that machine learning could help reviewers to detect sentences or documents containing risk of bias but are not be able to replace manual review by humans yet.”
BackgroundYoga as an Integrative Therapy for Mental Health Concerns: An Overview of Current Research Evidence
Park, C. L., & Slattery, J. M. (2021) · Psychiatry International · doi:10.3390/psychiatryint2040030 · Cited by 38
Brassey, J., Price, C., Edwards, J., Zlabinger, M., Bampoulidis, A., & Hanbury, A. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-111126 · Cited by 22
ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods studyPreprint
Pitre, T., Jassal, T., Talukdar, J. R., Shahab, M., Ling, M., & Zeraatkar, D. (2023) · medRxiv · doi:10.1101/2023.11.19.23298727 · Cited by 27
Eisele‐Metzger, A., Lieberum, J.-L., Toews, M., Siemens, W., Heilmeyer, F., Haverkamp, C., Böhringer, D., & Meerpohl, J. J. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.12 · Cited by 32
“7,8 Exemplary applications that use ML to support steps of systematic reviews include Rayyan, 9 Covidence, 10 and EPPI Reviewer, 11 which are particularly useful to support screening and data extraction, deduplication tools such as Deduklick, 12 and the RobotReviewer 13 for RoB assessment.”
Srinivasan, A., Berkowitz, J. S., Friedrich, N. A., Kivelson, S., & Tatonetti, N. P. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.29418 · Cited by 37
Jap, J., Saldanha, I. J., Smith, B. T., Lau, J., Schmid, C. H., Li, T., & on behalf of the Data Abstraction Assistant Investigators (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1326 · Cited by 20
“14,15 However, to date, most of the data elements that are typically abstracted during systematic reviews have not been explored for automated abstraction.”
BackgroundMethodology reporting improved over time in 176,469 randomized controlled trials
Kilicoglu, H., Jiang, L., Hoang, L., Mayo-Wilson, E., Vinkers, C. H., & Otte, W. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.08.004 · Cited by 19
Appraising the Quality of Systematic Reviews for Age-Related Macular Degeneration Interventions
Downie, L. E., Makrai, E., Bonggotgetsakul, Y., Dirito, L. J., Kristo, K., Pham, M.-A. N., You, M., Verspoor, K. M., & Pianta, M. J. (2018) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2018.2620 · Cited by 19
Post-stroke fatigue: a scoping review
Aali, G., Drummond, A. E., das Nair, R., & Shokraneh, F. (2020) · F1000Research · doi:10.12688/f1000research.22880.1 · Cited by 31
Muller, H., Pachnanda, S., Pahl, F., & Rosenqvist, C. (2022) · International Conference on Applied Artificial Intelligence (ICAPAI) · doi:10.1109/icapai55158.2022.9801564 · Cited by 30
“However, certain NLP and ML-powered software like RobotReviewer [25], is showing promising potential for expansion.”
Damen, J. A., Heus, P., Lamberink, H. J., Tijdink, J. K., Bouter, L. M., Glasziou, P., Moher, D., Otte, W. M., Vinkers, C. H., & Hooft, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.020 · Cited by 20
“Marshall IJ, Kuiper J, Banner E, et al. Automating Biomedical Evidence Synthesis: RobotReviewer.”
MethodInfluentialŠuster, S., Baldwin, T., Lau, J. H., Yepes, A. J., Martínez, D., Otmakhova, Y., & Verspoor, K. M. (2023) · Journal of Medical Internet Research · doi:10.2196/35568 · Cited by 27
“[14].”
Dale, V. H. (2020) · Journal of Perioperative Practice · doi:10.1177/1750458920943375 · Cited by 21
“The assessment was performed by the author, but in order to increase the validity of the assessment the Robot Reviewer automated tool for assessing bias based on the Cochrane methodology was also used (Marshall et al 2015).”
Method(2024) · doi:10.1101/2024.03.31.24305138 · Cited by 18
“The methods range from rule-based methods in early work 27,42 to (semi-)supervised machine learning methods in later work 28,31,35,43 , including deep learning approaches of the recent years 32–34,38 We presented CONSORT-TM, a corpus which represents the most comprehensive annotation of RCT…”
MethodRasalam, R. R., Atherton, J. J., Deed, G., Molloy‐Bland, M., Cohen, N. D., & Sindone, A. P. (2021) · ESC Heart Failure · doi:10.1002/ehf2.13483 · Cited by 19
“A semi-automated risk of bias (RoB) assessment approach was employed, whereby studies were assessed at the study level across four categories (random sequence generation, allocation concealment, blinding of participants and personnel, and blinding of outcomes assessment) by a single author (RR) and the results indepen…”
MethodWilson, E., Cruz, F. A., Maclean, D., Ghanawi, J., McCann, S., Brennan, P. M., Liao, J., Sena, E. S., & Macleod, M. R. (2023) · Clinical Science · doi:10.1042/cs20220594 · Cited by 18
Oami, T., Okada, Y., Sakuraya, M., Fukuda, T., Shime, N., & Nakada, T. (2023) · Journal of Epidemiology · doi:10.2188/jea.je20230227 · Cited by 18
Karagiannis, T., Andreadis, P., Manolopoulos, A., Malandris, K., Avgerinos, I., Karagianni, A., & Τσάπας, Α. (2019) · Diabetic Medicine · doi:10.1111/dme.13939 · Cited by 22
Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad036 · Cited by 14
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
“Other research has focused on automating risk of bias assessment using text classification 35–37 and rhetorical classification of medical abstracts (e.g., Objective, Methods) 38–40 .”
MethodASReview: Open Source Software for Efficient and Transparent Active Learning for Systematic Reviews.Preprint
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2020) · arXiv (Cornell University) · Cited by 21
Pianta, M. J., Makrai, E., Verspoor, K. M., Cohn, T. A., & Downie, L. E. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.07.015 · Cited by 15
Automatic recognition of self-acknowledged limitations in clinical research literature
Kilicoglu, H., Rosemblat, G., Malički, M., & ter Riet, G. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy038 · Cited by 15
Knowledge in Motion: A Comprehensive Review of Evidence-Based Human Kinetics
Ramalho, A. L. G., & Petrica, J. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20116020 · Cited by 18
“For example, if AI algorithms are not properly trained or validated, they may introduce bias or provide unreliable results, and ethical concerns have been raised about the use of AI in healthcare [76].”
BackgroundMethodRapid review methods series: Guidance on the use of supportive software
Affengruber, L., Nußbaumer-Streit, B., Hamel, C., van der Maten, M. M., Thomas, J., Mavergames, C., Spijker, R., & Gartlehner, G. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112530 · Cited by 17
“Covidence, 12 DistillerSR 13 and EPPI-Reviewer 15 require payment for use, Robot Reviewer 50 is free of charge.”
InfluentialDevelopments in the Design, Conduct, and Reporting of Child Health Trials
Baba, A., Aregbesola, A., Caldwell, P. H. Y., Elliott, S. A., Elsman, E. B. M., Fernandes, R. M., Hartling, L., Heath, A., Kelly, L. E., Preston, J., et al. (2024) · PEDIATRICS · doi:10.1542/peds.2024-065799 · Cited by 16
Golinelli, D., Nuzzolese, A. G., Sanmarchi, F., Bulla, L., Mongiovì, M., Gangemi, A., & Rucci, P. (2022) · Information · doi:10.3390/info13030117 · Cited by 16
Wang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13
Swinton, P. A., Shim, J., Pavlova, A. V., Moss, R. A., MacLean, C., Brandie, D., Mitchell, L., Greig, L., Parkinson, E., Brown, V. T., et al. (2023) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2022-001389 · Cited by 21
“The Cochrane’s Risk of Bias tool 40 was selected as a recent review of popular risk of bias tools in tendinopathy management highlighted none were superior 43 and Cochrane’s Risk of Bias tool could be semiautomated with RobotReviewer, 44 a machine learning system software.”
MethodŠuster, S., Baldwin, T., & Verspoor, K. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.04.006 · Cited by 16
Pang, X., Saif‐Ur‐Rahman, K., Berhane, S., Yao, X., Kothari, K. U., Taneri, P. E., Thomas, J., & Devane, D. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111738 · Cited by 17
Scells, H., Zuccon, G., & Koopman, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_27 · Cited by 19
“A wide range of systematic review creation processes have been considered for automation or improvement using semi-automatic techniques [40] , including: query formulation [27, 46, 48] , screening prioritisation [1] [2] [3] 7, 28, 29, 37, 47, 54, 56] , stopping prediction [7, 16, 24] , assessment of bias [33, 43] , am…”
Hacia revisiones de la literatura más eficientes potenciadas por inteligencia artificial
Carbajal-Degante, E., Gutiérrez, M. H., & Mendiola, M. S. (2023) · Investigación en Educación Médica · doi:10.22201/fm.20075057e.2023.47.23526 · Cited by 22
Cohen, A. M., Schneider, J., Fu, Y., McDonagh, M. S., Das, P., Holt, A. W., & Smalheiser, N. R. (2021) · medRxiv · doi:10.1101/2021.07.13.21260468 · Cited by 16
“Yet publication type tagging is complementary to the use of textual or semantic similarity measures to predict relevance for inclusion (e.g., risk of bias [39]), so one would expect that publication type tagging will help improve the performance of other existing machine learning approaches.”
BackgroundA Neural Candidate-Selector Architecture for Automatic Structured Clinical Text Annotation.
Singh, G., Marshall, I. J., Thomas, J., Shawe-Taylor, J., & Wallace, B. C. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132989 · Cited by 11
“This work has entailed jointly extracting the sentences supporting these assessments [20, 21, 24, 32].”
MethodSelf-citationTransformative Automation: AI in Scientific Literature Reviews
Zala, K., Acharya, B., Mashru, M., Palaniappan, D., Gerogiannis, V. C., Kanavos, A., & Karamitsos, I. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01501122 · Cited by 21
“RobotReviewer for experimental research [48].”
BackgroundTang, Y., Xiao, Z., Li, X., Fang, Q., Zhang, Q., Fong, D. Y. T., Lai, F. T. T., Chui, C. S. L., Chan, E. W., & Wong, I. C. K. (2024) · medRxiv · doi:10.1101/2024.03.20.24304572 · Cited by 26
A neuro-symbolic method for understanding free-text medical evidence.
Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab077 · Cited by 11
ACTA A Tool for Argumentative Clinical Trial Analysis
Mayer, T., Cabrio, E., & Villata, S. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/953 · Cited by 21
Design of Meta-Analysis Studies
Macleod, M. R., Tanriver-Ayder, E., Hair, K., & Sena, E. S. (2019) · Handbook of experimental pharmacology · doi:10.1007/164_2019_289 · Cited by 16
Hassanzadeh, H., Nguyen, A. N., & Verspoor, K. M. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103321 · Cited by 13
Uthman, O. A., Al-Khudairy, L., Nduka, C. U., Court, R. A., Mistry, H., Meléndez‐Torres, G. J., Taylor‐Phillips, S., & Clarke, A. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01366-x · Cited by 11
“Risk of bias for the included trials will be assessed using the Robot Reviewer [18].”
MethodŞahin, A., Kara, B. C., & Dirsehan, T. (2025) · SoftwareX · doi:10.1016/j.softx.2025.102198 · Cited by 15
Evaluating the Accuracy of scite, a Smart Citation Index
Bakker, C., Theis-Mahon, N., & Brown, S. J. (2023) · Hypothesis: Research Journal for Health Information Professionals · doi:10.18060/26528 · Cited by 10
MS2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · arXiv · doi:10.48550/arxiv.2104.06486 · Cited by 10
“…the systematic review domain, work has typically focused on information retrieval (Boudin et al., 2010; Ho et al., 2016; Znaidi et al., 2015; Schoot et al., 2020), extracting findings (Lehman et al., 2019; DeYoung et al., 2020; Nye et al., 2020), and quality assessment (Marshall et al., 2015, 2016).”
BackgroundMethodInfluentialTian, Y. Y., Yang, X., Doi, S. A. R., Furuya‐Kanamori, L., Lin, L., Kwong, J. S., & Xu, C. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1761 · Cited by 11
“5 By harnessing the power of machine learning and leveraging a vast dataset, RobotReviewer aims to offer a valuable and accessible tool for RoB assessment.”
Marques‐Cruz, M., Vieira, R. J., Martinho-Dias, D., Barbosa, J. P., Cardoso‐Fernandes, A., Franco-Pêgo, F., Perestrelo, P., Gil‐Mata, S., Taveira‐Gomes, T., Pêgo, J. M., et al. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2024.14 · Cited by 12
“Analogous automated processes already exist for randomised controlled trials, 45 but are yet to be developed for systematic reviews and meta-analyses.”
Moens, M., Nagels, G., Wake, N., & Goudman, L. (2025) · iScience · doi:10.1016/j.isci.2025.113559 · Cited by 13
“36,49,50 The final stage that was evaluated in this study was the creation of the summary table.”
Kolaski, K., Clarke, M. J., & Logan, L. R. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111460 · Cited by 10
A practical guide to living evidence: reducing the knowledge-to-practice gap
Allida, S. M., Hackett, M. L., Lindley, R. I., Hill, K. M., & Ferguson, C. (2024) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvae048 · Cited by 8
Are open science practices in dentistry associated with higher Altmetric scores and citation rates?
Cenci, J., Mendes, F. M., Bouter, L. M., Pereira‐Cenci, T., Acosta, C. D. P., Brondani, B., Moher, D., Huysmans, M. C. D. N. J. M., & Cenci, M. S. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105393 · Cited by 9
Zhou, F., Parrish, R., Afzal, M., Saha, A., Haynes, R. B., Iorio, A., & Lokker, C. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104825 · Cited by 9
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Potrebny, T., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses‐Echavez, J. F., & Berg, R. C. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70048 · Cited by 8
“Semi ‐ automated tools like RobotReviewer [9] have been available to aid RoB assessment for some time, but are based on relatively simple models built using relatively small task ‐ specific training sets.”
Overview of the TAC 2018 Systematic Review Information Extraction Track.
Schmitt, C., Walker, V. R., Williams, A. R., Varghese, A., Ahmad, Y., Rooney, A., & Wolfe, M. S. (2018) · Theory and applications of categories · Cited by 8
Xia, Y., Cai, J., Li, Y., Dou, Z., Zhang, Y., Wu, L., Huang, Z., Xu, S., Sun, J., Liu, Y., et al. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22748 · Cited by 9
“20 Although the balance of precision and recall serves as a criterion for machine ‐ learning ‐ based information extraction methods, errors may result in misleading conclusions in systematic reviews or meta ‐ analyses of clinical RCTs.”
BackgroundLan, M., Cheng, M., Hoang, L., ter Riet, G., & Kilicoglu, H. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104628 · Cited by 15
Machine Learning Methods for Systematic Reviews:: A Rapid Scoping Review.
Roth, S. C., & Wermer‐Colan, A. (2023) · PubMed Central · doi:10.32481/djph.2023.11.008 · Cited by 39
“…of machine learning to assist with the risk of bias assessment was like the method introduced Doi: 10.32481/djph.2023.11.008 by Marshall et al., 48,49 one difference according to Millard et al. 54 is that their team tested its method using full text articles rather than only titles and abstracts.”
MethodInfluentialMethodological information extraction from randomized controlled trial publications: a pilot study.
Hoang, L., Guan, Y., & Kilicoglu, H. (2022) · PubMed Central · Cited by 10
Liu, R., Li, J., Zivkovic, M., & Abbasi, A. (2025) · MIS Quarterly · doi:10.25300/misq/2024/18573 · Cited by 10
“…this paper focuses on automating the data extraction step using ML. Existing solutions are confined to a limited number of PICO elements and article types, and their performance was less satisfactory (DeYoung et al., 2020; Kiritchenko et al., 2010; Marshall et al., 2016; Schmidt et al., 2021).”
Thapa, R. K., Bagchi, A., Uysal, H. Ş., Talukdar, K., Afonso, J. R., & Ramírez‐Campillo, R. (2026) · Sports Medicine · doi:10.1007/s40279-026-02395-7 · Cited by 6
“In addition to manual assessment, a reliable automation tool [41] was used to assess the risk of bias (https:// www. robot revie wer. net/, accessed on 02.07.2024).”
InfluentialDu, H., & Xing, W. (2023) · Distance Education · doi:10.1080/01587919.2022.2150145 · Cited by 9
“Marshall et al. (2016) developed a machine learning system which can automatically identify and assess bias in clinical trials.”
BackgroundWang, S., Zhang, K., & Du, J. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2024-101017 · Cited by 7
Hatakeyama, Y., Seto, K., Hirata, K., Onishi, R., Matsumoto, K., & Hasegawa, T. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-07492-7 · Cited by 7
“Additionally, some tools for efficient systematic review have been developed and available [34, 35].”
MethodSmalheiser, N. R., & Cohen, A. M. (2018) · Data and Information Management · doi:10.2478/dim-2018-0004 · Cited by 6
Hoanga, L., Jiang, L., & Kilicoglu, H. (2022) · PubMed Central · Cited by 7
Aguirre, A. M. G., & Figueroa-Vallejo, C. J. (2022) · Revista de Investigación Desarrollo e Innovación · doi:10.19053/20278306.v12.n2.2022.15271 · Cited by 9
Jiang, L., Vorland, C. J., Ying, X., Brown, A. W., Menke, J. D., Hong, G., Lan, M., Mayo‐Wilson, E., & Kilicoglu, H. (2025) · Scientific Data · doi:10.1038/s41597-025-04629-1 · Cited by 11
“…sentences in abstracts or full-text articles by PICO elements (Population, Intervention, Comparator, and Outcome) for article screening 18–21 , extracting PICO-related or other methodological terms to aid data extraction 22–28 , and classifying text for automated risk-of-bias assessment 29–31 .”
Hoang, L., Jiang, L., & Kilicoglu, H. (2021) · doi:10.1101/2021.09.14.21263586 · Cited by 5
“For example, risk of bias judgements in the Cochrane database of systematic reviews were used to automatically label sentences in RCT publications and train models for assessing risk of bias in the publications 15 .”
BackgroundMethodRoBIn: A Transformer-based model for risk of bias inference with machine reading comprehension
Dias, A. C., Moreira, V. P., & Comba, J. L. D. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104819 · Cited by 12
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02631-0 · Cited by 5
“For example, RobotReviewer [12] was developed to assess risk of bias in clinical trials.”
AI Tools for Automating Systematic Literature Reviews
Mikriukov, A., Senokosov, A., Succi, G., Tormasov, A. G., Plaksin, Y., Trofimova, E., & Sitnikov, V. (2025) · International Conference on Software Engineering and Computer Applications · doi:10.1145/3747912.3747962 · Cited by 12
“As a result, it is often difficult for researchers to combine the best available tools for screening, data extraction, and synthesis into a single, consistent workflow [10, 14].”
Can Systematic Reviews Be Automated?
Laynor, G. (2022) · Journal of Electronic Resources in Medical Libraries · doi:10.1080/15424065.2022.2113350 · Cited by 5
“For example, RobotReviewer (http://www.robotreviewer.net) uses machine learning to appraise clinical trials for risk of bias, what can otherwise be a very time-intensive step in systematic review projects (Marshall et al., 2016).”
On How Cognitive Computing Will Plan Your Next Systematic Review
Badami, M., Báez, M., Zamanirad, S., & Kang, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-76352-7_32 · Cited by 5
Opportunities for Computer Support for Systematic Reviewing - A Gap Analysis
Hoang, L., & Schneider, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-78105-1_40 · Cited by 5
“Being able to check machine results in a natural way (such as RobotReviewer’s inline annotation of extracted data [16]) could help reviewers gain trust and identify further development needs for specific software.”
Fast‐and‐frugal decision tree for the rapid critical appraisal of systematic reviews
Lorenz, R. C., Jenny, M. A., Jacobs, A., & Matthias, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1754 · Cited by 5
“37,38 The Screening FFT may be used to conduct OoR that includes only non-critically low SRs.”
BASIL DB: bioactive semantic integration and linking database
Jackson, D., Groth, P., & Harmouch, H. (2025) · Journal of Biomedical Semantics · doi:10.1186/s13326-025-00336-3 · Cited by 6
“We use NLP tools, including a large language model (LLM), RobotReviewer [18], MetaMap [19], and Table 1 Overview of key data sources utilized in the construction process, sorted alphabetically.”
Perspective: Towards Automated Tracking of Content and Evidence Appraisal of Nutrition Research
Yang, C., Hawwash, D., Baets, B. D., Bouwman, J., & Lachat, C. K. (2020) · Advances in Nutrition · doi:10.1093/advances/nmaa057 · Cited by 5
Lake, J., Lawrence, K. A., Alonso, E. M., Gonzales, V., & LaFleur, J. (2021) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12423 · Cited by 4
A Study of Explainability Features to Scrutinize Faceted Filtering Results
Qu, J., Arguello, J., & Wang, Y. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482409 · Cited by 9
“highlight sentences from biomedical articles that may indicate a biased clinical trial [29].”
BackgroundTimely, Efficient, and Living Systematic Reviews: Opportunities in International Development
Snilstveit, B., Stevenson, J., Shemilt, I., Clarke, M. J., Jiménez, E., & Thomas, J. (2018) · doi:10.51744/cip7 · Cited by 6
“, 2010) Critical appraisal Automated risk of bias assessment (Marshall et al., 2015; Millard et al., 2016) Report production Automated write up based on extracted data (Torres and Adams 2017)”
BackgroundMethodInfluentialNivolumab for adult individuals with Hodgkin lymphoma (an exemplar rapid review using RobotReviewer)
Skoetz, N., Goldkuhle, M., Gartlehner, G., Monsef, I., Dahm, P., Glossmann, J., Engert, A., & von Tresckow, B. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012556 · Cited by 4
Gates, A., Gates, M., Sim, S., Elliott, S. A., Pillay, J., & Hartling, L. (2021) · doi:10.23970/ahrqepcmethodscreatingefficiencies · Cited by 3
The FAIR database: facilitating access to public health research literature
Zhao, Z., Thomas, J., Kell, G., Stansfield, C., Clowes, M., Graziosi, S., Brunton, J., Marshall, I. J., & Stevenson, M. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae139 · Cited by 3
Self-citationdos Santos, Á. O., Belo, V. S., Machado, T. M., & da Silva, E. S. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-113123 · Cited by 5
“The literature indicates that RegEx combined with machine learning enhances clinical text classification, 29 and Marshall et al 30 demonstrated good accuracy in automating risk- of-bias analysis using RobotRe-viewer.”
Oami, T., Okada, Y., & Nakada, T. (2026) · Hong Kong Journal of Emergency Medicine · doi:10.1002/hkj2.70085 · Cited by 3
Maleki, F., & Karami, M. (2024) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12665 · Cited by 4
Clinical Research Informatics: Supporting the Research Study Lifecycle
Johnson, S. B. (2017) · Yearbook of Medical Informatics · doi:10.1055/s-0037-1606503 · Cited by 6
Assessing Review Reports of Scientific Articles: A Literature Review
Sizo, A., Lino, A. D. P., & Rocha, Á. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-77703-0_14 · Cited by 3
DeSpin: a prototype system for detecting spin in biomedical publications
Koroleva, A., Kamath, S., Bossuyt, P. M., & Paroubek, P. (2020) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2020.bionlp-1.5 · Cited by 5
Controversial Trials First: Identifying Disagreement Between Clinical Guidelines and New Evidence.
Borchert, F., Meister, L., Langer, T. G., Follmann, M., Arnrich, B., & Schapranow, M.-P. (2021) · PubMed · Cited by 5
Modernizing evidence synthesis for evidence-based medicine
Saldanha, I. J., Adam, G. P., Schmid, C. H., Trikalinos, T. A., & Konnyu, K. J. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00006-1 · Cited by 4
Metadata
- Authors
- Marshall, I. J., Kuiper, J., & Wallace, B. C.
- Affiliations
- King's College London (GB)
- University Medical Center Groningen (NL)
- The University of Texas at Austin (US)
- Corresponding author
- Marshall, Iain J
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-06-22
- Volume
- 23
- Issue
- 1
- Pages
- 193-201
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC4713900
- OpenAlex
- W2184378182
- Cited by
- 317
- References
- 22
- Citations by year
- 2026: 34 · 2025: 39 · 2024: 39 · 2023: 32 · 2022: 23 · 2021: 41 · 2020: 49 · 2019: 13 · 2018: 23 · 2017: 16 · 2016: 5 · 2015: 2
- FWCI
- 13.97
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, cc-by-nc)
- Research Portal (King's College London) (submittedVersion, cc-by-nc)
- HighWire Press Open Archive (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by-nc)
- Funders
- Brown University
- King's College Hospital NHS Foundation Trust
- National Institute for Health and Care Research
- King's College London
- Monash University
- NIHR Collaboration for Leadership in Applied Health Research and Care South London
- Medical Research Council (MR/N015185/1)
- Institute of Education, University of London
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk of bias assessment, clinical trials, Cochrane risk of bias tool, multi-task learning, Cochrane Database of Systematic Reviews, evidence synthesis
- MeSH terms
- Machine Learning, Algorithms, Clinical Trials as Topic, Natural Language Processing, Review Literature as Topic, Bias, Peer Review, Research, Databases as Topic, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.