Automating Biomedical Evidence Synthesis: RobotReviewer
Open accessAbstract
, an open-source web-based system that uses machine learning and NLP to semi-automate biomedical evidence synthesis, to aid the practice of Evidence-Based Medicine. RobotReviewer processes full-text journal articles (PDFs) describing randomized controlled trials (RCTs). It appraises the reliability of RCTs and extracts text describing key trial characteristics (e.g., descriptions of the population) using novel NLP methods. RobotReviewer then automatically generates a report synthesising this information. Our goal is for RobotReviewer to automatically extract and synthesise the full-range of structured data needed to inform evidence-based practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For systematic reviews of RCTs, only a few prototype platforms make data extraction technologies available, including ExaCT and RobotReviewer.
“For systematic reviews of RCTs, there exist only a few prototype platforms that make such technologies available (ExaCT [33] and RobotReviewer [12, 34, 35] being among these).”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 30 · Cited by 96
Research works printed but not linked: 17
- Bastian H Glasziou P Chalmers I 2010 Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS medicine 7 9 PMC2943439 20877712
- Boudin F Nie J-Y Dawes M 2010 Positional language models for clinical information retrieval EMNLP 108 115
- Chalmers I Enkin M Keirse MJNC 1993 Preparing and updating systematic reviews of randomized controlled trials of health care Milbank Q 71 3 411 8413069
- Higgins JPT Altman DG Gøtzsche PC Jüni P Moher D Oxman AD Savović J Schulz KF Weeks L Sterne JAC 2011 The Cochrane Collaborations tool for assessing risk of bias in randomised trials BMJ 343 d5928 22008217 PMC3196245
- Kim Y 2014 Convolutional neural networks for sentence classification arXiv preprint arXiv:1408.5882 · arXiv
- Kiritchenko S de Bruijn B Carini S Martin J Sim I 2010 ExaCT: automatic extraction of clinical trial characteristics from journal publications BMC medical informatics and decision making 10 1 56 20920176 PMC2954855
- Kuiper J Marshall IJ Wallace BC Swertz MA 2014 Spá: A web-based viewer for text mining in evidence based medicine ECML-PKDD 452 455 Springer
- Le QV Mikolov T 2014 Distributed representations of sentences and documents ICML 14 1188 1196
- Moss AJ Marcus FI 2017 Changing times in cardiovascular publications: A commentary Am J Med 130 1 11 13 1 27491468
- Sackett DL 1997 Evidence-based medicine: how to practice and teach EBM WB Saunders Company
- Shojania KG Sampson M Ansari MT Ji J Garritty C Rader T Moher D 2007 Updating Systematic Reviews Technical Review No 16 Agency for Healthcare Research and Quality (US) 1 9 20734512
- Smith Richard 2013 The Cochrane collaboration at 20 BMJ 347 f7383 18 12 24353100
- Tricco Andrea C Antony Jesmin Zarin Wasifa Strifler Lisa Ghassemi Marco Ivory John Perrier Laure Hutton Brian Moher David Straus Sharon E 2015 A scoping review of rapid review methods BMC Med 13 224 16 9 PMC4574114 26377409
- Wallace BC Dahabreh IJ Schmid CH Lau J Trikalinos TA 2013 Modernizing the systematic review process to inform comparative effectiveness: tools and methods Journal of Comparative Effectiveness Research (JCER) 2 3 273 282 24236626
- Wallace BC Kuiper J Sharma A Zhu M Marshall IJ 2016 Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision Journal of Machine Learning Research 17 132 1 25 PMC5065023 27746703
- Zaidan O Eisner J Piatko CD 2007 Using “annotator rationales” to improve machine learning for text categorization NAACL 260 267
- Zhang Y Wallace B 2015 A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification arXiv preprint arXiv: 1510.03820 · arXiv
- Authors
- Marshall, I., Kuiper, J., Banner, E., & Wallace, B. C.
- Affiliations
- King's College London (GB)
- Northeastern University (US)
- Venue
- Proceedings of ACL 2017, System Demonstrations
- Published
- 2017
- Volume
- 2017
- Pages
- 7-12
- PMC
- PMC5662138
- OpenAlex
- W2600107025
- Cited by
- 96
- References
- 30
- Citations by year
- 2026: 10 · 2025: 11 · 2024: 18 · 2023: 12 · 2022: 7 · 2021: 14 · 2020: 15 · 2019: 6 · 2018: 1 · 2017: 2
- FWCI
- 17.42
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (5r01lm012086-08)
- Agency for Healthcare Research and Quality (R03-HS025024)
- U.S. National Library of Medicine (R01-LM012086)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- randomized controlled trials, text extraction, natural language processing, machine learning, structured data extraction, evidence-based medicine, open source software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
