Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine
Open accessThe DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Abstract
Objective: For many literature review tasks, including systematic review (SR) and other aspects of evidence-based medicine, it is important to know whether an article describes a randomized controlled trial (RCT). Current manual annotation is not complete or flexible enough for the SR process. In this work, highly accurate machine learning predictive models were built that include confidence predictions of whether an article is an RCT. Materials and Methods: The LibSVM classifier was used with forward selection of potential feature sets on a large human-related subset of MEDLINE to create a classification model requiring only the citation, abstract, and MeSH terms for each article. Results: The model achieved an area under the receiver operating characteristic curve of 0.973 and mean squared error of 0.013 on the held out year 2011 data. Accurate confidence estimates were confirmed on a manually reviewed set of test articles. A second model not requiring MeSH terms was also created, and performs almost as well. Discussion: Both models accurately rank and predict article RCT confidence. Using the model and the manually reviewed samples, it is estimated that about 8000 (3%) additional RCTs can be identified in MEDLINE, and that 5% of articles tagged as RCTs in Medline may not be identified. Conclusion: Retagging human-related studies with a continuously valued RCT confidence is potentially more useful for article ranking and review than a simple yes/no prediction. The automated RCT tagging tool should offer significant savings of time and effort during the process of writing SRs, and is a key component of a multistep text mining pipeline that we are building to streamline SR workflow. In addition, the model may be useful for identifying errors in MEDLINE publication types. The RCT confidence predictions described here have been made available to users as a web service with a user query form front end at: http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/RCT_Tagger.cgi.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Two validated systems are freely available for general use [16, 25].”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“The team validated the performance on a withheld portion of the same dataset, finding the system discriminated accurately between RCTs and non-RCTs (area under the receiver operating characteristics curve (AUROC) = 0.973).”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“These include automated classification of articles by study method4,7,8”
From Trialstreamer: A living, automatically updated database of clinical trial reports (2020) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999
Yeh, M.-L. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,829
Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning
Fayyad, U. M., & Irani, K. B. (1993) · International Joint Conference on Artificial Intelligence · Cited by 2,870
Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis
Cooke, A., Smith, D., & Booth, A. (2012) · Qualitative Health Research · doi:10.1177/1049732312452938 · Cited by 2,608
Evidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232 · cited 1×
“There are also efforts underway to produce quicker, smaller reviews, called “Rapid Reviews” that will require faster ways to get at the most important publications.(6,7)”
BackgroundRobinson, K. A., & Dickersin, K. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.150 · Cited by 833
Opioid antagonists for alcohol dependence
Srisurapanont, M., & Jarusuraisin, N. (2002) · The Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001867 · Cited by 764
Interventions for treating oral mucositis for patients with cancer receiving treatment
Clarkson, J. E., Worthington, H. V., Furness, S., McCabe, M., Khalid, T., & Meyer, S. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001973.pub4 · Cited by 674
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Self-citationLIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
Early versus delayed laparoscopic cholecystectomy for people with acute cholecystitis
Gurusamy, K. S., Davidson, C., Gluud, C., & Davidson, B. R. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005440.pub3 · Cited by 327
Haynes, R. B. (2002) · BMC Health Services Research · doi:10.1186/1472-6963-2-3 · Cited by 296
The paths from research to improved health outcomes
Glasziou, P., & Haynes, B. (2005) · Evidence Based Nursing · doi:10.1136/ebn.8.2.36 · Cited by 283
Interventions for preventing oral mucositis for patients with cancer receiving treatment
Worthington, H. V., Clarkson, J. E., Bryan, G., Furness, S., Glenny, A.-M., Littlewood, A., McCabe, M. G., Meyer, S., Khalid, T., & Riley, P. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000978.pub5 · Cited by 262
Opioid antagonists for alcohol dependence
Rösner, S., Hackl-Herrwerth, A., Leucht, S., Vecchi, S., Srisurapanont, M., & Soyka, M. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001867.pub3 · Cited by 247
Indexing consistency in MEDLINE.
Funk, M. E., & Reid, C. A. (1983) · Europe PMC (PubMed Central) · Cited by 180
Text Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177
When and how to update systematic reviews
Moher, D., Tsertsvadze, A., Tricco, A., Eccles, M., Grimshaw, J., Sampson, M., & Barrowman, N. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000023.pub3 · Cited by 146
Kanuga, S. (2013) · The Journal of the American Dental Association · doi:10.14219/jada.archive.2013.0211 · Cited by 137
Harker, J., & Kleijnen, J. (2012) · International Journal of Evidence-Based Healthcare · doi:10.1111/j.1744-1609.2012.00290.x · Cited by 137 · cited 1×
“There are also efforts underway to produce quicker, smaller reviews, called “Rapid Reviews” that will require faster ways to get at the most important publications.(6,7)”
BackgroundAgarwal, S., & Yu, H. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp548 · Cited by 104
Towards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101
A new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100
Surveillance search techniques identified the need to update systematic reviews
Sampson, M., Shojania, K. G., McGowan, J., Daniel, R., Rader, T., Iansavitchene, A., Ji, J., Ansari, M. T., & Moher, D. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.003 · Cited by 86
Preconception lifestyle advice for people with subfertility
Anderson, K., Norman, R. J., & Middleton, P. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008189.pub2 · Cited by 85
Sentence retrieval for abstracts of randomized controlled trials.
Chung, G. Y. (2009) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-9-10 · Cited by 83
Fiszman, M., Demner-Fushman, D., Kilicoglu, H., & Rindflesch, T. C. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.10.002 · Cited by 77
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · cited 5×
“Interestingly, Bekhuis and Demner-Fushman have published work on using machine learning to screen nonrandomized studies for inclusion in SRs.(33,34) There has also been a significant amount of general Natural Language Processing (NLP) research (beyond machine learning) with applications to improving the SR process and…”
BackgroundInfluentialOptimizing feature representation for automated systematic review work prioritization.
Cohen, A. M. (2008) · PubMed Central · Cited by 62
Self-citationCohen, A. M., Ambert, K., & McDonagh, M. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-33 · Cited by 57
Self-citationMEDLINE clinical queries are robust when searching in recent publishing years
Wilczynski, N. L., McKibbon, K. A., Walter, S. D., Garg, A. X., & Haynes, R. B. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001075 · Cited by 50
Smalheiser, N. R., Lin, C., Jia, L., Jiang, Y., Cohen, A. M., Yu, C., Davis, J. M., Adams, C. E., McDonagh, M. S., & Meng, W. (2014) · Health Information Science and Systems · doi:10.1186/2047-2501-2-1 · Cited by 47
Self-citationUpdating comparative effectiveness reviews: Current efforts in AHRQ's Effective Health Care Program
Tsertsvadze, A., Maglione, M., Chou, R., Garritty, C., Coleman, C. I., Lux, L. J., Bass, E. B., Balshem, H., & Moher, D. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.03.011 · Cited by 46
Exploiting the systematic review protocol for classification of medical abstracts
Frunza, O., Inkpen, D. Z., Matwin, S., Klement, W., & O’Blenis, P. (2010) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2010.10.005 · Cited by 45
Bekhuis, T., Tseytlin, E., Mitchell, K. J., & Demner-Fushman, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086277 · Cited by 45 · cited 5×
“Interestingly, Bekhuis and Demner-Fushman have published work on using machine learning to screen nonrandomized studies for inclusion in SRs.(33,34) There has also been a significant amount of general Natural Language Processing (NLP) research (beyond machine learning) with applications to improving the SR process and…”
BackgroundInfluentialCohen, A. M., Adams, C. E., Davis, J. M., Yu, C., Yu, P. S., Meng, W., Duggan, L., McDonagh, M., & Smalheiser, N. R. (2010) · Proceedings of the 1st ACM International Health Informatics Symposium · doi:10.1145/1882992.1883046 · Cited by 45
Self-citationAn effective general purpose approach for automated biomedical document classification.
(2006) · Europe PMC (PubMed Central) · Cited by 44
Self-citationRule-based deduplication of article records from bibliographic databases
Jiang, Y., Lin, C., Meng, W., Yu, C., Cohen, A. M., & Smalheiser, N. R. (2014) · Database · doi:10.1093/database/bat086 · Cited by 41
Self-citationMethods for the Drug Effectiveness Review Project
McDonagh, M. S., Jonas, D. E., Gartlehner, G., Little, A., Peterson, K., Carson, S., Gibson, M., & Helfand, M. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-140 · Cited by 38
Self-citationWieland, L. S., Robinson, K. A., & Dickersin, K. (2012) · BMJ · doi:10.1136/bmj.d7501 · Cited by 34
Edinger, T., & Cohen, A. M. (2013) · Europe PMC (PubMed Central) · Cited by 33
Self-citationBlake, C. L., & Pratt, W. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20486 · Cited by 32
Preconception lifestyle advice for people with subfertility
Anderson, K. J., Norman, R. J., & Middleton, P. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008189 · Cited by 32
Cohen, A. M., Ambert, K. H., & McDonagh, M. S. (2010) · PubMed · Cited by 26
Self-citationAphinyanaphongs, Y., Statnikov, A., & Aliferis, C. F. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2031 · Cited by 24
Finding the gold in Medline: clinical queries
Haynes, R. B., & Wilczynski, N. (2005) · BMJ Evidence-Based Medicine · doi:10.1136/ebm.10.4.101 · Cited by 24
A prototype system to support evidence-based practice.
Demner‐Fushman, D., SECKMAN, C. A., Fisher, C. A., Hauser, S. E., Clayton, J., & Thoma, G. R. (2008) · PubMed · Cited by 23
A Simple Method for Estimating Conditional Probabilities for SVMsPreprint
Stefan Rüping, A. (2004) · RePEc: Research Papers in Economics · doi:10.17877/de290r-15242 · Cited by 20
Shao, W., Adams, C. E., Cohen, A. M., Davis, J. M., McDonagh, M. S., Thakurta, S., Yu, P. S., & Smalheiser, N. R. (2015) · Methods · doi:10.1016/j.ymeth.2014.11.006 · Cited by 19
Self-citationFinding the gold in MEDLINE: Clinical Queries
R. Brian Haynes, M., & Nancy Wilczynski, M. (2005) · ACP Journal Club · doi:10.7326/acpjc-2005-142-1-a08 · Cited by 17
Cited by
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Marshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190
Wallace, B. C., Noel-Storr, A., Marshall, I. J., Cohen, A. M., Smalheiser, N. R., & Thomas, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx053 · Cited by 186
Self-citationPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
“Tools to automatically recognise these 69 , 67 - 70 , 71 can be”
Background
Metadata
- Authors
- Cohen, A. M., Smalheiser, N. R., McDonagh, M. S., Yu, C., Adams, C. E., Davis, J. M., & Yu, P. S.
- Affiliations
- Oregon Health & Science University (US)
- University of Illinois Chicago (US)
- University of Nottingham (GB)
- Corresponding author
- Cohen, Aaron M
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-02-05
- Volume
- 22
- Issue
- 3
- Pages
- 707-717
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC4457112
- OpenAlex
- W2122514515
- Cited by
- 78
- References
- 53
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 8 · 2023: 5 · 2022: 16 · 2021: 10 · 2020: 8 · 2019: 4 · 2018: 5 · 2017: 4 · 2016: 1 · 2015: 1
- FWCI
- 3.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health/National Library of Medicine (R01LM010817)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- randomized controlled trials, article classification, systematic reviews, evidence-based medicine, MEDLINE, MeSH terms, text mining, automatic annotation
- MeSH terms
- Artificial Intelligence, Humans, Review Literature as Topic, ROC Curve, Randomized Controlled Trials as Topic, MEDLINE, Information Storage and Retrieval, Evidence-Based Medicine, Support Vector Machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.