A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
BACKGROUND: A major barrier to the practice of evidence-based medicine is efficiently finding scientifically sound studies on a given clinical topic. OBJECTIVE: To investigate a deep learning approach to retrieve scientifically sound treatment studies from the biomedical literature. METHODS: We trained a Convolutional Neural Network using a noisy dataset of 403,216 PubMed citations with title and abstract as features. The deep learning model was compared with state-of-the-art search filters, such as PubMed's Clinical Query Broad treatment filter, McMaster's textword search strategy (no Medical Subject Heading, MeSH, terms), and Clinical Query Balanced treatment filter. A previously annotated dataset (Clinical Hedges) was used as the gold standard. RESULTS: The deep learning model obtained significantly lower recall than the Clinical Queries Broad treatment filter (96.9% vs 98.4%; P<.001); and equivalent recall to McMaster's textword search (96.9% vs 97.1%; P=.57) and Clinical Queries Balanced filter (96.9% vs 97.0%; P=.63). Deep learning obtained significantly higher precision than the Clinical Queries Broad filter (34.6% vs 22.4%; P<.001) and McMaster's textword search (34.6% vs 11.8%; P<.001), but was significantly lower than the Clinical Queries Balanced filter (34.6% vs 40.9%; P<.001). CONCLUSIONS: Deep learning performed well compared to state-of-the-art search filters, especially when citations were not indexed. Unlike previous machine learning approaches, the proposed deep learning model does not require feature engineering, or time-sensitive or proprietary features, such as MeSH terms and bibliometrics. Deep learning is a promising approach to identifying reports of scientifically rigorous clinical research. Further work is needed to optimize the deep learning model and to assess generalizability to other areas, such as diagnosis, etiology, and prognosis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
Living systematic review: 1. Introduction-the why, what, when, and how
Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.010 · Cited by 655
Marshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435
Text Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177 · cited 1×
“A polynomial Support Vector Machine classifier based on MeSH terms, publication type, and title/abstract words obtained a recall of 96% and precision of 18% against a gold standard of internal medicine articles included in the American College of Physicians Journal Club [9].”
MethodTowards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101 · cited 4×
“A Naïve Bayes classifier with these features obtained a recall and precision of 91.4% and 52.5% for treatment studies in the Clinical Hedges database [8].”
BackgroundMethodInfluentialSelf-citationCross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · cited 1×
“In addition, citation ranking could help with systematic review development, since front-loading “eligible” citations can be used to help train and calibrate citation screeners and prioritize work [22].”
Background
Research works printed but not linked: 25
- Haynes, RB, 1993, ACP Journal club, 119, A22, 10.7326/ACPJC-1993-119-3-A22 · DOI
- 10.1371/journal.pmed.0020124 · DOI · Paper page
- Haynes, RB, 1994, J Am Med Inform Assoc, 1, 447, 10.1136/jamia.1994.95153434 · DOI · Paper page
- Wilczynski, NL, 2001, Stud Health Technol Inform, 84, 390
- 10.1186/1472-6947-5-20 · DOI
- 10.1136/amiajnl-2012-001075 · DOI · Paper page
- 10.1016/j.sapharm.2016.04.006 · DOI
- 10.1197/jamia.M1909 · DOI
- LeeHPhamPLargmanYNgAYUnsupervised feature learning for audio classification using convolutional deep belief networks2009Advances in neural information processing systems2009Vancouver, Canada10961104
- KimYarXiv20142018-05-30Convolutional neural networks for sentence classification https://arxiv.org/abs/1408.5882 · arXiv
- MikolovTChenKCorradoGDeanJarXiv20132018-05-30Efficient estimation of word representations in vector space https://arxiv.org/abs/1301.3781 · arXiv
- YinWKannKYuMSchützeHarXiv20172018-05-30Comparative study of CNN and RNN for natural language processing https://arxiv.org/abs/1702.01923 · arXiv
- Liberati, A, 2009, BMJ, 339, b2700, 10.1136/bmj.b2700 · DOI · Paper page
- 10.1016/j.jclinepi.2007.10.003 · DOI
- Shojania, K, 2007, AHRQ Technical Reviews. Updating Systematic Reviews
- 10.1001/jamainternmed.2014.368 · DOI · Paper page
- 10.1016/S0306-4573(99)00056-4 · DOI · Paper page
- KrauseJSappBHowardAZhouHToshevADuerigTPhilbinJFei-FeiLThe unreasonable effectiveness of noisy data for fine-grained recognition2016European Conference on Computer Vision2016Amsterdam, The Netherlands301320 · DOI · Paper page
- Hughes, M, 2017, Stud Health Technol Inform, 235, 246
- WangYLiuSRastegar-MojaradMWangLShenFLiuFLiuHDependency and AMR embeddings for drug-drug interaction extraction from biomedical literature20178th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics2017Boston, MA, USA · DOI
- 10.1109/JBHI.2016.2633963 · DOI · Paper page
- CheZChengYSunZLiuYarXiv20172018-05-30Exploiting convolutional neural network for risk prediction with medical feature embedding https://arxiv.org/abs/1701.07474 · arXiv
- LiuPQiuXHuangXRecurrent neural network for text classification with multi-task learning201625th International Joint Conference on Artificial Intelligence2016New York, NY, USAAAAI PressAAA · DOI
- LaiSXuLLiuKZhaoJRecurrent convolutional neural networks for text classification201529th AAAI Conference on Artificial Intelligence2015Austin, Texas, USAAAAI Press22672273 · DOI · Paper page
- Bergstra, J, 2012, J Mach Learn Res, 13, 281
Cited by
Toğaçar, M., Ergen, B., Cömert, Z., & Özyurt, F. (2020) · IRBM · doi:10.1016/j.irbm.2019.10.006 · Cited by 297
Deep Learning Intervention for Health Care Challenges: Some Biomedical Domain Considerations
Tobore, I., Li, J., Yuhang, L., Al-Handarish, Y., Kandwal, A., Nie, Z., & Wang, L. (2019) · JMIR mHealth and uHealth · doi:10.2196/11966 · Cited by 180
Michelson, M., & Reuter, K. (2019) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2019.100443 · Cited by 152
Self-citationAfzal, M., Alam, F., Malik, K. M., & Malik, G. M. (2020) · Journal of Medical Internet Research · doi:10.2196/19810 · Cited by 65
“Several promising approaches have been explored [34-41] to retrieve high-quality (ie, scientifically sound) studies from the PubMed database.”
BackgroundMethodInfluentialImproving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62
Metadata
- Authors
- Del Fiol, G., Michelson, M., Iorio, A., Cotoi, C., & Haynes, R. B.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-06-25
- Volume
- 20
- Issue
- 6
- Pages
- e10281
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W2801170057
- Cited by
- 51
- References
- 6
- FWCI
- 11.25
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study. https://app.cochat.ai/papers/10.2196/10281
- The paper
- Del Fiol, G., Michelson, M., Iorio, A., Cotoi, C., & Haynes, R. B. (2018). A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study. Journal of Medical Internet Research. https://doi.org/10.2196/10281
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.