CoChat

A Deep Learning Method to Automatically Identify Reports of Scientifically Rigorous Clinical Research from the Biomedical Literature: Comparative Analytic Study

Open access
Del Fiol, Guilherme, Michelson, Matthew, Iorio, Alfonso, Cotoi, Chris, Haynes, R Brian · 2018 · Journal of Medical Internet Research

Published version of a preprint

Cited by 51

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Abstract

BACKGROUND: A major barrier to the practice of evidence-based medicine is efficiently finding scientifically sound studies on a given clinical topic. OBJECTIVE: To investigate a deep learning approach to retrieve scientifically sound treatment studies from the biomedical literature. METHODS: We trained a Convolutional Neural Network using a noisy dataset of 403,216 PubMed citations with title and abstract as features. The deep learning model was compared with state-of-the-art search filters, such as PubMed's Clinical Query Broad treatment filter, McMaster's textword search strategy (no Medical Subject Heading, MeSH, terms), and Clinical Query Balanced treatment filter. A previously annotated dataset (Clinical Hedges) was used as the gold standard. RESULTS: The deep learning model obtained significantly lower recall than the Clinical Queries Broad treatment filter (96.9% vs 98.4%; P<.001); and equivalent recall to McMaster's textword search (96.9% vs 97.1%; P=.57) and Clinical Queries Balanced filter (96.9% vs 97.0%; P=.63). Deep learning obtained significantly higher precision than the Clinical Queries Broad filter (34.6% vs 22.4%; P<.001) and McMaster's textword search (34.6% vs 11.8%; P<.001), but was significantly lower than the Clinical Queries Balanced filter (34.6% vs 40.9%; P<.001). CONCLUSIONS: Deep learning performed well compared to state-of-the-art search filters, especially when citations were not indexed. Unlike previous machine learning approaches, the proposed deep learning model does not require feature engineering, or time-sensitive or proprietary features, such as MeSH terms and bibliometrics. Deep learning is a promising approach to identifying reports of scientifically rigorous clinical research. Further work is needed to optimize the deep learning model and to assess generalizability to other areas, such as diagnosis, etiology, and prognosis.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/10281

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 6 references

Research works printed but not linked: 25

  1. Haynes, RB, 1993, ACP Journal club, 119, A22, 10.7326/ACPJC-1993-119-3-A22 · DOI
  2. 10.1371/journal.pmed.0020124 · DOI · Paper page
  3. Haynes, RB, 1994, J Am Med Inform Assoc, 1, 447, 10.1136/jamia.1994.95153434 · DOI · Paper page
  4. Wilczynski, NL, 2001, Stud Health Technol Inform, 84, 390
  5. 10.1186/1472-6947-5-20 · DOI
  6. 10.1136/amiajnl-2012-001075 · DOI · Paper page
  7. 10.1016/j.sapharm.2016.04.006 · DOI
  8. 10.1197/jamia.M1909 · DOI
  9. LeeHPhamPLargmanYNgAYUnsupervised feature learning for audio classification using convolutional deep belief networks2009Advances in neural information processing systems2009Vancouver, Canada10961104
  10. KimYarXiv20142018-05-30Convolutional neural networks for sentence classification https://arxiv.org/abs/1408.5882 · arXiv
  11. MikolovTChenKCorradoGDeanJarXiv20132018-05-30Efficient estimation of word representations in vector space https://arxiv.org/abs/1301.3781 · arXiv
  12. YinWKannKYuMSchützeHarXiv20172018-05-30Comparative study of CNN and RNN for natural language processing https://arxiv.org/abs/1702.01923 · arXiv
  13. Liberati, A, 2009, BMJ, 339, b2700, 10.1136/bmj.b2700 · DOI · Paper page
  14. 10.1016/j.jclinepi.2007.10.003 · DOI
  15. Shojania, K, 2007, AHRQ Technical Reviews. Updating Systematic Reviews
  16. 10.1001/jamainternmed.2014.368 · DOI · Paper page
  17. 10.1016/S0306-4573(99)00056-4 · DOI · Paper page
  18. KrauseJSappBHowardAZhouHToshevADuerigTPhilbinJFei-FeiLThe unreasonable effectiveness of noisy data for fine-grained recognition2016European Conference on Computer Vision2016Amsterdam, The Netherlands301320 · DOI · Paper page
  19. Hughes, M, 2017, Stud Health Technol Inform, 235, 246
  20. WangYLiuSRastegar-MojaradMWangLShenFLiuFLiuHDependency and AMR embeddings for drug-drug interaction extraction from biomedical literature20178th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics2017Boston, MA, USA · DOI
  21. 10.1109/JBHI.2016.2633963 · DOI · Paper page
  22. CheZChengYSunZLiuYarXiv20172018-05-30Exploiting convolutional neural network for risk prediction with medical feature embedding https://arxiv.org/abs/1701.07474 · arXiv
  23. LiuPQiuXHuangXRecurrent neural network for text classification with multi-task learning201625th International Joint Conference on Artificial Intelligence2016New York, NY, USAAAAI PressAAA · DOI
  24. LaiSXuLLiuKZhaoJRecurrent convolutional neural networks for text classification201529th AAAI Conference on Artificial Intelligence2015Austin, Texas, USAAAAI Press22672273 · DOI · Paper page
  25. Bergstra, J, 2012, J Mach Learn Res, 13, 281

Cited by

Metadata

Authors
Del Fiol, G., Michelson, M., Iorio, A., Cotoi, C., & Haynes, R. B.
Venue
Journal of Medical Internet Research
Published
2018-06-25
Volume
20
Issue
6
Pages
e10281
ISSN
1438-8871 (ISSN-L)
OpenAlex
W2801170057
Cited by
51
References
6
FWCI
11.25
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.