The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Deep learning in clinical natural language processing: a methodical review
Open accessAbstract
Objective: This article methodically reviews the literature on deep learning (DL) for natural language processing (NLP) in the clinical domain, providing quantitative analysis to answer 3 research questions concerning methods, scope, and context of current research. Materials and Methods: We searched MEDLINE, EMBASE, Scopus, the Association for Computing Machinery Digital Library, and the Association for Computational Linguistics Anthology for articles using DL-based approaches to NLP problems in electronic health records. After screening 1,737 articles, we collected data on 25 variables across 212 papers. Results: DL in clinical NLP publications more than doubled each year, through 2018. Recurrent neural networks (60.8%) and word2vec embeddings (74.1%) were the most popular methods; the information extraction tasks of text classification, named entity recognition, and relation extraction were dominant (89.2%). However, there was a “long tail” of other methods and specific tasks. Most contributions were methodological variants or applications, but 20.8% were new methods of some kind. The earliest adopters were in the NLP community, but the medical informatics community was the most prolific. Discussion: Our analysis shows growing acceptance of deep learning as a baseline for NLP research, and of DL-based NLP in the medical community. A number of common associations were substantiated (eg, the preference of recurrent neural networks for sequence-labeling named entity recognition), while others were surprisingly nuanced (eg, the scarcity of French language clinical NLP with deep learning). Conclusion: Deep learning has not yet fully penetrated clinical NLP and is growing rapidly. This review highlighted both the popular and unique trends in this active field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,839
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310
Recurrent Convolutional Neural Networks for Text Classification
Lai, S., Xu, L., Liu, K., & Zhao, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9513 · Cited by 2,280
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“A renewed emphasis on scientific rigor has permeated the academic establishment, 108,109 with replicability and reproducibility studies (eg, from ACM’s definitions 110,111 ) showing how much of the literature has fallen short of its scientific goals.”
BackgroundShickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Xiao, C., Choi, E., & Sun, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy068 · Cited by 792
Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review
Sheikhalishahi, S., Miotto, R., Dudley, J. T., Lavelli, A., Rinaldi, F., & Osmani, V. (2019) · JMIR Medical Informatics · doi:10.2196/12239 · Cited by 558
The Unified Medical Language System
Humphreys, B. L., McCray, A. T., & Lindberg, D. A. B. (1993) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1637976 · Cited by 507
Reproducibility vs. Replicability: A Brief History of a Confused Terminology
Plesser, H. E. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00076 · Cited by 424
Enhancing clinical concept extraction with contextual embeddings
Si, Y., Wang, J., Xu, H., & Roberts, K. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz096 · Cited by 322
Self-citationClinical Natural Language Processing in languages other than English: opportunities and challenges
Névéol, A., Dalianis, H., Velupillai, S., Savova, G., & Zweigenbaum, P. (2018) · Journal of Biomedical Semantics · doi:10.1186/s13326-018-0179-8 · Cited by 311
Velupillai, S., Suominen, H., Liakata, M., Roberts, A., Shah, A. D., Morley, K., Osborn, D., Hayes, J., Stewart, R., Downs, J., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.10.005 · Cited by 247
Dreisbach, C., Koleck, T. A., Bourne, P. E., & Bakken, S. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.02.008 · Cited by 231
De-identification of clinical notes via recurrent neural network and conditional random field
Liu, Z., Tang, B., Wang, X., & Chen, Q. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.05.023 · Cited by 218
Natural Language Processing for EHR-Based Computational Phenotyping
Zeng, Z., Deng, Y., Li, X., Naumann, T., & Luo, Y. (2019) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2018.2849968 · Cited by 209
Weng, W.-H., Wagholikar, K. B., McCray, A. T., Szolovits, P., & Chueh, H. C. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0556-8 · Cited by 163
Gonzalez‐Hernandez, G., Sarker, A., O’Connor, K., & Savova, G. (2017) · Yearbook of Medical Informatics · doi:10.15265/iy-2017-029 · Cited by 160
Dickie, D. A., Shenkin, S. D., Anblagan, D., Lee, J. Y., Blesa, M., Rodriguez, D., Boardman, J. P., Waldman, A. D., Job, D. E., & Wardlaw, J. M. (2017) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00001 · Cited by 155
Hierarchical attention networks for information extraction from cancer pathology reports
Gao, S. B., Young, M. T., Qiu, J. X., Yoon, H., Christian, J. B., Fearn, P., Tourassi, G. D., & Ramanthan, A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx131 · Cited by 152
Wang, Q., Zhou, Y., Ruan, T., Gao, D., Xia, Y., & He, P. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103133 · Cited by 150
Self-citationA Study of Neural Word Embeddings for Named Entity Recognition in Clinical Text.
Wu, Y., Xu, J., Jiang, M., Zhang, Y., & Xu, H. (2015) · PubMed · Cited by 119
Self-citationNatural language generation for electronic health records
Lee, S. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0070-0 · Cited by 105
Dandala, B., Joopudi, V., & Devarakonda, M. V. (2019) · Drug Safety · doi:10.1007/s40264-018-0764-x · Cited by 92
Banerjee, I., Chen, M. C., Lungren, M. P., & Rubin, D. L. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.012 · Cited by 87
Clinical Abbreviation Disambiguation Using Neural Word Embeddings
Wu, Y., Xu, J., Zhang, Y., & Xu, H. (2015) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/w15-3822 · Cited by 85
Self-citationMunkhdalai, T., Liu, F., & Yu, H. Q. (2018) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.9361 · Cited by 81
Li, J., Parrott, S., Sweeting, M. J., Farmer, A. A., Ross, J., Dack, C., Pal, K., Yardley, L., Barnard, M., Hudda, M. T., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9256 · Cited by 81
Li, F., Liu, W., & Yu, H. Q. (2018) · JMIR Medical Informatics · doi:10.2196/12159 · Cited by 81
Ontoserver: a syndicated terminology server
Metke‐Jimenez, A., Steel, J. R. H., Hansen, D. T., & Lawley, M. (2018) · Journal of Biomedical Semantics · doi:10.1186/s13326-018-0191-z · Cited by 80
A hybrid approach for named entity recognition in Chinese electronic medical record
Ji, B., Liu, R., Li, S., Yu, J., Wu, Q., Tan, Y., & Wu, J. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0767-2 · Cited by 78
Wunnava, S., Qin, X., Kakar, T., Şen, C., Rundensteiner, E. A., & Kong, X. (2019) · Drug Safety · doi:10.1007/s40264-018-0765-9 · Cited by 72
Word2Vec inversion and traditional text classifiers for phenotyping lupus
Turner, C. A., Jacobs, A. D., Marques, C. K., Oates, J. C., Kamen, D. L., Anderson, P. E., & Obeid, J. S. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0518-1 · Cited by 71
Qiu, J., Zhou, Y., Wang, Q., Ruan, T., & Gao, J. (2019) · IEEE Transactions on NanoBioscience · doi:10.1109/tnb.2019.2908678 · Cited by 67
Self-citationLi, Z., Yang, Z., Shen, C., Xu, J., Zhang, Y., & Xu, H. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0736-9 · Cited by 58
Self-citationA convolutional route to abbreviation disambiguation in clinical text
Joopudi, V., Dandala, B., & Devarakonda, M. V. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.07.025 · Cited by 50
Li, R., Hu, B., Liu, F., Liu, W., Cunningham, F. E., McManus, D. D., & Yu, H. Q. (2018) · JMIR Medical Informatics · doi:10.2196/10788 · Cited by 50
Improving Clinical Named Entity Recognition with Global Neural Attention
Xu, G., Wang, C., & He, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-96893-3_20 · Cited by 47
Moen, H., Ginter, F., Marsi, E., Peltonen, L., Salakoski, T., & Salanterä, S. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-15-s2-s2 · Cited by 46
Patient representation learning and interpretable evaluation using clinical notes
Sushil, M., Šuster, S., Luyckx, K., & Daelemans, W. M. P. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.06.016 · Cited by 45
A Frame-Based NLP System for Cancer-Related Information Extraction.
Si, Y., & Roberts, K. E. (2018) · Europe PMC (PubMed Central) · Cited by 40
Self-citationDeep generative learning for automated EHR diagnosis of traditional Chinese medicine
Liang, Z., Liu, J. L., Ou, A., Zhang, H., Li, Z., & Huang, J. X. (2018) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.05.008 · Cited by 39
Exploring Joint AB-LSTM With Embedded Lemmas for Adverse Drug Reaction Discovery
Santiso, S., Pérez, A., & Casillas, A. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2879744 · Cited by 38
Wu, Y., Yang, X., Bian, J., Guo, Y. M., Xu, H., & Hogan, W. R. (2018) · Europe PMC (PubMed Central) · Cited by 29
Self-citationExtraction of risk factors for cardiovascular diseases from Chinese electronic medical records
Su, J., Hu, J., Jiang, J., Xie, J., Yang, Y., He, B., Yang, J., & Guan, Y. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.01.007 · Cited by 22
Li, F., & Yu, H. Q. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz018 · Cited by 21
A syntactic path-based hybrid neural network for negation scope detection
Lazib, L., Qin, B., Zhao, Y., Zhang, W., & Liu, T. (2018) · Frontiers of Computer Science · doi:10.1007/s11704-018-7368-6 · Cited by 20
MfeCNN: Mixture Feature Embedding Convolutional Neural Network for Data Mapping
Li, D., Huang, M. S., Li, X., Ruan, Y., & Yao, L. (2018) · IEEE Transactions on NanoBioscience · doi:10.1109/tnb.2018.2841053 · Cited by 15
Névéol, A., Zweigenbaum, P., & Processing, S. E. F. T. I. Y. S. O. C. N. L. (2018) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1667080 · Cited by 15
Fivez, P., Šuster, S., & Daelemans, W. M. P. (2017) · Computational Linguistics in the Netherlands Journal · doi:10.48550/arxiv.1710.07045 · Cited by 14
Li, R., Jagannatha, A., & Yu, H. Q. (2017) · PubMed Central · Cited by 14
Miftahutdinov, Z., & Tutubalina, E. V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98932-7_19 · Cited by 10
Sahu, S. K., & Anand, A. (2018) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2018.03.006 · Cited by 9
An approach for medical event detection in Chinese clinical notes of electronic health records
Zhou, X., Xiong, H., Zeng, S., Fu, X., & Ji, W. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0756-5 · Cited by 5
Hierarchical sequence labeling for extracting BEL statements from biomedical literature
Liu, S., Shao, Y., Qian, L., et al. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0758-3 · Cited by 4
Cited by
Using machine learning approaches for multi-omics data analysis: A review
Reel, P. S., Reel, S., Pearson, E., Trucco, E., & Jefferson, E. (2021) · Biotechnology Advances · doi:10.1016/j.biotechadv.2021.107739 · Cited by 903
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
Neural Natural Language Processing for unstructured data in electronic health records: A review
Li, I., Pan, J., Goldwasser, J., Verma, N., Wong, W. P., Nuzumlalı, M. Y., Rosand, B., Li, Y., Zhang, M., Chang, D., et al. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100511 · Cited by 254
Limitations of transformers on clinical text classification
Gao, S., Alawad, M., Young, M. T., Gounley, J., Schaefferkoetter, N., Yoon, H. J., Wu, X.-C., Durbin, E. B., Doherty, J., Stroup, A., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3062322 · Cited by 157
Metadata
- Authors
- Wu, S., Roberts, K., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., Zhao, B., & Xu, H.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-12-03
- Volume
- 27
- Issue
- 3
- Pages
- 457-470
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2993873509
- Cited by
- 491
- References
- 61
- FWCI
- 21.64
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.