The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
De-identification of patient notes with recurrent neural networks
Open accessAbstract
Objective: Patient notes in electronic health records (EHRs) may contain critical information for medical investigations. However, the vast majority of medical investigators can only access de-identified notes, in order to protect the confidentiality of patients. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) defines 18 types of protected health information that needs to be removed to de-identify patient notes. Manual de-identification is impractical given the size of electronic health record databases, the limited number of researchers with access to non-de-identified notes, and the frequent mistakes of human annotators. A reliable automated de-identification system would consequently be of high value. Materials and Methods: We introduce the first de-identification system based on artificial neural networks (ANNs), which requires no handcrafted features or rules, unlike existing systems. We compare the performance of the system with state-of-the-art systems on two datasets: the i2b2 2014 de-identification challenge dataset, which is the largest publicly available de-identification dataset, and the MIMIC de-identification dataset, which we assembled and is twice as large as the i2b2 2014 dataset. Results: Our ANN model outperforms the state-of-the-art systems. It yields an F1-score of 97.85 on the i2b2 2014 dataset, with a recall of 97.38 and a precision of 98.32, and an F1-score of 99.23 on the MIMIC de-identification dataset, with a recall of 99.25 and a precision of 99.21. Conclusion: Our findings support the use of ANNs for de-identification of patient notes, as they show better performance than previously published systems while requiring no manual feature engineering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Self-citationThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“As a result, methods based on ANNs have shown promising results for various tasks in natu-ral language processing, such as language modeling (Mikolov et al., 2010), text classification (Socher et al., 2013; Kim, 2014; Blunsom et al., 2014; Lee and Dernoncourt, 2016), question answering (We-ston et…”
MethodNatural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,491
Saeed, M., Villarroel, M., Reisner, A. T., Clifford, G., Lehman, L.-W., Moody, G., Heldt, T., Kyaw, T. H., Moody, B., & Mark, R. G. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e31820a92c6 · Cited by 1,162
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670
Automated de-identification of free-text medical records
Neamatullah, I., Douglass, M. M., Lehman, L.-W. H., Reisner, A., Villarroel, M., Long, W. J., Szolovits, P., Moody, G. B., Mark, R. G., & Clifford, G. D. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-32 · Cited by 413
Self-citationA Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.2188 · Cited by 407
Standards for privacy of individually identifiable health information. Final rule.
(2002) · PubMed · Cited by 400
A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering
Wang, D., & Nyberg, E. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.3115/v1/p15-2116 · Cited by 353
Meystre, S. M., Friedlin, F. J., South, B. R., Shen, S., & Samore, M. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-70 · Cited by 329
Standards for Privacy of Individually Identifiable Health Information
(2001) · Best Practices · doi:10.1201/9781420000177.ch38 · Cited by 284
Replacing Personally-Identifying Information in Medical Records, the Scrub System
Sweeney, L. (2018) · PubMed · doi:10.1184/r1/6625565 · Cited by 251
Rule-Based Information Extraction is Dead! Long Live Rule-Based Information Extraction Systems!
Chiticariu, L., Li, Y., & Reiss, F. R. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1079 · Cited by 246
Stubbs, A., Kotfila, C., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.007 · Cited by 245
Translation Modeling with Bidirectional Recurrent Neural Networks
Sundermeyer, M., Alkhouli, T., Wuebker, J., & Ney, H. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1003 · Cited by 176
Gupta, D., Saul, M., & Gilbertson, J. (2004) · American Journal of Clinical Pathology · doi:10.1309/e6k3-3gbp-e5c2-7fyu · Cited by 165
The MITRE Identification Scrubber Toolkit: Design, training, and assessment
Aberdeen, J. S., Bayer, S., Yeniterzi, R., Wellner, B., Clark, C., Hanauer, D. A., Malin, B., & Hirschman, L. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.09.007 · Cited by 146
Automatic detection of protected health information from clinic narratives
Yang, H., & Garibaldi, J. M. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.015 · Cited by 136
Development and evaluation of an open source software tool for deidentification of pathology reports
Beckwith, B. A., Mahaadevan, R., Balis, U. J., & Kuo, F. C. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-12 · Cited by 110
A Software Tool for Removing Patient Identifying Information from Clinical Documents
Friedlin, F. J., & McDonald, C. J. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2702 · Cited by 108
Szarvas, G., Farkas, R., & Kocsor, A. (2006) · Lecture notes in computer science · doi:10.1007/11893318_27 · Cited by 105
Liu, Z., Chen, Y., Tang, B., Wang, X., Chen, Q., Li, H., Wang, J., Deng, Q., & Zhu, S. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.009 · Cited by 104
Medical document anonymization with a semantic lexicon.
Ruch, P., Baud, R. H., Rassinoux, A. M., Bouillon, P., & Robert, G. (2000) · PubMed Central · Cited by 101
A de-identifier for medical discharge summaries
Uzuner, Ö., Sibanda, T. C., Luo, Y., & Szolovits, P. (2007) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2007.10.001 · Cited by 99
Self-citationEarly Results of the Meaningful Use Program for Electronic Health Records
Wright, A. T., Henkin, S., Feblowitz, J. C., McCoy, A. B., Bates, D. W., & Sittig, D. F. (2013) · New England Journal of Medicine · doi:10.1056/nejmc1213481 · Cited by 96
Concept-Match Medical Data Scrubbing: How Pathology Text Can Be Used in Research
Berman, J. J. (2003) · Archives of Pathology & Laboratory Medicine · doi:10.5858/2003-127-680-cmds · Cited by 93
South, B. R., Mowery, D. L., Suo, Y., Leng, J., Ferrández, Ó., Meystre, S. M., & Chapman, W. W. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.05.002 · Cited by 74
Thomas, S. M., Mamlin, B. W., Schadow, G., & McDonald, C. J. (2002) · PubMed Central · Cited by 73
De-identification algorithm for free-text nursing notes
Douglass, M. M., Cliffford, G. D., Reisner, A. T., Long, W. J., Moody, G. B., & Mark, R. G. (2005) · Computers in cardiology · doi:10.1109/cic.2005.1588104 · Cited by 70
Computer-assisted de-identification of free text in the MIMIC II database
Douglass, M. M., Clifford, G. D., Reisner, A. T., Moody, G. B., & Mark, R. G. (2005) · Computers in cardiology · doi:10.1109/cic.2004.1442942 · Cited by 67
Recurrent Neural Networks for Word Alignment Model
Tamura, A., Watanabe, T., & Sumita, E. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1138 · Cited by 66
Non-lexical neural architecture for fine-grained POS Tagging
Labeau, M., Löser, K., & Allauzen, A. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1025 · Cited by 62
Assisted annotation of medical free text using RapTAT
Gobbel, G. T., Garvin, J. H., Reeves, R., Cronin, R. M., Heavirland, J., Williams, J., Weaver, A., Jayaramaraja, S., Giuse, D. A., Speroff, T., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002255 · Cited by 47
Morrison, F. P., Li, L., Lai, A. M., & Hripcsak, G. M. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2862 · Cited by 45
DesRoches, C. M., Worzala, C., & Bates, S. (2013) · Health Affairs · doi:10.1377/hlthaff.2013.0469 · Cited by 37
Bootstrapping a de-identification system for narrative patient records: Cost-performance tradeoffs
Hanauer, D. A., Aberdeen, J. S., Bayer, S., Wellner, B., Clark, C., Zheng, K., & Hirschman, L. (2013) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.03.005 · Cited by 36
Cited by
Deep learning for healthcare: review, opportunities and challenges
Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx044 · Cited by 3,076
“Consequently, the authors used the LSTM RNN model and identified three interesting subtypes for Parkinson’s disease, wherein each subtype demonstrates common disease progression trends.”
BackgroundInfluentialShickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688
Xiao, C., Choi, E., & Sun, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy068 · Cited by 792
CLAMP – a toolkit for efficiently building customized clinical natural language processing pipelines
Soysal, E., Wang, J., Jiang, M., Wu, Y., Pakhomov, S., Liu, H., & Xu, H. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx132 · Cited by 360
Neural Natural Language Processing for unstructured data in electronic health records: A review
Li, I., Pan, J., Goldwasser, J., Verma, N., Wong, W. P., Nuzumlalı, M. Y., Rosand, B., Li, Y., Zhang, M., Chang, D., et al. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100511 · Cited by 254
Metadata
- Authors
- Dernoncourt, F., Lee, J. Y., Uzuner, O., & Szolovits, P.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-12-31
- Volume
- 24
- Issue
- 3
- Pages
- 596-606
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2433185791
- Cited by
- 378
- References
- 49
- FWCI
- 31.81
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.