Deidentification of free-text medical records using pre-trained bidirectional transformers
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
The ability of caregivers and investigators to share patient data is fundamental to many areas of clinical practice and biomedical research. Prior to sharing, it is often necessary to remove identifiers such as names, contact details, and dates in order to protect patient privacy. Deidentification, the process of removing identifiers, is challenging, however. High-quality annotated data for developing models is scarce; many target identifiers are highly heterogenous (for example, there are uncountable variations of patient names); and in practice anything less than perfect sensitivity may be considered a failure. As a result, patient data is often withheld when sharing would be beneficial, and identifiable patient data is often divulged when a deidentified version would suffice. In recent years, advances in machine learning methods have led to rapid performance improvements in natural language processing tasks, in particular with the advent of large-scale pretrained language models. In this paper we develop and evaluate an approach for deidentification of clinical notes based on a bidirectional transformer model. We propose human interpretable evaluation measures and demonstrate state of the art performance against modern baseline models. Finally, we highlight current challenges in deidentification, including the absence of clear annotation guidelines, lack of portability of models, and paucity of training data. Code to develop our model is open source, allowing for broad reuse.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Finally, we combined two published algorithms for removing PHI from free-text17,18.”
From MIMIC-IV, a freely accessible electronic health record dataset (2023) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Briefly, our model consists of L identical layers applied sequentially, where each layer is a single transformer block [37].”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“1 Model We adopt the model architecture used in BERT [8] with additional components for performing named entity recognition.”
MethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“The advent of large, open access text corpuses and the resurgence of neural networks has driven advances in state-of-the-art model performance in natural language processing [10, 27].”
BackgroundPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“New models including RoBERTa [19], ALBERT [15], and XLNet [41] have demonstrated enhanced performance in several transfer learning tasks, and so could be expected to offer performance gains in deidentification.”
MethodTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Self-citationBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“As BERT was trained using a general language corpus, we evaluate model performance when using pretrained weights from scientific corpora (SciBERT) or biomedical corpora (BioBERT) [3, 17].”
MethodXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“New models including RoBERTa [19], ALBERT [15], and XLNet [41] have demonstrated enhanced performance in several transfer learning tasks, and so could be expected to offer performance gains in deidentification.”
MethodGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 1×
“New models including RoBERTa [19], ALBERT [15], and XLNet [41] have demonstrated enhanced performance in several transfer learning tasks, and so could be expected to offer performance gains in deidentification.”
MethodHuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · cited 1×
“3.0 1 [39].”
InfluentialSciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 1×
“As BERT was trained using a general language corpus, we evaluate model performance when using pretrained weights from scientific corpora (SciBERT) or biomedical corpora (BioBERT) [3, 17].”
MethodPublicly Available Clinical BERT Embeddings
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 693 · cited 1×
“We did not use versions of BERT fine-tuned to clinical corpora, such as ClinicalBERT, as these are trained using deidentified text and are known to suffer weaker performance for the task of deidentification itself [2].”
MethodEvaluating the State-of-the-Art in Automatic De-identification
Uzuner, O., Luo, Y., & Szolovits, P. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2444 · Cited by 452
Automated de-identification of free-text medical records
Neamatullah, I., Douglass, M. M., Lehman, L.-W. H., Reisner, A., Villarroel, M., Long, W. J., Szolovits, P., Moody, G. B., Mark, R. G., & Clifford, G. D. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-32 · Cited by 413
De-identification of patient notes with recurrent neural networks
Dernoncourt, F., Lee, J. Y., Uzuner, O., & Szolovits, P. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw156 · Cited by 378 · cited 7×
“demonstrated that exceptional performance could be achieved by combining recurrent neural networks (RNNs) with CRFs and training the model end-toend [7].”
BackgroundMethodInfluentialStubbs, A., Kotfila, C., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.007 · Cited by 245 · cited 2×
“In 2006 and 2014, for example, the National NLP Clinical (n2c2, formerly known as i2b2) Challenges focused on automated systems for deidentification of clinical text [32, 33, 36].”
BackgroundDo NLP Models Know Numbers? Probing Numeracy in Embeddings
Wallace, E., Wang, Y., Li, S., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1534 · Cited by 227 · cited 1×
“Despite training on a large corpus of text, BERT based models struggle with numeracy, in particular due to the poor representation of large numbers in sub-word tokenization [38].”
BackgroundAnnotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus
Stubbs, A., & Uzuner, Ö. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.07.020 · Cited by 215 · cited 2×
“We highlight difficulties in cross-dataset comparisons using a re-annotated version of the PhysioNet corpus which assigns entities according to the guidelines set forth by the organizers of the i2b2 2014 challenge [11, 33].”
BackgroundMethodWhy is my classifier discriminatory?
Chen, I., Johansson, F. D., & Sontag, D. (2018) · arXiv · doi:10.48550/arxiv.1805.12002 · Cited by 185 · cited 1×
“This paucity of data is to the detriment of important issues including bias, generalizability, and reproducibility [5].”
BackgroundThe MITRE Identification Scrubber Toolkit: Design, training, and assessment
Aberdeen, J. S., Bayer, S., Yeniterzi, R., Wellner, B., Clark, C., Hanauer, D. A., Malin, B., & Hirschman, L. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.09.007 · Cited by 175 · cited 1×
“who tailored their algorithm to the training data using new lexicons [1].”
MethodNeuroNER: an easy-to-use program for named-entity recognition based on neural networks
Dernoncourt, F., Lee, J. Y., & Szolovits, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d17-2017 · Cited by 161 · cited 2×
“adapted the open source implementation NeuroNER [6], which is based upon the work of Dernoncourt et al.”
MethodDe-identification of clinical notes via recurrent neural network and conditional random field
Liu, Z., Tang, B., Wang, X., & Chen, Q. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.05.023 · Cited by 152 · cited 3×
“We compare the performance of our models to state of the art models developed using the i2b2 2014 challenge dataset [7, 11, 21].”
MethodStockdale, J., Cassell, J., & Ford, E. (2019) · Wellcome Open Research · doi:10.12688/wellcomeopenres.13531.2 · Cited by 124 · cited 1×
“Enquiries into public views on the use of patient data for research broadly suggest that there is a willingness to share data where it is for the common good [9, 30].”
BackgroundDe-identification of psychiatric intake records: Overview of 2016 CEGS N-GRID shared tasks Track 1
Stubbs, A., Filannino, M., & Uzuner, Ö. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.011 · Cited by 105 · cited 4×
“Consequently, they are often somewhat rigid and generalize weakly beyond their development environment [31].”
BackgroundInfluentialLiu, Z., Chen, Y., Tang, B., Wang, X., Chen, Q., Li, H., Wang, J., Deng, Q., & Zhu, S. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.06.009 · Cited by 79 · cited 1×
“Liu et al., for example, combined a character level CRF, a token level CRF, and a bidirectional Long Short Term Memory (LSTM) network [20].”
BackgroundStockdale, J., Cassell, J., & Ford, E. (2018) · Wellcome Open Research · doi:10.12688/wellcomeopenres.13531.1 · Cited by 76
Customization scenarios for de-identification of clinical notes
Hartman, T., Howell, M. D., Dean, J., Hoory, S., Slyper, R., Laish, I., Gilon, O., Vainstein, D., Corrado, G. S., Chou, K., et al. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1026-2 · Cited by 57 · cited 3×
“We compare the performance of our models to state of the art models developed using the i2b2 2014 challenge dataset [7, 11, 21].”
BackgroundMethodA unified framework for evaluating the risk of re-identification of text de-identification tools
Scaiano, M., Middleton, G., Arbuckle, L., Kolhatkar, V., Peyton, L., Dowling, M., Gipson, D. S., & Emam, K. E. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.07.015 · Cited by 43
A hybrid approach to automatic de-identification of psychiatric notes
Lee, H.-J., Wu, Y., Zhang, Y., Xu, J., Xu, H., & Roberts, K. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.006 · Cited by 42 · cited 1×
“used rule based approaches along with a CRF, and incorporated additional data using domain adaptation, along with rules for correcting errors in final predictions [16].”
MethodSciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · arXiv · doi:10.48550/arxiv.1903.10676 · Cited by 41 · cited 1×
“As BERT was trained using a general language corpus, we evaluate model performance when using pretrained weights from scientific corpora (SciBERT) or biomedical corpora (BioBERT) [3, 17].”
MethodA survey of automatic de-identification of longitudinal clinical narrativesPreprint
Yogarajan, V., Mayo, M., & Pfahringer, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06765 · Cited by 13
For the greater good? Patient and public attitudes to use of medical free text data in research
Ford, E. M., Stockdale, J., Jackson, R., & Cassell, J. (2017) · International Journal for Population Data Science · doi:10.23889/ijpds.v1i1.249 · Cited by 5 · cited 1×
“Enquiries into public views on the use of patient data for research broadly suggest that there is a willingness to share data where it is for the common good [9, 30].”
BackgroundAnnotating longitudinal clinical narratives for de-identification
StubbsAmber, & UzunerÖzlem (2015) · Journal of Biomedical Informatics · doi:10.5555/2869975.2870333 · Cited by 4
Custodes: Auditable Hypothesis Testing.Preprint
Servan-Schreiber, S., Ohrimenko, O., Kraska, T., & Zgraggen, E. (2019) · arXiv (Cornell University) · Cited by 2
Research works printed but not linked: 13
- 10.5555/3327144.3327272 · doi:10.5555/3327144.3327272
- Halevy Alon, 2009, The Unreasonable Effectiveness of Data. Intelligent Systems, IEEE, 24, 8
- Johnson Alistair EW, 2016, Leo Anthony Celi, and Roger G Mark
- 10.13026/0757-0y85 · doi:10.13026/0757-0y85
- Lafferty John D., Proceedings of the Eighteenth International Conference on Machine Learning (ICML '01)
- 10.1093/bioinformatics/btz682 · DOI · Paper page
- Paszke Adam, 2017, Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration.
- Paszke Adam, PyTorch: An Imperative Style
- Peters Matthew E, 2018, Deep contextualized word representations. arXiv preprint arXiv:1802.05365 · arXiv
- Sacha, 2019, Auditable Hypothesis Testing. arXiv
- 10.5555/3169406.3169752 · doi:10.5555/3169406.3169752
- Employee Benefits Security Administration U.S. Dept., 2004, of Labor
- Yang Zhilin, 2019, Le
Cited by
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401
“Finally, we combined two published algorithms for removing PHI from free-text 17,18 .”
MethodSelf-citationAMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
Transformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 172
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?
Lehman, E., Jain, S., Pichotta, K., Goldberg, Y., & Wallace, B. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.73 · Cited by 93
Pilán, I., Lison, P., Øvrelid, L., Papadopoulou, A. I., Sánchez, D., & Batet, M. (2022) · Computational Linguistics · doi:10.1162/coli_a_00458 · Cited by 227
Metadata
- Authors
- Johnson, A. E. W., Bulgarelli, L., & Pollard, T. J.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- Proceedings of the ACM Conference on Health, Inference, and Learning
- Publisher
- ACM
- Published
- 2020-04-02
- Volume
- 2020
- Pages
- 214-221
- PMC
- PMC8330601
- OpenAlex
- W3012137864
- Cited by
- 55
- References
- 33
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 7 · 2023: 8 · 2022: 5 · 2021: 10 · 2020: 2
- FWCI
- 2.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, other-oa)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by-nc)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- NIH (NIH-R01-EB017205)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Thoreau and American Literature
- Keywords
- de-identification, clinical notes, patient identifiers, bidirectional transformer, pre-trained language models, patient privacy, data sharing, annotation guidelines, model portability, limited training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.