Automated de-identification of free-text medical records
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Background: Text-based patient medical records are a vital resource in medical research. In order to preserve patient confidentiality, however, the U.S. Health Insurance Portability and Accountability Act (HIPAA) requires that protected health information (PHI) be removed from medical records before they can be disseminated. Manual de-identification of large medical record databases is prohibitively expensive, time-consuming and prone to error, necessitating automatic methods for large-scale, automated de-identification. Methods: We describe an automated Perl-based de-identification software package that is generally usable on most free-text medical records, e.g., nursing notes, discharge summaries, X-ray reports, etc. The software uses lexical look-up tables, regular expressions, and simple heuristics to locate both HIPAA PHI, and an extended PHI set that includes doctors' names and years of dates. To develop the de-identification approach, we assembled a gold standard corpus of re-identified nursing notes with real PHI replaced by realistic surrogate information. This corpus consists of 2,434 nursing notes containing 334,000 words and a total of 1,779 instances of PHI taken from 163 randomly selected patient records. This gold standard corpus was used to refine the algorithm and measure its sensitivity. To test the algorithm on data not used in its development, we constructed a second test corpus of 1,836 nursing notes containing 296,400 words. The algorithm's false negative rate was evaluated using this test corpus. Results: Performance evaluation of the de-identification software on the development corpus yielded an overall recall of 0.967, precision value of 0.749, and fallout value of approximately 0.002. On the test corpus, a total of 90 instances of false negatives were found, or 27 per 100,000 word count, with an estimated recall of 0.943. Only one full date and one age over 89 were missed. No patient names were missed in either corpus. Conclusion: We have developed a pattern-matching de-identification system based on dictionary look-ups, regular expressions, and heuristics. Evaluation based on two different sets of nursing notes collected from a U.S. hospital suggests that, in terms of recall, the software out-performs a single human de-identifier (0.81) and performs at least as well as a consensus of two human de-identifiers (0.94). The system is currently tuned to de-identify PHI in nursing notes and discharge summaries but is sufficiently generalized and can be customized to handle text files of any format. Although the accuracy of the algorithm is high, it is probably insufficient to be used to publicly disseminate medical data. The open-source de-identification software and the gold standard re-identified corpus of medical records have therefore been made available to researchers via the PhysioNet website to encourage improvements in the algorithm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Finally, we combined two published algorithms for removing PHI from free-text17,18.”
From MIMIC-IV, a freely accessible electronic health record dataset (2023) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Self-citationThe Unified Medical Language System
Humphreys, B. L., McCray, A. T., & Lindberg, D. A. B. (1993) · Methods of Information in Medicine · doi:10.1055/s-0038-1634945 · Cited by 1,182
Proceedings of the AMIA Symposium
Schulz, S., Daumke, P., Smith, B., & Hahn, U. (2005) · Cited by 606
Evaluating the State-of-the-Art in Automatic De-identification
Uzuner, O., Luo, Y., & Szolovits, P. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2444 · Cited by 452 · cited 2×
“A very recent development has been a competition run at the first Workshop on Challenges in Natural Language Processing for Clinical Data to de-identify discharge summary free-text data [27].”
Self-citationStandards for privacy of individually identifiable health information. Final rule.
(2002) · PubMed · Cited by 400
Saeed, M., Lieu, C., Raber, G., & Mark, R. G. (2003) · Computers in Cardiology · doi:10.1109/cic.2002.1166854 · Cited by 271
Self-citationReplacing Personally-Identifying Information in Medical Records, the Scrub System
Sweeney, L. (2018) · PubMed · doi:10.1184/r1/6625565 · Cited by 251
Hypertrophic Obstructive Cardiomyopathy
Nishimura, R. A., & Holmes, D. R. (2004) · New England Journal of Medicine · doi:10.1056/nejmcp030779 · Cited by 236
Guaranteeing anonymity when sharing medical data, the Datafly system
Sweeney, L. (2018) · PubMed · doi:10.1184/r1/6622412 · Cited by 235
Federal Policy for the Protection of Human Subjects. Final rule.
(1991) · PubMed · Cited by 234
Gupta, D., Saul, M., & Gilbertson, J. (2004) · American Journal of Clinical Pathology · doi:10.1309/e6k3-3gbp-e5c2-7fyu · Cited by 165
State-of-the-art Anonymization of Medical Records Using an Iterative Machine Learning Framework
Szarvas, G., Farkas, R., & Busa‐Fekete, R. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/j.jamia.m2441 · Cited by 161 · cited 3×
“Evaluation of our system using discharge summaries similar to those used in [ 27-29 ], as described in this article, will allow a more meaningful comparison between our approach and others, especially if a common corpus can be used to evaluate multiple algorithms.”
InfluentialComputational disclosure control: a primer on data privacy protection
Sweeney, L., & Abelson, H. (2001) · Cited by 138
Rapidly Retargetable Approaches to De-identification in Medical Records
Wellner, B., Huyck, M., Mardis, S., Aberdeen, J. S., Morgan, A. A., Peshkin, L., Yeh, A., Hitzeman, J. M., & Hirschman, L. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2435 · Cited by 128 · cited 2×
“[29], with recall and precision performance in the range of 96%– 98% and 98%–99% respectively.”
Development and evaluation of an open source software tool for deidentification of pathology reports
Beckwith, B. A., Mahaadevan, R., Balis, U. J., & Kuo, F. C. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-12 · Cited by 110 · cited 3×
“A de-identification system similar to ours is the one developed by Beckwith [16] was tested on a pathology report corpus containing 3,499 PHI identifiers and was found to remove all identifying words in pathology reports with a sensitivity of 98.”
InfluentialMedical document anonymization with a semantic lexicon.
Ruch, P., Baud, R. H., Rassinoux, A. M., Bouillon, P., & Robert, G. (2000) · PubMed Central · Cited by 101
Concept-Match Medical Data Scrubbing: How Pathology Text Can Be Used in Research
Berman, J. J. (2003) · Archives of Pathology & Laboratory Medicine · doi:10.5858/2003-127-680-cmds · Cited by 93
State-of-the-art anonymisation of medical records using an iterative machine learning framework
Szarvas, G., Farkas, R., & Busa‐Fekete, R. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2441 · Cited by 85
Taira, R. K., Bui, A. A. T., & Kangarloo, H. (2002) · PubMed · Cited by 82
Thomas, S. M., Mamlin, B. W., Schadow, G., & McDonald, C. J. (2002) · PubMed Central · Cited by 73
De-identification algorithm for free-text nursing notes
Douglass, M. M., Cliffford, G. D., Reisner, A. T., Long, W. J., Moody, G. B., & Mark, R. G. (2005) · Computers in cardiology · doi:10.1109/cic.2005.1588104 · Cited by 70 · cited 3×
“A corpus of nursing notes was thoroughly de-identified manually and then re-identified with surrogate PHI (similar in nature to each PHI type removed) to create a gold standard for developing and testing de-identification software [3,4,10].”
InfluentialSelf-citationComputer-assisted de-identification of free text in the MIMIC II database
Douglass, M. M., Clifford, G. D., Reisner, A. T., Moody, G. B., & Mark, R. G. (2005) · Computers in cardiology · doi:10.1109/cic.2004.1442942 · Cited by 67 · cited 6×
“In prior studies, we have demonstrated that manual de-identification by medical professionals is prohibitively timeconsuming, expensive [3], and unreliable [ 4 ].”
InfluentialSelf-citationPhysioNet : Research Resource for Complex Physiologic Signals
George, B. (2009) · 心電図 = Electrocardiology · Cited by 29
Syntactically-informed semantic category recognition in discharge summaries.
Sibanda, T. C., He, T. Y., Peter Szolovits, B., & Uzuner, Ö. (2006) · Europe PMC (PubMed Central) · Cited by 29
Was the Patient Cured? Understanding Semantic Categories and Their Relationships in Patient Records
Sibanda, T. C. (2006) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 21
Grace, M. (2003) · BDJ · doi:10.1038/sj.bdj.4810393 · Cited by 20
Haas, H., & Kraushaar-Baldauf, E. (1949) · PubMed · Cited by 16
Namini Ahmad, H., Berkowicz, D., Kohane, I. S., & Chueh, H. C. (2004) · Studies in health technology and informatics · doi:10.3233/978-1-60750-949-3-1264 · Cited by 15
Computer-Assisted De-Identification of Free-text Nursing Notes
Douglass, M. M. (2005) · Cited by 11
Self-citationDe-identification of ICU patient records
Levine, J. M. (2003) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 7
Research works printed but not linked: 3
- Gupta D, Saul M, Gilbertson J: Evaluation of a de-identification software engine: Progress towards sharing clinical documents and pathology reports. Am J Clin Pathol. 2004, 121 (2): 176-186. · doi:10.1309/e6k33gbpe5c27fyu · Paper page
- Miller R, Boitnott JK, Moore GW: Web-Based Free-Text Query System for Surgical Pathology Reports with Automatic Case Deidentification. Arch Pathol Lab Med. 2001, 125: 1011-
- Berman JJ: Ruby Programming for Medicine and Biology. 2008, Jones & Bartlett Publishers, Sudbury, MA, 11: 157-163.
Cited by
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Self-citationMIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401
Self-citationPollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018) · Scientific Data · doi:10.1038/sdata.2018.178 · Cited by 1,993
Self-citationMIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
Self-citationSaeed, M., Villarroel, M., Reisner, A. T., Clifford, G., Lehman, L.-W., Moody, G., Heldt, T., Kyaw, T. H., Moody, B., & Mark, R. G. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e31820a92c6 · Cited by 1,162
“For example, electronic algorithms that automatically extract information from free-text nursing notes and discharge summaries have been developed and tested (7).”
Self-citation
Metadata
- Authors
- Neamatullah, I., Douglass, M. M., Lehman, L.-W. H., Reisner, A., Villarroel, M., Long, W. J., Szolovits, P., Moody, G. B., Mark, R. G., & Clifford, G. D.
- Affiliations
- Massachusetts Institute of Technology (US)
- Harvard–MIT Division of Health Sciences and Technology (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Corresponding author
- Lehman, Li-wei H
- Clifford, Gari D
- ORCID
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2008-07-24
- Volume
- 8
- Issue
- 1
- Pages
- 32-32
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC2526997
- OpenAlex
- W1995228216
- Cited by
- 413
- References
- 30
- Citations by year
- 2026: 25 · 2025: 26 · 2024: 27 · 2023: 30 · 2022: 24 · 2021: 32 · 2020: 32 · 2019: 21 · 2018: 23 · 2017: 27 · 2016: 23 · 2015: 26 · 2014: 21 · 2013: 15 · 2012: 25
- FWCI
- 18.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- National Institutes of Health
- National Institute of Biomedical Imaging and Bioengineering (R01EB001659, R01-EB017205, EB001659)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Health Sciences › Health Professions › Health Information Management › Thoreau and American Literature
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- protected health information, HIPAA, nursing notes, discharge summaries, pattern matching, regular expressions, re-identification, patient confidentiality, gold standard corpus, false negative rate, natural language processing
- MeSH terms
- Algorithms, Confidentiality, Dictionaries as Topic, Humans, Medical Records, Natural Language Processing, Patient Discharge, Programming Languages, Software, United States, Health Insurance Portability and Accountability Act
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
