The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Evaluating the State-of-the-Art in Automatic De-identification
Open accessAbstract
To facilitate and survey studies in automatic de-identification, as a part of the i2b2 (Informatics for Integrating Biology to the Bedside) project, authors organized a Natural Language Processing (NLP) challenge on automatically removing private health information (PHI) from medical discharge records. This manuscript provides an overview of this de-identification challenge, describes the data and the annotation process, explains the evaluation metrics, discusses the nature of the systems that addressed the challenge, analyzes the results of received system runs, and identifies directions for future research. The de-indentification challenge data consisted of discharge summaries drawn from the Partners Healthcare system. Authors prepared this data for the challenge by replacing authentic PHI with synthesized surrogates. To focus the challenge on non-dictionary-based de-identification methods, the data was enriched with out-of-vocabulary PHI surrogates, i.e., made up names. The data also included some PHI surrogates that were ambiguous with medical non-PHI terms. A total of seven teams participated in the challenge. Each team submitted up to three system runs, for a total of sixteen submissions. The authors used precision, recall, and F-measure to evaluate the submitted system runs based on their token-level and instance-level performance on the ground truth. The systems with the best performance scored above 98% in F-measure for all categories of PHI. Most out-of-vocabulary PHI could be identified accurately. However, identifying ambiguous PHI proved challenging. The performance of systems on the test data set is encouraging. Future evaluations of these systems will involve larger data sets from more heterogeneous sources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
(1992) · Cited by 23,431
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Message Understanding Conference-6: a brief history
Grishman, R., & Sundheim, B. (1996) · Proceedings of the 16th conference on Computational linguistics - · doi:10.3115/992628.992709 · Cited by 1,408
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,118
Agreement, the F-Measure, and Reliability in Information Retrieval
Hripcsak, G. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1733 · Cited by 949
Boutet, E., Lieberherr, D., Tognolli, M., Schneider, M., & Bairoch, A. (2007) · Plant Bioinformatics · doi:10.1007/978-1-59745-535-0_4 · Cited by 763
Issel-Tarver, L., Christie, K. R., Dolinski, K., Andrada, R., Balakrishnan, R., Ball, C. A., Binkley, G., Dong, S., Dwight, S. S., Fisk, D. G., et al. (2002) · Methods in Enzymology · doi:10.1016/s0076-6879(02)50972-1 · Cited by 672
Cited by
2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,252
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Savova, G. K., Kipper-Schuler, K. C., Hurdle, J. F., & Meystre, S. M. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638592 · Cited by 895
Privacy in the Internet of Things: threats and challenges
Ziegeldorf, J. H., Morchon, O. G., & Wehrle, K. (2014) · Security and Communication Networks · doi:10.1002/sec.795 · Cited by 610
The Co-Morbidity Burden of Children and Young Adults with Autism Spectrum Disorders
Kohane, I. S., McMurry, A., Weber, G., MacFadden, D., Rappaport, L., Kunkel, L., Bickel, J., Wattanasin, N., Spence, S., Murphy, S., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033224 · Cited by 562
Evaluating temporal relations in clinical text: 2012 i2b2 Challenge
Sun, W., Rumshisky, A., & Uzuner, O. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001628 · Cited by 473
Self-citationExtracting medication information from clinical text
Uzuner, Ö., Solti, I., & Cadag, E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.003947 · Cited by 466
A Survey on Recent Advances in Named Entity Recognition from Deep Learning models
Yadav, V., & Bethard, S. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11470 · Cited by 464
Automated de-identification of free-text medical records
Neamatullah, I., Douglass, M. M., Lehman, L.-W. H., Reisner, A., Villarroel, M., Long, W. J., Szolovits, P., Moody, G. B., Mark, R. G., & Clifford, G. D. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-32 · Cited by 413
Self-citationCLAMP – a toolkit for efficiently building customized clinical natural language processing pipelines
Soysal, E., Wang, J., Jiang, M., Wu, Y., Pakhomov, S., Liu, H., & Xu, H. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx132 · Cited by 360
Identifying Patient Smoking Status from Medical Discharge Records
Uzuner, O., Goldstein, I., Luo, Y., & Kohane, I. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2408 · Cited by 359
Self-citationMeystre, S. M., Friedlin, F. J., South, B. R., Shen, S., & Samore, M. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-70 · Cited by 329
Clinical Natural Language Processing in languages other than English: opportunities and challenges
Névéol, A., Dalianis, H., Velupillai, S., Savova, G., & Zweigenbaum, P. (2018) · Journal of Biomedical Semantics · doi:10.1186/s13326-018-0179-8 · Cited by 311
2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records
Henry, S., Buchan, K., Filannino, M., Stubbs, A., & Uzuner, O. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz166 · Cited by 303
Self-citationChapman, W. W., Nadkarni, P. M., Hirschman, L., D'Avolio, L. W., Savova, G. K., & Uzuner, O. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000465 · Cited by 296
Neural Natural Language Processing for unstructured data in electronic health records: A review
Li, I., Pan, J., Goldwasser, J., Verma, N., Wong, W. P., Nuzumlalı, M. Y., Rosand, B., Li, Y., Zhang, M., Chang, D., et al. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100511 · Cited by 254
Clinical Data Reuse or Secondary Use: Current Status and Potential Future Progress
Meystre, S. M., Lovis, C., Bürkle, T., Tognola, G., Budrionis, A., & Lehmann, C. U. (2017) · Yearbook of Medical Informatics · doi:10.15265/iy-2017-007 · Cited by 204
Evaluating the state of the art in coreference resolution for electronic medical records
Uzuner, O., Bodnari, A., Shen, S., Forbush, T., Pestian, J., & South, B. R. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000784 · Cited by 164
Self-citationPatient privacy in the era of big data
Kayaalp, M. (2018) · Balkan Medical Journal · doi:10.4274/balkanmedj.2017.0966 · Cited by 142
A study of deep learning methods for de-identification of clinical notes in cross-institute settings
Yang, X., Lyu, T., Li, Q., Lee, C.-Y., Bian, J., Hogan, W. R., & Wu, Y. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0935-4 · Cited by 73
Deidentification of free-text medical records using pre-trained bidirectional transformers
Johnson, A. E. W., Bulgarelli, L., & Pollard, T. J. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384455 · Cited by 55
Clay, B., Bergman, H. I., Salim, S., Pergola, G., Shalhoub, J., & Davies, A. H. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109808 · Cited by 41
A scoping review of publicly available language tasks in clinical natural language processing
Gao, Y., Dligach, D., Christensen, L., Tesch, S., Laffin, R., Xu, D., Miller, T., Uzuner, O., Churpek, M. M., & Afshar, M. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac127 · Cited by 28
Self-citationAlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs.
Ghosh, M., Mukherjee, S., Ganguly, A., Basuchowdhuri, P., Naskar, S. K., & Ganguly, D. (2024) · Methods · doi:10.1016/j.ymeth.2024.04.005 · Cited by 27
“Additionally, to enhance the overall performance of the clinical trial task, the biomedical literature has witnessed a significant number of state-of-the-art (SOTA) pretrained models (PLMs) such as BNER [12, 13] to conduct the biomedical named entity recognition task.”
BackgroundBigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Fries, J. A., Weber, L., Seelam, N., Altay, G., Datta, D., Garda, S., Kang, M., Su, R., Kusa, W., Cahyawijaya, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.15076 · Cited by 17
Hoogenboom, W. S., Perlis, R. H., Smoller, J. W., Zeng-Treitler, Q., Gainer, V. S., Murphy, S. N., Churchill, S. E., Kohane, I. S., Shenton, M. E., & Iosifescu, D. V. (2013) · Psychiatry Research: Neuroimaging · doi:10.1016/j.pscychresns.2012.07.007 · Cited by 13
Metadata
- Authors
- Uzuner, O., Luo, Y., & Szolovits, P.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-09-01
- Volume
- 14
- Issue
- 5
- Pages
- 550-563
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2160987310
- Cited by
- 452
- References
- 38
- FWCI
- 20.49
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.