The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Evaluating the State-of-the-Art in Automatic De-identification
Open accessAbstract
To facilitate and survey studies in automatic de-identification, as a part of the i2b2 (Informatics for Integrating Biology to the Bedside) project, authors organized a Natural Language Processing (NLP) challenge on automatically removing private health information (PHI) from medical discharge records. This manuscript provides an overview of this de-identification challenge, describes the data and the annotation process, explains the evaluation metrics, discusses the nature of the systems that addressed the challenge, analyzes the results of received system runs, and identifies directions for future research. The de-indentification challenge data consisted of discharge summaries drawn from the Partners Healthcare system. Authors prepared this data for the challenge by replacing authentic PHI with synthesized surrogates. To focus the challenge on non-dictionary-based de-identification methods, the data was enriched with out-of-vocabulary PHI surrogates, i.e., made up names. The data also included some PHI surrogates that were ambiguous with medical non-PHI terms. A total of seven teams participated in the challenge. Each team submitted up to three system runs, for a total of sixteen submissions. The authors used precision, recall, and F-measure to evaluate the submitted system runs based on their token-level and instance-level performance on the ground truth. The systems with the best performance scored above 98% in F-measure for all categories of PHI. Most out-of-vocabulary PHI could be identified accurately. However, identifying ambiguous PHI proved challenging. The performance of systems on the test data set is encouraging. Future evaluations of these systems will involve larger data sets from more heterogeneous sources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
(1992) · Cited by 23,431
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Message Understanding Conference-6: a brief history
Grishman, R., & Sundheim, B. (1996) · Proceedings of the 16th conference on Computational linguistics - · doi:10.3115/992628.992709 · Cited by 1,408
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,118
Agreement, the F-Measure, and Reliability in Information Retrieval
Hripcsak, G. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1733 · Cited by 949
Boutet, E., Lieberherr, D., Tognolli, M., Schneider, M., & Bairoch, A. (2007) · Plant Bioinformatics · doi:10.1007/978-1-59745-535-0_4 · Cited by 763
Issel-Tarver, L., Christie, K. R., Dolinski, K., Andrada, R., Balakrishnan, R., Ball, C. A., Binkley, G., Dong, S., Dwight, S. S., Fisk, D. G., et al. (2002) · Methods in Enzymology · doi:10.1016/s0076-6879(02)50972-1 · Cited by 672
Computer Intensive Methods for Testing Hypotheses: An Introduction
Evans, K. J., & Noreen, E. W. (1990) · Journal of the Operational Research Society · doi:10.2307/2582908 · Cited by 651
GATE, a General Architecture for Text Engineering
Cunningham, H. (2002) · Computers and the Humanities · doi:10.1023/a:1014348124664 · Cited by 643
Ck, F. (1970) · PubMed · Cited by 641
Automated Encoding of Clinical Documents Based on Natural Language Processing
Friedman, C., Shagina, L., Lussier, Y., & Hripcsak, G. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1552 · Cited by 551
Overview of BioCreAtIvE: critical assessment of information extraction for biology
Hirschman, L., Yeh, A., Blaschke, C., & Valencia, A. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s1-s1 · Cited by 530
Miller, R. A. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236141 · Cited by 458
Sang, E. F. T. K., & Veenstra, J. (1999) · Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics - · doi:10.3115/977035.977059 · Cited by 456
Identifying Patient Smoking Status from Medical Discharge Records
Uzuner, O., Goldstein, I., Luo, Y., & Kohane, I. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2408 · Cited by 359
Self-citationBerner, E. S. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1664 · Cited by 340
Replacing Personally-Identifying Information in Medical Records, the Scrub System
Sweeney, L. (2018) · PubMed · doi:10.1184/r1/6625565 · Cited by 251
Recognizing names in biomedical texts: a machine learning approach
Zhou, G., Zhang, J., Su, J., Shen, D., & Tan, C. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth060 · Cited by 251
Natural language processing and its future in medicine
Friedman, C., & Hripcsak, G. (1999) · Academic Medicine · doi:10.1097/00001888-199908000-00012 · Cited by 179
Gupta, D., Saul, M., & Gilbertson, J. (2004) · American Journal of Clinical Pathology · doi:10.1309/e6k3-3gbp-e5c2-7fyu · Cited by 165
State-of-the-art Anonymization of Medical Records Using an Iterative Machine Learning Framework
Szarvas, G., Farkas, R., & Busa‐Fekete, R. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/j.jamia.m2441 · Cited by 161
Chapman, W. W., Christensen, L. M., D, M. M. W. M., Haug, P. J., Ivanov, O., Dowling, J., & Olszewski, R. T. (2004) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2004.04.001 · Cited by 146
A Boosting Algorithm for Classification of Semi - Structured Text
Kudo, T., & Matsumoto, Y. (2004) · Cited by 143
Friedman, C., Hripcsak, G. M., Shagina, L., & Liu, H. (1999) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1999.0060076 · Cited by 129
Rapidly Retargetable Approaches to De-identification in Medical Records
Wellner, B., Huyck, M., Mardis, S., Aberdeen, J. S., Morgan, A. A., Peshkin, L., Yeh, A., Hitzeman, J. M., & Hirschman, L. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2435 · Cited by 128
Development and evaluation of an open source software tool for deidentification of pathology reports
Beckwith, B. A., Mahaadevan, R., Balis, U. J., & Kuo, F. C. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-12 · Cited by 110
Medical document anonymization with a semantic lexicon.
Ruch, P., Baud, R. H., Rassinoux, A. M., Bouillon, P., & Robert, G. (2000) · PubMed Central · Cited by 101
Mitigating the paucity-of-data problem
Banko, M., & Brill, E. (2001) · International Conference on Human Language Technology Research · doi:10.3115/1072133.1072204 · Cited by 96
Huang, Y. H. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1695 · Cited by 80
Rollman, B. L., Hanusa, B. H., Gilbert, T., Lowe, H., Kapoor, W. N., & Schulberg, H. C. (2001) · Archives of Internal Medicine · doi:10.1001/archinte.161.2.189 · Cited by 75
Computer-assisted de-identification of free text in the MIMIC II database
Douglass, M. M., Clifford, G. D., Reisner, A. T., Moody, G. B., & Mark, R. G. (2005) · Computers in cardiology · doi:10.1109/cic.2004.1442942 · Cited by 67
Second i2b2 workshop on natural language processing challenges for clinical records.
Uzuner, Ö. (2008) · PubMed · Cited by 63
The statistical significance of the MUC-5 results
Chinchor, N. A. (1993) · Message Understanding Conference (MUC) · doi:10.3115/1072017.1072027 · Cited by 57
Cao, H., Stetson, P. D., & Hripcsak, G. M. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00058-3 · Cited by 28
The Second i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data
Uzuner, Ö., Ware, H., Mullett, C. J., et al. (2008) · Cited by 11
Cited by
2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,252
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Savova, G. K., Kipper-Schuler, K. C., Hurdle, J. F., & Meystre, S. M. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638592 · Cited by 895
Privacy in the Internet of Things: threats and challenges
Ziegeldorf, J. H., Morchon, O. G., & Wehrle, K. (2014) · Security and Communication Networks · doi:10.1002/sec.795 · Cited by 610
Metadata
- Authors
- Uzuner, O., Luo, Y., & Szolovits, P.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-09-01
- Volume
- 14
- Issue
- 5
- Pages
- 550-563
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2160987310
- Cited by
- 452
- References
- 38
- FWCI
- 20.49
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.