The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Identifying Patient Smoking Status from Medical Discharge Records
Open accessAbstract
Clinical narrative records contain much useful information. However, most clinical narratives are in the form of fragmented English free text, showing the characteristics of a clinical sublanguage. This makes their linguistic processing, search, and retrieval challenging.1 Traditional natural language processing (NLP) tools are not designed for the fragmented free text found in narrative clinical records; therefore, they do not perform well on this type of data.2 Limited access to clinical records has been a barrier to the widespread development of medical language processing (MLP) technologies. In the absence of a standardized, publicly available ground truth that encourages the development of MLP systems and allows their head-to-head comparison, successful MLP efforts have been limited, e.g., MedLEE3 and Symtxt.4 A few MLP systems have been developed,5 and such efforts have successfully shown the usefulness of MLP in clinical settings.6–8 To improve the availability of clinical records and to contribute to the advancement of the state of the art in MLP, within the i2b2 (Informatics for Integrating Biology to the Bedside) project, the authors de-identified and released a set of clinical records from Partners HealthCare. These records provided the basis for the development of ground truth for two challenge questions: Automatic de-identification of clinical data, i.e., de-identification challenge. Automatic evaluation of the smoking status of patients based on medical records, i.e., smoking challenge. Representative teams from the MLP community participated in the two challenges and met at a workshop organized by the authors to discuss the results of the challenges. The workshop was co-sponsored by the American Medical Informatics Association and met in conjunction with its Fall Symposium in November 2006. This article provides an overview of the smoking challenge and the findings of the workshop. An overview of the de-identification challenge can be found in Uzuner et al.9 The smoking challenge continues the tradition of attempting to identify the state of the art in automatic language processing. Outside of the medical domain, there have been many efforts in this direction. Most of these efforts have been led by Message Understanding Conferences (MUC)10 and the National Institute of Standards and Technology (NIST).11 MUC organized shared tasks on named entity recognition. NIST organized a series of Text Retrieval Evaluation Conferences (TREC) on various domains including blogs and legal documents; they also organized a series of shared tasks on topic detection and tracking, speaker recognition, language recognition, spoken document retrieval, machine translation, and entity extraction. In the biomedical domain, three such prominent efforts were BioCreAtIvE12 for information extraction, ImageCLEF13,14 for image retrieval, and TREC Genomics15 for question answering and information retrieval. Keeping the goals of TREC,16 MUC,17 BioCreAtIvE,18 etc., in mind for the smoking challenge, we created a collection of actual medical discharge records. We invited the development of systems that can predict the smoking status of patients based on the narratives in these medical discharge records. We limited the scope of this task to understanding only the explicitly reported smoking information. In other words, information that implicitly reveals the smoking status was excluded from this study. Our smoking challenge continued the work on the application of classification techniques to the medical domain,19–22 and extended the MLP studies on medical discharge records.8,23–29 Information on the smoking status of patients is important for many health studies, e.g., studies on asthma; however, before this challenge, the only system for the automatic evaluation of the smoking status of patients from their records was the HITEx system.30 The data for the smoking challenge consisted exclusively of discharge summaries from Partners HealthCare. We preprocessed these records so that they were de-identified, tokenized, broken into sentences, converted into XML format, and separated into training and test sets. Institutional review boards of Partners HealthCare, Massachusetts Institute of Technology, and the State University of New York at Albany approved the challenge and the data preparation process. The data for the challenge were annotated by pulmonologists. The pulmonologists were asked to classify patient records into five possible smoking status categories. For the purposes of this challenge, we defined these categories as follows: A Past Smoker is a patient whose discharge summary asserts explicitly that the patient was a smoker one year or more ago but who has not smoked for at least one year. The assertion “past smoker” without any temporal qualifications means Past Smoker unless there is text that says that the patient stopped smoking less than one year ago. A Current Smoker is a patient whose discharge summary asserts e
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
Introduction to Modern Information Retrieval
Salton, G., & McGill, M. J. (1983) · Virtual Defense Library (Ministerio de Defensa) · Cited by 11,390
A review of assessing the accuracy of classifications of remotely sensed data
Congalton, R. G. (1991) · Remote Sensing of Environment · doi:10.1016/0034-4257(91)90048-b · Cited by 7,724
A re-examination of text categorization methods
Yang, Y., & Liu, X. (1999) · Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/312624.312647 · Cited by 2,652
Message Understanding Conference-6: a brief history
Grishman, R., & Sundheim, B. (1996) · Proceedings of the 16th conference on Computational linguistics - · doi:10.3115/992628.992709 · Cited by 1,408
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,118
Agreement, the F-Measure, and Reliability in Information Retrieval
Hripcsak, G. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1733 · Cited by 949
A General Natural-language Text Processor for Clinical Radiology
Friedman, C., Alderson, P. O., Austin, J. H. M., Cimino, J. J., & Johnson, S. B. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236146 · Cited by 778
Gospodnetić, O., Hatcher, E., & Cutting, D. (2005) · No Category · Cited by 604
Overview of BioCreAtIvE: critical assessment of information extraction for biology
Hirschman, L., Yeh, A., Blaschke, C., & Valencia, A. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s1-s1 · Cited by 530
Detecting Adverse Events Using Information Technology
Bates, D. W., Evans, R. S., Murff, H., Stetson, P. D., Pizziferri, L., & Hripcsak, G. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1074 · Cited by 467
Evaluating the State-of-the-Art in Automatic De-identification
Uzuner, O., Luo, Y., & Szolovits, P. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2444 · Cited by 452
Self-citationZeng, Q. T., Goryachev, S., Weiss, S., Sordo, M., Murphy, S. N., & Lazarus, R. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-30 · Cited by 360
Automated Detection of Adverse Events Using Natural Language Processing of Discharge Summaries
Melton, G. B., & Hripcsak, G. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1794 · Cited by 297
Measuring agreement in medical informatics reliability studies
Hripcsak, G., & Heitjan, D. F. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(02)00500-2 · Cited by 226
Do Online Information Retrieval Systems Help Experienced Clinicians Answer Clinical Questions?
Westbrook, J. I. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1717 · Cited by 181
Clinical Classification and Terminology: Some History and Current Observations
Chute, C. G. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070298 · Cited by 157
Chapman, W. W., Christensen, L. M., D, M. M. W. M., Haug, P. J., Ivanov, O., Dowling, J., & Olszewski, R. T. (2004) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2004.04.001 · Cited by 146
Mayo Clinic NLP System for Patient Smoking Status Identification
Savova, G., Ogren, P. V., Duffy, P. H., Buntrock, J. D., & Chute, C. G. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2437 · Cited by 126
Cross-Language Evaluation Forum: Objectives, Results, Achievements
Braschler, M., & Peters, C. (2003) · Information Retrieval · doi:10.1023/b:inrt.0000009438.69013.fa · Cited by 112
Chapman, W. W., Dowling, J., & D, M. M. W. M. (2005) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2005.04.012 · Cited by 98
The Evolution of evaluation: Lessons from the Message Understanding Conferences
Hirschman, L. (1998) · Computer Speech & Language · doi:10.1006/csla.1998.0102 · Cited by 85
A temporal constraint structure for extracting temporal information from clinical narrative
Zhou, L., Melton, G. B., Parsons, S., & Hripcsak, G. M. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.07.002 · Cited by 83
Identifying Smokers with a Medical Extraction System
Clark, C., Good, K., Jezierny, L., Macpherson, M., Wilson, B., & Chajewska, U. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2442 · Cited by 78
Experience with a mixed semantic/syntactic parser.
Haug, P. J., Koehler, S., Lau, L. M. S., Wang, P., Rocha, R. A., & Huff, S. M. (1995) · PubMed Central · Cited by 75
Jones, K. S. (1995) · Information Processing & Management · doi:10.1016/0306-4573(94)00048-8 · Cited by 68
Information handling in the nursing discharge note
Hellesø, R. (2005) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.2005.01235.x · Cited by 64
Second i2b2 workshop on natural language processing challenges for clinical records.
Uzuner, Ö. (2008) · PubMed · Cited by 63
The statistical significance of the MUC-5 results
Chinchor, N. A. (1993) · Message Understanding Conference (MUC) · doi:10.3115/1072017.1072027 · Cited by 57
Xu, H., Markatou, M., Dimova, R. B., Liu, H., & Friedman, C. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-334 · Cited by 54
A suite of natural language processing tools developed for the I2B2 project.
(2006) · PubMed Central · Cited by 51
Cohen, A. M. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2434 · Cited by 49
Kukafka, R., Bales, M. E., Burkhardt, A., & Friedman, C. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2107 · Cited by 46
Modeling Electronic Discharge Summaries as a Simple Temporal Constraint Satisfaction Problem
Hripcsak, G. M. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1623 · Cited by 46
Using Implicit Information to Identify Smoking Status in Smoke-blind Medical Discharge Summaries
Wicentowski, R., & Sydes, M. R. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2440 · Cited by 36
Medical i2b2 NLP Smoking Challenge: The A-Life System Architecture and Methodology
Heinze, D. T., Morsch, M. L., Potter, B. C., & Sheffer, R. E. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2438 · Cited by 34
Syntactically-informed semantic category recognition in discharge summaries.
Sibanda, T. C., He, T. Y., Peter Szolovits, B., & Uzuner, Ö. (2006) · Europe PMC (PubMed Central) · Cited by 29
Liu, H., & Friedman, C. (2004) · Studies in health technology and informatics · doi:10.3233/978-1-60750-949-3-639 · Cited by 23
Reference Standards in Evaluating System Performance
Miller, R. A. (2002) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2002.0090087 · Cited by 12
The Second i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data
Uzuner, Ö., Ware, H., Mullett, C. J., et al. (2008) · Cited by 11
Research works printed but not linked: 7
- Chang, 2002, Drug discovery today, 7, 992, 10.1016/S1359-6446(02)02457-1 · doi:10.1016/s1359-6446(02)02457-1
- 2000, Journal of the American Medical Informatics Association, 7, 378, 10.1136/jamia.2000.0070378 · doi:10.1136/jamia.2000.0070378
- 1995, Proceedings / the ... Annual Symposium on Computer Application [sic] in Medical Care. Symposium on Computer Applications in Medical Care, 19, 284
- 2006, Journal of the American Medical Informatics Association, 13, 488, 10.1197/jamia.M2082 · doi:10.1197/jamia.m2082
- 2004, Medinfo, 11, 773
- 2005, BMC bioinformatics [electronic resource], 6, S1
- 2004, Medinfo, 11, 639
Cited by
Savova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967
Natural language processing: an introduction
Nadkarni, P. M., Ohno-Machado, L., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 1,975
Mining electronic health records: towards better research applications and clinical care
Jensen, P. B., Jensen, L. J., & Brunak, S. (2012) · Nature Reviews Genetics · doi:10.1038/nrg3208 · Cited by 1,680
2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,252
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Metadata
- Authors
- Uzuner, O., Goldstein, I., Luo, Y., & Kohane, I.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-01-01
- Volume
- 15
- Issue
- 1
- Pages
- 14-24
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2109206523
- Cited by
- 359
- References
- 42
- FWCI
- 5.03
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.