CoChat
Sign Up
Cited by 117Biomedical Text Mining and OntologiesMachine Learning in HealthcareNatural Language Processing Techniques

A hybrid system for temporal information extraction from clinical text

Open access
Tang, Buzhou, Wu, Yonghui, Jiang, Min, Chen, Yukun, Denny, Joshua C, Xu, Hua·2013·Journal of the American Medical Informatics Association
DOI 10.1136/amiajnl-2013-001635PMC PMC3756274
Open this paper in CoChat — free account

Abstract

OBJECTIVE: To develop a comprehensive temporal information extraction system that can identify events, temporal expressions, and their temporal relations in clinical text. This project was part of the 2012 i2b2 clinical natural language processing (NLP) challenge on temporal information extraction. MATERIALS AND METHODS: The 2012 i2b2 NLP challenge organizers manually annotated 310 clinic notes according to a defined annotation guideline: a training set of 190 notes and a test set of 120 notes. All participating systems were developed on the training set and evaluated on the test set. Our system consists of three modules: event extraction, temporal expression extraction, and temporal relation (also called Temporal Link, or 'TLink') extraction. The TLink extraction module contains three individual classifiers for TLinks: (1) between events and section times, (2) within a sentence, and (3) across different sentences. The performance of our system was evaluated using scripts provided by the i2b2 organizers. Primary measures were micro-averaged Precision, Recall, and F-measure. RESULTS: Our system was among the top ranked. It achieved F-measures of 0.8659 for temporal expression extraction (ranked fourth), 0.6278 for end-to-end TLink track (ranked first), and 0.6932 for TLink-only track (ranked first) in the challenge. We subsequently investigated different strategies for TLink extraction, and were able to marginally improve performance with an F-measure of 0.6943 for TLink-only track.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/amiajnl-2013-001635

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Conventional NLP approaches for temporal information extraction suffer from limited scalability and generalizability and struggle to interpret implicit or vague temporal expressions.

    “Conventional natural language processing (NLP) approaches suffer from limited scalability and generalizability, and struggle to interpret implicit or vague temporal expressions.”

    From TimeX: Phenotype Onset Extraction from Clinical Narratives (2026) · cites this paper as [15]

    The Background (page unavailable) reports that a temporal-expression toolkit developed for newswire performed poorly on clinical notes, supporting a domain-generalizability concern. The Discussion (page unavailable) says the authors’ approach to implicit date/time inference was very limited. The paper does not establish limited scalability or a general inability of conventional NLP approaches to interpret vague expressions. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • A previous study reported that generating candidate pairs from all combinations of clinical concepts in document level often generates too many negative samples, causing an extremely imbalanced positive-to-negative sample ratio.

    “However, a previous study [43] has reported that this method often generates too many negative samples (ie, nonrelation), causing an extremely imbalanced positive-to-negative sample ratio.”

    From Extracting Family History of Patients From Clinical Narratives: Exploring an End-to-End Solution With Deep Learning Models (2020) · cites this paper as [43]

    The Methods (page unavailable) say that considering any two events or times as a candidate pair would generate a huge number of candidate TLinks and might be unsuitable for classifier training. They do not say that those candidates would predominantly be negative samples or establish an extremely imbalanced positive-to-negative ratio. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. [1]Pustejovsky J, Castaño J, Ingria R, et al. {TimeML:} Robust specification of event and temporal expressions in text. Fifth International Workshop on Computational Semantics {(IWCS-5)}, Tilburg, The Netherlands, 2003: 1–11 [Google Scholar]
  2. [2]Verhagen M, Gaizauskas R, Schilder F, et al. The TempEval challenge: identifying temporal relations in text. Lang Resources Eval 2009;43:161–79 [Google Scholar]
  3. [3]Verhagen M, Sauri R, Caselli T, et al. SemEval-2010 task 13: TempEval-2. Proceedings of the 5th International Workshop on Semantic Evaluation. Los Angeles, California: Association for Computational Linguistics, 2010: 57–62 [Google Scholar]
  4. [4]Sun W, Rumshisky A, Uzuner O. Evaluating Temporal Relations in Clinical Text: 2012 i2b2 Challenge Overview. J Am Med Inform Assoc 2013;20:806–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. [6]Sundheim BM. Overview of results of the MUC-6 evaluation. Proceedings of the 6th conference on Message Understanding. Columbia, Maryland: Association for Computational Linguistics, 1995: 13–31 [Google Scholar]
  6. [7]Chinchor N. Overview of MUC-7/MET-2. Proceedings of the 7th Message Understanding Conference, Fairfax, Virginia, 1998 [Google Scholar]
  7. [8]Mani I, Wilson G, Ferro L, et al. Guidelines for annotating temporal information. Proceedings of the First International Conference on Human Language Technology Research. San Diego: Association for Computational Linguistics, 2001:1–3 [Google Scholar]
  8. [9]Ferro L, Kozierok R, Gerber L, et al. Annotating temporal information: from theory to practice. Proceedings of the Second International Conference on Human Language Technology Research. San Diego, California: Morgan Kaufmann Publishers, 2002: 226–30 [Google Scholar]
  9. [10]Doddington G, Mitchell A, Przybocki M, et al. {The Automatic Content Extraction (ACE) Program—Tasks, Data, and Evaluation}. Proceedings of LREC 2004, Lisbon, Portugal, 2004: 837–40 [Google Scholar]
  10. [11]Setzer A. Temporal information in newswire articles: an annotation scheme and corpus study. University of Sheffield, UK, 2001 [Google Scholar]
  11. [12]Setzer A, Gaizauskas R. A pilot study on annotating temporal relations in text. Proceedings of the Workshop on Temporal and Spatial Information Processing—Volume 13. Association for Computational Linguistics, Toulouse, France, 2001: 1–8 [Google Scholar]
  12. [13]Pustejovsky J, Verhagen M. SemEval-2010 task 13: evaluating events, time expressions, and temporal relations (TempEval-2). Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions. Boulder, Colorado: Association for Computational Linguistics, 2009: 112–16 [Google Scholar]
  13. [14]Strötgen J, Gertz M. HeidelTime: high quality rule-based extraction and normalization of temporal expressions. Proceedings of the 5th International Workshop on Semantic Evaluation. Los Angeles, California: Association for Computational Linguistics, 2010: 321–4 [Google Scholar]
  14. [15]UzZaman N, Allen JF. TRIPS and TRIOS system for TempEval-2: extracting temporal information from text. Proceedings of the 5th International Workshop on Semantic Evaluation. Los Angeles, California: Association for Computational Linguistics, 2010: 276–83 [Google Scholar]
  15. [16]Zhou L, Friedman C, Parsons S, et al. System architecture for temporal information extraction, representation and reasoning in clinical narrative reports AMIA Annu Symp Proc 2005, Austin, TX, USA, 2005:869–873. [PMC free article] [PubMed]
  16. [17]Savova G, Bethard S, Styler W, et al. Towards temporal relation discovery from the clinical narrative. AMIA Annual Symposium Proceedings/AMIA Symposium AMIA Symposium 2009, Washington DC, USA, 2009: 568–72 [PMC free article] [PubMed] [Google Scholar]
  17. [18]Galescu L, Blaylock N. A corpus of clinical narratives annotated with temporal information. Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. Miami, Florida, USA: ACM, 2012: 715–20 [Google Scholar]
  18. [19]Zhou L, Melton GB, Parsons S, et al. A temporal constraint structure for extracting temporal information from clinical narrative. J Biomed Inform 2006;39:424–39 [DOI] [PubMed] [Google Scholar]
  19. [20]Gaizauskas R, Harkema H, Hepple M, et al. Task-oriented extraction of temporal information: the case of clinical narratives. Proceedings of the Thirteenth International Symposium on Temporal Representation and Reasoning. IEEE Computer Society, Washington DC, USA, 2006: 188–95 [Google Scholar]
  20. [21]Savova GK, Masanz JJ, Ogren PV, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc 2010;17:507–13 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. [22]Reeves RM, Ong FR, Matheny ME, et al. Detecting temporal expressions in medical narratives. Int J Med Inform 2013;82:118–27 [DOI] [PubMed] [Google Scholar]
  22. [23]Denny JC, Peterson JF, Choma NN, et al. Extracting timing and status descriptors for colonoscopy testing from electronic medical records. J Am Med Inform Assoc 2010;17:383–91 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. [24]Patnaik D, Butler P, Ramakrishnan N, et al. Experiences with mining temporal event sequences from electronic medical records: initial successes and some challenges. Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Diego, California, USA: ACM, 2011: 360–8 [Google Scholar]
  24. [25]Chapman WW, Chu D, Dowling JN. ConText: an algorithm for identifying contextual features from clinical text. Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing. Prague, Czech Republic: Association for Computational Linguistics, 2007: 81–8 [Google Scholar]
  25. [26]Friedman C, Alderson PO, Austin JH, et al. A general natural-language text processor for clinical radiology. J Am Med Inform Assoc 1994;1:161–74 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. [27]Zhou L, Hripcsak G. Temporal reasoning with medical data—a review with emphasis on medical natural language processing. J Biomed Inform 2007;40:183–202 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. [28]Jiang M, Chen Y, Liu M, et al. A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. J Am Med Inform Assoc 2011;18:601–30 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. [29]Tang B, Cao H, Wu Y, et al. Clinical entity recognition using structural support vector machines with rich features. ACM Sixth International Workshop on Data and Text Mining in Biomedical Informatics, Maui, HI, USA, 2012: 13–20 [Google Scholar]
  29. [30]Cer D, De Marneffe M-c, Jurafsky D, et al. Parsing to stanford dependencies: trade-offs between speed and accuracy. LREC, Floriana, Malta, 2010 [Google Scholar]

Cited by

Sorted by citations

Authors
Tang, B., Wu, Y., Jiang, M., Chen, Y., Denny, J. C., & Xu, H.
Affiliations
  • Harbin Institute of Technology (CN)
  • The University of Texas Health Science Center at Houston (US)
  • Vanderbilt University (US)
Corresponding author
Xu, Hua
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2013-09
Volume
20
Issue
5
Pages
828-835
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1136/amiajnl-2013-001635
PMC
PMC3756274
OpenAlex
W2162364899
Cited by
117
References
1
Citations by year
2026: 2 · 2025: 3 · 2024: 2 · 2023: 10 · 2022: 4 · 2021: 13 · 2020: 9 · 2019: 12 · 2018: 8 · 2017: 14 · 2016: 10 · 2015: 13 · 2014: 7 · 2013: 10
FWCI
5.98
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • National Institutes of Health (5r01lm007995-06)
  • National Cancer Institute (R01CA141307)
  • U.S. National Library of Medicine (U54LM008748, R13LM011411, R01-LM-007995)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
temporal information extraction, clinical text, temporal relation extraction, temporal expressions, event extraction, clinical notes, hybrid systems, clinical natural language processing
MeSH terms
Artificial Intelligence, Humans, Natural Language Processing, Time, Translational Research, Biomedical, Electronic Health Records, Patient Discharge Summaries

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review