Comprehensive temporal information detection from clinical text: medical events, time, and TLINK identification
Open accessAbstract
BACKGROUND: Temporal information detection systems have been developed by the Mayo Clinic for the 2012 i2b2 Natural Language Processing Challenge. OBJECTIVE: To construct automated systems for EVENT/TIMEX3 extraction and temporal link (TLINK) identification from clinical text. MATERIALS AND METHODS: The i2b2 organizers provided 190 annotated discharge summaries as the training set and 120 discharge summaries as the test set. Our Event system used a conditional random field classifier with a variety of features including lexical information, natural language elements, and medical ontology. The TIMEX3 system employed a rule-based method using regular expression pattern match and systematic reasoning to determine normalized values. The TLINK system employed both rule-based reasoning and machine learning. All three systems were built in an Apache Unstructured Information Management Architecture framework. RESULTS: Our TIMEX3 system performed the best (F-measure of 0.900, value accuracy 0.731) among the challenge teams. The Event system produced an F-measure of 0.870, and the TLINK system an F-measure of 0.537. CONCLUSIONS: Our TIMEX3 system demonstrated good capability of regular expression rules to extract and normalize time information. Event and TLINK machine learning systems required well-defined feature sets to perform well. We could also leverage expert knowledge as part of the machine learning features to further improve TLINK identification performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Conventional NLP approaches for temporal information extraction suffer from limited scalability and generalizability and struggle to interpret implicit or vague temporal expressions.
“Conventional natural language processing (NLP) approaches suffer from limited scalability and generalizability, and struggle to interpret implicit or vague temporal expressions.”
From TimeX: Phenotype Onset Extraction from Clinical Narratives (2026) · cites this paper as [19]
The Discussion (page unavailable) says relative time expressions requiring inference or reasoning were often difficult to catch, supporting the claim about implicit temporal expressions. It discusses difficulties with the authors’ system, however, and does not establish that conventional NLP approaches generally have limited scalability or generalizability. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 21
- Pustejovsky J, Castano J, Ingria R, et al. TimeML: Robust specification of event and temporal expressions in text. New Directions in Question Answering. In AAAI Spring Symposium on New Directions in Question-Answering, Stanford, CA, 2003:28-34.
- Uzuner O, South BR, Shen S, et al. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc 2011;18:552-6.
- Torii M, Wagholikar K, Liu H. Using machine learning for concept extraction on clinical documents from multiple data sources. J Am Med Inform Assoc 2011;18:580-7.
- Mani I, Wilson G. Annual Meeting on Association for Computational Linguistics, New Brunswick, NJ, 2000:69-76.
- Verhagen M, Pustejovsky J. Temporal processing with the TARSQI toolkit. International Conference on Computational Linguistics, Manchester, UK. 2008:189-92.
- Strötgen J, Gertz M. HeidelTime: High quality rule-based extraction and normalization of temporal expressions. International Workshop on Semantic Evaluation. Uppsala Sweden, 2010:321-4.
- Verhagen M, Sauri R, Caselli T, et al. SemEval-2010 task 13: TempEval-2. International Workshop on Semantic Evaluation. Uppsala Sweden. 2010:57-62.
- Chang AX, Manning CD. SUTIME: a library for recognizing and normalizing time expressions. International Conference on Language Resources and Evaluation (LREC 2012), Istanbul, Turkey, 2012:3735-40.
- Savova G, Masanz J, Ogren P, et al. Mayo Clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc 2010;17:507-13.
- Sohn S, Savova GK. Mayo Clinic Smoking Status Classification System: Extensions and Improvements. AMIA Annual Symposium; 2009, San Francisco, CA. 2009:619-23.
- Passonneau RJ. A computational model of the semantics of tense and aspect. Comput Linguist 1988;14:44-60.
- Webber BL. Tense as discourse anaphor. Comput Linguist 1988;14:61-73.
- Hwang CH, Schubert LK. Tense Trees as the fine structure of discourse. Annual Meeting on Association for Computational Linguistics, Newark, DE, 1992:232-40.
- Lascarides A, Asher N. Temporal interpretation, discourse relations and commonsense entailment. Linguist Philos 1993;16:437-93.
- Mani I, Pustejovsky J. Temporal discourse models for narrative structure. ACL Workshop on Discourse Annotation, Barcelona, Spain, 2004:57-64.
- Mani I, Verhagen M, Wellner B, et al. Machine learning of temporal relations. Annual Meeting on Association for Computational Linguistics. Sydney, Australia, 2006:753-60.
- Wagholikar K, Torii M, Jonnalagadda S, et al. Feasibility of pooling annotated corpora for clinical concept extraction. AMIA Summits Transl Sci Proc 2012;2012:38.
- Savova GK, Masanz JJ, Ogren PV, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc 2010;17:507-13.
- Liu HWS, Li D, Jonnalagadda S, et al. Towards a semantic lexicon for clinical natural language processing Annual Symposium of American Medical Informatics Association; Chicago: 2012.
- Chapman WW, Bridewell W, Hanbury P, et al. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 2001;34:301-10.
- Jonnalagadda SR, Li D, Sohn S, et al. Coreference analysis in clinical notes: a multi-pass sieve with alternate anaphora resolution modules. J Am Med Inform Assoc 2012;19:867-74.
Cited by
Sorted by citations
Electronic health records-driven phenotyping: challenges, recent advances, and perspectives
Pathak, J., Kho, A. N., & Denny, J. C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002428 · Cited by 274 · see citing works
TimeX: Phenotype Onset Extraction from Clinical Narratives
Chen, F., Jiang, S., Nguyen, Q. M., Ta, C. N., Wang, K., Chung, W. K., & Weng, C. (2026) · npj Health Systems · doi:10.1038/s44401-026-00099-8
- Authors
- Sohn, S., Wagholikar, K. B., Li, D., Jonnalagadda, S. R., Tao, C., Komandur Elayavilli, R., & Liu, H.
- Affiliations
- Mayo Clinic (US)
- Corresponding author
- Sohn, Sunghwan
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-09
- Volume
- 20
- Issue
- 5
- Pages
- 836-842
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC3756269
- OpenAlex
- W2163241422
- Cited by
- 62
- References
- 0
- Citations by year
- 2026: 1 · 2025: 3 · 2023: 3 · 2022: 3 · 2021: 7 · 2020: 5 · 2019: 9 · 2018: 2 · 2017: 6 · 2016: 6 · 2015: 6 · 2014: 6 · 2013: 4 · 2012: 1
- FWCI
- 3.29
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0845523)
- National Institutes of Health
- Office of the National Coordinator for Health Information Technology
- National Heart, Lung, and Blood Institute
- National Institute of General Medical Sciences (R01GM102282)
- U.S. National Library of Medicine (U54LM008748, K99LM011389, R13LM011411, R01LM009959A1, R01LM009959)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- temporal information extraction, clinical text, discharge summaries, conditional random fields, regular expression matching, time normalization
- MeSH terms
- Artificial Intelligence, Humans, Natural Language Processing, Time, Information Storage and Retrieval, Electronic Health Records, Patient Discharge Summaries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
