Evaluating the state of the art in coreference resolution for electronic medical records
Open accessAbstract
BACKGROUND: The fifth i2b2/VA Workshop on Natural Language Processing Challenges for Clinical Records conducted a systematic review on resolution of noun phrase coreference in medical records. Informatics for Integrating Biology and the Bedside (i2b2) and the Veterans Affair (VA) Consortium for Healthcare Informatics Research (CHIR) partnered to organize the coreference challenge. They provided the research community with two corpora of medical records for the development and evaluation of the coreference resolution systems. These corpora contained various record types (ie, discharge summaries, pathology reports) from multiple institutions. METHODS: The coreference challenge provided the community with two annotated ground truth corpora and evaluated systems on coreference resolution in two ways: first, it evaluated systems for their ability to identify mentions of concepts and to link together those mentions. Second, it evaluated the ability of the systems to link together ground truth mentions that refer to the same entity. Twenty teams representing 29 organizations and nine countries participated in the coreference challenge. RESULTS: The teams' system submissions showed that machine-learning and rule-based approaches worked best when augmented with external knowledge sources and coreference clues extracted from document structure. The systems performed better in coreference resolution when provided with ground truth mentions. Overall, the systems struggled in solving coreference resolution for cases that required domain knowledge.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A successful system also needs to consider co-reference resolution, which could be considered a standalone task for NLP systems.
“A successful system also needs to consider co-reference resolution, which could be considered a standalone task for NLP systems [24].”
From Family History Extraction From Synthetic Clinical Narratives Using Natural Language Processing: Overview and Evaluation of a Challenge Data Set and Solutions for the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing (OHNLP) Competition (2021) · cites this paper as [24]
The Introduction describes coreference resolution in electronic medical records as a distinct NLP challenge, and the Results evaluate systems on that task. This supports treating it as a standalone task, but the paper does not say that NLP systems generally need coreference resolution to succeed. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 37
- McCarthy JF, Lehnert WG. Using decision trees for coreference resolution. 40th International Joint Conference on Artificial Intelligence. Montreal, Canada: IJCAII, 1995:1050e5.
- Haghighi A, Klein D. Simple coreference resolution with rich syntactic and semantic features. Proceedings of the Conference on Empirical Methods in Natural Language Processing. East Stroudsburg, PA, USA: Association for Computational Linguistics, 2009;3:1152e61.
- Gasperin C, Briscoe T. Statistical anaphora resolution in biomedical texts. Proceedings of the 22nd International Conference on Computational Linguistics. East Stroudsburg, PA, USA: Association for Computational Linguistics, 2008;1:257e64.
- Iglesias JE, Rocks K, Jahanshad N, et al. Tracking medication information across medical records. AMIA Annu Symp Proc 2009;2009:266e70.
- Coden A, Savova G, Sominsky I, et al. Automatically extracting cancer disease characteristics from pathology reports into a Disease Knowledge Representation Model. J Biomed Inform 2009;42:937e49.
- Zheng J, Chapman WW, Crowley RS, et al. Coreference resolution: a review of general methodologies and applications in the clinical domain. J Biomed Inform 2011;44:1113e22.
- He T. Coreference Resolution on Entities and Events for Hospital Discharge Summaries. Cambridge: MIT, 2007.
- Uzuner O, Luo Y, Szolovits P. Evaluating the state-of-the-art in automatic de- identification. J Am Med Inform Assoc 2007;14:550e63.
- Uzuner O, Goldstein I, Luo Y. Identifying patient smoking status from medical discharge summaries. J Am Med Inform Assoc 2008;15:14e24.
- Uzuner O. Recognizing obesity and co-morbidities in sparse data. J Am Med Inform Assoc 2009;16:561e70.
- Uzuner O, Solti I, Cadag E. Extracting medication information from clinical text. J Am Med Inform Assoc 2010;17:514e18.
- Uzuner O, South B, Shen S, et al. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical texts. J Am Med Inform Assoc 2011;18:552e6.
- Pradhan S, Ramshaw L, Marcus M, et al. CoNLL-2011 shared task: modeling unrestricted coreference in OntoNotes. Proceedings of the Fifteenth Conference on Computational natural language learning. East Stroudsburg, PA, USA: Association of Computational Linguistics, 2011:1e27.
- Savova G, Chapman WW, Zheng J, et al. Anaphoric relations in the clinical narrative: corpus creation. J Am Med Inform Assoc 2011;18:459e65.
- Ogren P, Savova G, Chut C. Constructing evaluation corpora for Automated clinical Named entity Recognition. Proceedings of the 6th International language resources and evaluation. Paris, France: European Language Resources Association (ELRA), 2008:3134e50.
- Vilain M, Burger J, Aberdeen J, et al. A model-theoretic coreference scoring scheme. Proceedings of the 6th Conference on Message Understanding (MUC). East Stroudsburg, PA, USA: Association for Computational Linguistics, 1995: 45e52.
- Bagga A, Baldwin B. Algorithms for scoring coreference chains. The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference. Granada, Spain, 1998:563e6.
- Luo X. On coreference resolution performance metrics. Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. East Stroudsburg, PA, USA: Association for Computational Linguistics, 2005:28e36.
- Noreen E. Computer Intensive Methods for Testing Hypotheses: an Introduction. New-York: Wiley-Interscience, 1989.
- Gooch P. Coreference resolution in clinical discharge summaries, progress notes, surgical and pathology reports: a unified lexical approach. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Grouin C, Dinarelli M, Rosset S, et al. Coreference resolution in clinical reports-the LIMSI participation in the i2b2/VA 2011 challenge. Proceedings of the 2011 i2b2/VA/ Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Hinote D, Ramirez C, Chen P. A comparative study of co-refernece resolution in clinical text. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Fellbaum C. WordNet.An Electronic Lexical Database. Cambridge, MA: MIT Press, 1998:422.
- Yang H, Willis A, Ad Roeck, et al. A system for coreference resolution in clinical documents. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Jonnalagadda S, Li D, Sohn S, et al. Coreference analysis in clinical notes: a multi pass sieve with alternate anaphora resolution modules. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Rink B, Harabagiu S. A supervised multi-pass sieve approach for resolving coreference in clinical records. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Jindal P, Roth D. Using domain knowledge and domain-inspired discourse model for coreference resolution for clinical narratives. Proceedings of the 2011 i2b2/VA/ Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Anick P, Hong P, Xue N, et al. Coreference resolution for electronic medical records. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Cai J, Mujdricza E, Hou Y, et al. Weakly supervised graph-based coreference resolution for clinical texts. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Xu Y, Liu J, Wu J, et al. EHUATUO: a mention-pair coreference system by exploiting document intrinsic latent structures and world knowledge in discharge summaries: 2011 i2b2 challenge. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Song Y, Wang H, Wang Z, et al. Short text conceptualization using a probabilistic knowledgebase. Proceedings of 22nd International Joint Conference on Artificial Intelligence. Menlo Park, CA, USA: IJCAI/AAAI, 2011:2320e36.
- Lee T, Wang Z, Wang H, et al. Web scale taxonomy cleansing. Proceedings of the VLDB Endowment. Seattle, WA, USA: VLDB Endowment, 2011:1295e306.
- Lan M, Zhao J, Zhang K, et al. Comparative investigation on learning-based and rule-based approaches to coreference resolution in clinic domain: a case study in i2b2 challenge 2011 Task 1. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Glinos D. A search based method for clinical text coreference resolution. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Benajiba Y, Shaw J. A SVM-based coreference resolution syste based on Philips information extraction. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Dai H-J, Wu C-Y, Chen C-Y, et al. Co-reference resolution of the medical concepts in the patient discharge summaries. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
- Ware H, Mullet C, Jagannathan V, et al. Coreference resolution of concepts in clinical documents. Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2, 2011.
Cited by
Sorted by citations
Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235 · see citing works
Kreimeyer, K., Foster, M., Pandey, A., Arya, N., Halford, G., Jones, S. F., Forshee, R., Walderhaug, M., & Botsis, T. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.012 · Cited by 616 · see citing works
Clinical text data in machine learning: systematic review
Spasic, I., & Nenadic, G. (2020) · JMIR Medical Informatics · doi:10.2196/17984 · Cited by 358 · see citing works
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · see citing works
Natural Language Processing in Oncology: A Review
Yim, W.-W., Yetisgen, M., Harris, W. P., & Kwan, S. W. (2016) · JAMA Oncology · doi:10.1001/jamaoncol.2016.0213 · Cited by 298 · see citing works
Electronic health records-driven phenotyping: challenges, recent advances, and perspectives
Pathak, J., Kho, A. N., & Denny, J. C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002428 · Cited by 274 · see citing works
A scoping review of publicly available language tasks in clinical natural language processing
Gao, Y., Dligach, D., Christensen, L., Tesch, S., Laffin, R., Xu, D., Miller, T., Uzuner, O., Churpek, M. M., & Afshar, M. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac127 · Cited by 28 · see citing works
Self-citationShen, F., Liu, S., Fu, S., Wang, Y., Henry, S., Uzuner, O., & Liu, H. (2021) · JMIR Medical Informatics · doi:10.2196/24008 · Cited by 21 · see citing works
- Authors
- Uzuner, O., Bodnari, A., Shen, S., Forbush, T., Pestian, J., & South, B. R.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- University of Utah (US)
- Cincinnati Children's Hospital Medical Center (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-09
- Volume
- 19
- Issue
- 5
- Pages
- 786-791
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC3422835
- OpenAlex
- W1672757658
- Cited by
- 164
- References
- 0
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 5 · 2023: 10 · 2022: 9 · 2021: 13 · 2020: 11 · 2019: 9 · 2018: 12 · 2017: 9 · 2016: 18 · 2015: 14 · 2014: 14 · 2013: 21 · 2012: 12
- FWCI
- 1.78
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (1r13lm010743-01)
- U.S. National Library of Medicine (U54LM008748, R13LM011411, R13LM010743-01, R13 LM010743)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- coreference resolution, electronic health records, clinical natural language processing, discharge summaries, pathology reports, annotated corpus, entity linking, rule-based methods, machine learning, external knowledge sources, domain knowledge
- MeSH terms
- Humans, Natural Language Processing, Information Storage and Retrieval, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
