Assertion Detection in Clinical Notes: Medical Language Models to the Rescue?
Open accessAbstract
In order to provide high-quality care, health professionals must efficiently identify the presence, possibility, or absence of symptoms, treatments and other relevant entities in freetext clinical notes.Such is the task of assertion detection -to identify the assertion class (present, possible, absent) of an entity based on textual cues in unstructured text.We evaluate state-of-the-art medical language models on the task and show that they outperform the baselines in all three classes.As transferability is especially important in the medical domain we further study how the best performing model behaves on unseen data from two other medical datasets.For this purpose we introduce a newly annotated set of 5,000 assertions for the publicly available MIMIC-III dataset.We conclude with an error analysis that reveals situations in which the models still go wrong and points towards future research directions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The trained BERT-based model from Aken et al. effectively distinguishes affirmed from negated clinical concepts.
“Phenotype negation detection was performed using the trained BERT-based model from Aken et al.47, which effectively distinguishes affirmed from negated clinical concepts.”
From TimeX: Phenotype Onset Extraction from Clinical Narratives (2026) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · see citing works
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W. B., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,121 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · see citing works
Peng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899 · see citing works
The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes
Vincze, V., Szarvas, G., Farkas, R., Móra, G., & Csirik, J. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-s11-s9 · Cited by 422 · see citing works
Harkema, H., Dowling, J., Thornblade, T., & Chapman, W. W. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.05.002 · Cited by 406 · see citing works
de Bruijn, B., Cherry, C. A., Kiritchenko, S., Martin, J., & Zhu, X. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000150 · Cited by 235 · see citing works
Lewis, P., Ott, M., Du, J., & Stoyanov, V. (2020) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2020.clinicalnlp-1.17 · Cited by 175 · see citing works
Wu, S., Miller, T. A., Masanz, J. J., Coarr, M., Halgrim, S. R., Carrell, D., & Clark, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112774 · Cited by 119 · see citing works
Speculation and Negation Scope Detection via Convolutional Neural Networks
Zhong, Q., Li, P., Zhu, Q., Zhou, G., Luo, Z., & Luo, W. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1078 · Cited by 85 · see citing works
NegBERT: A Transfer Learning Approach for Negation Detection and Scope ResolutionPreprint
Khandelwal, A., & Sawant, S. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04211 · Cited by 73 · see citing works
Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration
van Aken, B., Papaioannou, J.-M., Mayrdorfer, M., Budde, K., Gers, F. A., & Loeser, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.75 · Cited by 69 · see citing works
Self-citationJoint Entity Extraction and Assertion Detection for Clinical Text
Bhatia, P., Celikkaya, B., & Khalilia, M. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1091 · Cited by 40 · see citing works
Challenges in understanding clinical notes
Perera, S., Sheth, A., Thirunarayan, K., Nair, S., & Shah, N. (2013) · doi:10.1145/2512410.2512427 · Cited by 28 · see citing works
Attention-Based Deep Learning System for Negation and Assertion Detection in Clinical Notes
Chen, L. (2019) · International Journal of Artificial Intelligence & Applications · doi:10.5121/ijaia.2019.10101 · Cited by 25 · see citing works
Sergeeva, E., Zhu, H., Prinsen, P., & Tahmasebi, A. M. (2019) · PubMed Central · Cited by 16 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 2
- Berry de Bruijn, Colin Cherry, Svetlana Kiritchenko, Joel D. Martin, and Xiaodan Zhu. 2011b. Machine- learned solutions for three stages of clinical informa- tion extraction: the state of the art at i2b2 2010. J. Am. Medical Informatics Assoc., 18(5):557-562.
- Zhong Qian, Peifeng Li, Qiaoming Zhu, Guodong Zhou, Zhunchen Luo, and Wei Luo. 2016. Specu- lation and Negation Scope Detection via Convolu- tional Neural Networks. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 815-825.
Cited by
Sorted by citations
Lightweight transformers for clinical natural language processing
Rohanian, O., Nouriborji, M., Jauncey, H., Kouchaki, S., Nooralahzadeh, F., Clifton, L., Merson, L., & Clifton, D. A. (2024) · Natural Language Engineering · doi:10.1017/s1351324923000542 · Cited by 41 · see citing works
Effective matching of patients to clinical trials using entity extraction and neural re-ranking
Kusa, W., Mendoza, Ó. E., Knoth, P., Pasi, G., & Hanbury, A. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104444 · Cited by 34 · see citing works
Falter, M., Godderis, D., Scherrenberg, M., Kizilkilic, S. E., Xu, L., Mertens, M., Jansen, J., Legroux, P., Kindermans, H. P. J., Sinnaeve, P., et al. (2024) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae008 · Cited by 15 · see citing works
Assertion detection in clinical natural language processing using large language models
Ji, Y., Yu, Z., & Wang, Y. (2024) · 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00039 · Cited by 14 · see citing works
Phenotype driven molecular genetic test recommendation for diagnosing pediatric rare disorders
Chen, F., Ahimaz, P. R., Nguyen, Q. M., Lewis, R., Chung, W. K., Ta, C., Szigety, K. M., Sheppard, S. E., Campbell, I. M., Wang, K., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01331-1 · Cited by 14 · see citing works
Trustworthy assertion classification through prompting
Wang, S., Tang, L., Majety, A., Rousseau, J. F., Shih, G., Ding, Y., & Peng, Y. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104139 · Cited by 14 · see citing works
Davies, H. A., Nenadić, G., Alfattni, G., Casteleiro, M. A., Moubayed, N. A., Farrell, S., Radford, A. D., & Noble, P. M. (2024) · Frontiers in Veterinary Science · doi:10.3389/fvets.2024.1352239 · Cited by 11 · see citing works
LongHealth: A Question Answering Benchmark with Long Clinical Documents
Adams, L., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J., Kather, J. N., Truhn, D., & Bressem, K. (2024) · arXiv · doi:10.48550/arxiv.2401.14490 · Cited by 10 · see citing works
LongHealth: A Question Answering Benchmark with Long Clinical Documents
Adams, L. C., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J. W. L., Kather, J. N., Truhn, D., & Bressem, K. K. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00204-w · Cited by 15 · see citing works
Liu, X., Yeung, W. L. K., Chen, Z., Hao, S., Yang, Z., Sun, X., Liu, C., Mao, Z., Yan, M., Yan, W., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01875-w · Cited by 6 · see citing works
Investigating the Impact of Query Representation on Medical Information Retrieval
Peikos, G., Alexander, D., Pasi, G., & de Vries, A. P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_42 · Cited by 4 · see citing works
Automating clinical phenotyping using natural language processing
Schmidt, L., Ibing, S., Borchert, F., Hugo, J., Marshall, A. A., Peraza, J., Cho, J. H., Böttinger, E. P., Renard, B. Y., & Ungaro, R. C. (2026) · Communications Medicine · doi:10.1038/s43856-025-01337-0 · Cited by 4 · see citing works
Wen, A., Wang, L., He, H., Fu, S., Liu, S., Hanauer, D. A., Harris, D. R., Kavuluru, R., Zhang, R., Natarajan, K., et al. (2024) · JMIR Medical Informatics · doi:10.2196/49997 · Cited by 3 · see citing works
The Impact of a Hybrid Hospital at Home Program in Reducing Subacute Rehabilitation
Yadav, R., Mahyoub, M. A., Capriotti, M. W., Berio-Dorta, R. L., Dougherty, K., & Shukla, A. (2023) · Risk Management and Healthcare Policy · doi:10.2147/rmhp.s419862 · Cited by 3 · see citing works
Semantic Reasoning with NLI for Assertion Detection in Medical Text
Du, Z., Liu, X., Xu, J., Nan, G., & Wang, G. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995166 · Cited by 2 · see citing works
Zero-Shot Approaches for the Extraction of Event Logs from Medical Notes
Susaiyah, A., & Sidorova, N. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-92474-3_26 · Cited by 2 · see citing works
Montoya-Lora, D., Giraldo-Forero, A. F., Torres-Silva, E. A., Rúa, S., Valencia, S. A., Barrientos-Gómez, J. G., Florez-Arango, J. F., & Orozco‐Duque, A. (2025) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-89514-2_5 · Cited by 2 · see citing works
Bai, Z., Xu, Z., Sun, C. H., Zang, C., Bunnell, H. T., Sinfield, C., Rutter, J., Martinez, A. T., Bailey, L. C., Weiner, M., et al. (2025) · npj Health Systems · doi:10.1038/s44401-025-00033-4 · Cited by 1 · see citing works
Vanderkerken, M. A., Eygen, K. V., Galle, V., Verbiest, A., Janssens, A. M., Masuy, I., Theys, K., Cuppens, T., Muylle, K. M., & Becker, A. D. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00159 · Cited by 1 · see citing works
Zekaoui, N. E., Rhanoui, M., Yousfi, S., & Mikram, M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110651 · Cited by 1 · see citing works
TimeX: Phenotype Onset Extraction from Clinical Narratives
Chen, F., Jiang, S., Nguyen, Q. M., Ta, C. N., Wang, K., Chung, W. K., & Weng, C. (2026) · npj Health Systems · doi:10.1038/s44401-026-00099-8
The impact of a hybrid hospital at home program in reducing subacute rehabilitation referralsPreprint
Yadav, R., Mahyoub, M. A., Capriotti, M. W., Berio-Dorta, R. L., Dougherty, K., & Shukla, A. (2023) · medRxiv · doi:10.1101/2023.05.26.23290592
Framework for the Design and Evaluation of Patient-Friendly AI-Generated Medical Reports
Mruthyunjaya, S., & Dutta, A. K. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476042
Lakshmi.Challa, V. M., & Ramakrishnudu, T. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.110962
Zoha, A. E., Gondy Leroy, A., Galgiani, J. N., & Donovan, F. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00012
Woldaregay, A. Z., Lund, J. A., Ngo, P. D., Tayefi, M., Burman, J., Hansen, S., Sillesen, M., Dalianis, H., Jenssen, R., Lindsetmo, R. O., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/91880
- Authors
- van Aken, B., Trajanovska, I., Siu, A., Mayrdorfer, M., Budde, K., & Loeser, A.
- Affiliations
- Berliner Hochschule für Technik (DE)
- Charité – Universitätsmedizin Berlin (DE)
- Venue
- Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations
- Published
- 2021
- Pages
- 35-40
- OpenAlex
- W3171324702
- Cited by
- 25
- References
- 22
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 6 · 2023: 4 · 2022: 2
- FWCI
- 10.86
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Funders
- Bundesministerium für Wirtschaft und Energie
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- clinical notes, clinical language models, transferability, MIMIC-III, error analysis, clinical text
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
