Limitations of transformers on clinical text classification
Open accessAbstract
Bidirectional Encoder Representations from Transformers (BERT) and BERT-based approaches are the current state-of-the-art in many natural language processing (NLP) tasks; however, their application to document classification on long clinical texts is limited. In this work, we introduce four methods to scale BERT, which by default can only handle input sequences up to approximately 400 words long, to perform document classification on clinical texts several thousand words long. We compare these methods against two much simpler architectures - a word-level convolutional neural network and a hierarchical self-attention network - and show that BERT often cannot beat these simpler baselines when classifying MIMIC-III discharge summaries and SEER cancer pathology reports. In our analysis, we show that two key components of BERT - pretraining and WordPiece tokenization - may actually be inhibiting BERT's performance on clinical text classification tasks where the input document is several thousand words long and where correctly identifying labels may depend more on identifying a few key words or phrases rather than understanding the contextual meaning of sequences of text.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BlueBERT did not outperform baseline non-LLM models in pan-cancer contexts.
“Gao et al [44] reported that BlueBERT did not outperform baseline nonLLM models in pan-cancer contexts”
From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · see citing works
Recent Trends in Deep Learning Based Natural Language ProcessingPreprint
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · IEEE Computational Intelligence Magazine · doi:10.48550/arxiv.1708.02709 · Cited by 3,133 · see citing works
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921 · see citing works
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
Text Summarization with Pretrained Encoders
Liu, Y., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1387 · Cited by 1,779 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · see citing works
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534 · see citing works
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · see citing works
Peng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899 · see citing works
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Xiao, C., Choi, E., & Sun, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy068 · Cited by 792 · see citing works
Matching the Blanks: Distributional Similarity for Relation Learning
Baldini Soares, L., FitzGerald, N., Ling, J., & Kwiatkowski, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1279 · Cited by 731 · see citing works
Explainable Prediction of Medical Codes from Clinical Text
Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., & Eisenstein, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1100 · Cited by 597 · see citing works
Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review
Sheikhalishahi, S., Miotto, R., Dudley, J. T., Lavelli, A., Rinaldi, F., & Osmani, V. (2019) · JMIR Medical Informatics · doi:10.2196/12239 · Cited by 558 · see citing works
Deep learning in clinical natural language processing: a methodical review
Wu, S., Roberts, K. E., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz200 · Cited by 495 · see citing works
BioWordVec, improving biomedical word embeddings with subword information and MeSH
Zhang, Y., Chen, Q., Yang, Z., Lin, H., & Lu, Z. (2019) · Scientific Data · doi:10.1038/s41597-019-0055-0 · Cited by 455 · see citing works
DocBERT: BERT for Document ClassificationPreprint
Adhikari, A., Ram, A., Tang, R., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08398 · Cited by 286 · see citing works
Rationale-Augmented Convolutional Neural Networks for Text Classification
Zhang, Y., Marshall, I., & Wallace, B. C. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1076 · Cited by 249 · see citing works
Ayala Solares, J. R., Diletta Raimondi, F. E., Zhu, Y., Rahimian, F., Canoy, D., Tran, J., Pinho Gomes, A. C., Payberah, A. H., Zottoli, M., Nazarzadeh, M., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103337 · Cited by 247 · see citing works
Pretraining-Based Natural Language Generation for Text Summarization
Zhang, H., Cai, J., Xu, J., & Wang, J. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1074 · Cited by 247 · see citing works
Gehrmann, S., Dernoncourt, F., Li, Y., Carlson, E. T., Wu, J. T.-Y., Welt, J. A., Foote, J., Moseley, E. T., Grant, D. W., Tyler, P. D., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192360 · Cited by 243 · see citing works
Medical Text Classification using Convolutional Neural Networks
Hughes, M., Li, I., Kotoulas, S., & Suzumura, T. (2017) · Studies in Health Technology and Informatics · doi:10.48550/arxiv.1704.06841 · Cited by 220 · see citing works
A Survey on Contextual EmbeddingsPreprint
Liu, Q., Kusner, M. J., & Blunsom, P. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.12 · Cited by 178 · see citing works
Classification and Clustering of Arguments with Contextualized Word Embeddings
Reimers, N., Schiller, B., Beck, T., Daxenberger, J., Stab, C., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1054 · Cited by 157 · see citing works
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Talmor, A., & Berant, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1485 · Cited by 157 · see citing works
Deep Learning for Automated Extraction of Primary Sites From Cancer Pathology Reports
Qiu, J. X., Yoon, H., Fearn, P., & Tourassi, G. D. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2700722 · Cited by 146 · see citing works
Self-citationHierarchical attention networks for information extraction from cancer pathology reports
Gao, S. B., Young, M. T., Qiu, J. X., Yoon, H., Christian, J. B., Fearn, P., Tourassi, G. D., & Ramanthan, A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx131 · Cited by 141 · see citing works
Self-citationA BERT Baseline for the Natural QuestionsPreprint
Alberti, C., Lee, K., & Collins, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08634 · Cited by 136 · see citing works
Using machine learning to parse breast pathology reports
Yala, A., Barzilay, R., Salama, L., Griffin, M., Sollender, G. E., Bardia, A., Lehman, C. D., Buckley, J. M., Coopey, S. B., Polubriaginof, F. C. G., et al. (2016) · Breast Cancer Research and Treatment · doi:10.1007/s10549-016-4035-1 · Cited by 114 · see citing works
Alawad, M., Gao, S. B., Qiu, J. X., Yoon, H. J., Christian, J. B., Penberthy, L. T., Mumphrey, B., Wu, X., Coyle, L. M., & Tourassi, G. D. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz153 · Cited by 95 · see citing works
Self-citationClassifying cancer pathology reports with hierarchical self-attention networks
Gao, S., Qiu, J. X., Alawad, M. M., Hinkle, J., Schaefferkoetter, N., Yoon, H.-J., Christian, J., Fearn, P., Penberthy, L., Wu, X.-C., et al. (2019) · Artif. Intell. Medicine · doi:10.1016/j.artmed.2019.101726 · Cited by 62 · see citing works
Self-citationTumor information extraction in radiology reports for hepatocellular carcinoma patients.
Yim, W.-W., Denman, T., Kwan, S. W., & Yetişgen, M. (2016) · PubMed Central · Cited by 31 · see citing works
Automated extraction of Biomarker information from pathology reports
Lee, J., Song, H.-J., Yoon, E., et al. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0609-7 · Cited by 28 · see citing works
Xie, F., Lee, J. B., Munoz‐Plaza, C. E., Hahn, E. E., & Chen, W. (2017) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_55_17 · Cited by 27 · see citing works
Classification of Cancer Pathology Reports: A Large-Scale Comparative Study
Martina, S., Ventura, L., & Frasconi, P. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3005016 · Cited by 18 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 2
- [32]Baumel T, Nassour-Kassis J, Cohen R, Elhadad M, and Elhadad N, “ Multi-label classification of patient notes a case study on icd code assignment,” in AAAI Workshops, 2017, pp. 409– 416.
- [43]Xiong W, Du J, Wang WY, and Stoyanov V, “ Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model,” in ICLR 2020: Eighth International Conference on Learning Representations, 2020.
Cited by
Sorted by citations
Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review
Chen, D., Alnassar, S. A., Avison, K. E., Huang, R. S., & Raman, S. (2025) · JMIR Cancer · doi:10.2196/65984 · Cited by 66 · see citing works
“Another minor disadvantage is the token limit of many LLMs, including both BERT and ChatGPT [20,42,44].”
Afshar, M., Sharma, B., Dligach, D., Oguss, M., Brown, R., Chhabra, N., Thompson, H. M., Markossian, T., Joyce, C., Churpek, M. M., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00041-3 · Cited by 55 · see citing works
- Authors
- Gao, S., Alawad, M., Young, M. T., Gounley, J., Schaefferkoetter, N., Yoon, H. J., Wu, X.-C., Durbin, E. B., Doherty, J., Stroup, A., Coyle, L., & Tourassi, G.
- Affiliations
- Oak Ridge National Laboratory (US)
- Louisiana State University Health Sciences Center New Orleans (US)
- University of Kentucky (US)
- University of Utah (US)
- Information Management Services (US)
- Corresponding author
- Gao, Shang
- Tourassi, Georgia
- ORCID
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2021-09
- Volume
- 25
- Issue
- 9
- Pages
- 3596-3607
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- PMC
- PMC8387496
- OpenAlex
- W3132259035
- Cited by
- 157
- References
- 46
- Citations by year
- 2026: 17 · 2025: 35 · 2024: 41 · 2023: 32 · 2022: 25 · 2021: 7
- FWCI
- 9.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Journal of Biomedical and Health Informatics (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- UKnowledge (University of Kentucky) (submittedVersion, cc-by)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Funders
- Joint Design of Advanced Computing Solutions for Cancer
- U.S. Department of Energy (DE-AC05-00OR22725)
- National Cancer Institute
- National Institutes of Health
- U.S. Department of Energy by Argonne National Laboratory (DE-AC02-06-CH11357)
- Lawrence Livermore National Laboratory (DEAC52-07NA27344)
- Los Alamos National Laboratory (DE-AC5206NA25396)
- Oak Ridge National Laboratory (DE-AC05-00OR22725)
- Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory
- Office of Science
- Exascale Computing Project (17-SC-20-SC)
- U.S. Department of Energy Office of Science
- National Nuclear Security Administration
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- BERT, clinical text classification, document classification, pre-training, hierarchical self-attention
- MeSH terms
- Humans, Natural Language Processing, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
