A question-and-answer system to extract data from free-text oncological pathology reports (CancerBERT Network): development study
Open accessPublished version of a preprint
Abstract
BACKGROUND: Information in pathology reports is critical for cancer care. Natural language processing (NLP) systems used to extract information from pathology reports are often narrow in scope or require extensive tuning. Consequently, there is growing interest in automated deep learning approaches. A powerful new NLP algorithm, bidirectional encoder representations from transformers (BERT), was published in late 2018. BERT set new performance standards on tasks as diverse as question answering, named entity recognition, speech recognition, and more. OBJECTIVE: The aim of this study is to develop a BERT-based system to automatically extract detailed tumor site and histology information from free-text oncological pathology reports. METHODS: We pursued three specific aims: extract accurate tumor site and histology descriptions from free-text pathology reports, accommodate the diverse terminology used to indicate the same pathology, and provide accurate standardized tumor site and histology codes for use by downstream applications. We first trained a base language model to comprehend the technical language in pathology reports. This involved unsupervised learning on a training corpus of 275,605 electronic pathology reports from 164,531 unique patients that included 121 million words. Next, we trained a question-and-answer (Q&A) model that connects a Q&A layer to the base pathology language model to answer pathology questions. Our Q&A system was designed to search for the answers to two predefined questions in each pathology report: What organ contains the tumor? and What is the kind of tumor or carcinoma? This involved supervised training on 8197 pathology reports, each with ground truth answers to these 2 questions determined by certified tumor registrars. The data set included 214 tumor sites and 193 histologies. The tumor site and histology phrases extracted by the Q&A model were used to predict International Classification of Diseases for Oncology, Third Edition (ICD-O-3), site and histology codes. This involved fine-tuning two additional BERT models: one to predict site codes and another to predict histology codes. Our final system includes a network of 3 BERT-based models. We call this CancerBERT network (caBERTnet). We evaluated caBERTnet using a sequestered test data set of 2050 pathology reports with ground truth answers determined by certified tumor registrars. RESULTS: caBERTnet's accuracies for predicting group-level site and histology codes were 93.53% (1895/2026) and 97.6% (1993/2042), respectively. The top 5 accuracies for predicting fine-grained ICD-O-3 site and histology codes with 5 or more samples each in the training data set were 92.95% (1794/1930) and 96.01% (1853/1930), respectively. CONCLUSIONS: We have developed an NLP system that outperforms existing algorithms at predicting ICD-O-3 codes across an extensive range of tumor sites and histologies. Our new system could help reduce treatment delays, increase enrollment in clinical trials of new therapies, and improve patient outcomes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BERT data extraction performance exceeded 90% accuracy in pan-cancer contexts.
“Fang et al [22] and Mitchell et al (2022) [23] reported that the data extraction performance of BERT exceeded 90% accuracy in pan-cancer contexts.”
From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
(2016) · Springer eBooks · doi:10.1007/978-3-319-40618-3 · Cited by 17,280 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · see citing works
The Inevitable Application of Big Data to Health Care
Murdoch, T. B., & Detsky, A. S. (2013) · JAMA · doi:10.1001/jama.2013.393 · Cited by 1,676 · see citing works
Federated Learning for Healthcare Informatics
Xu, J., Glicksberg, B. S., Su, C., Walker, P., Bian, J., & Wang, F. (2021) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-020-00082-4 · Cited by 1,542 · cited 1× · see citing works
“…on ground truth labeling of training and test data sets from highly standardized registry data or the application of federated multitask learning [75] that distributes copies of a central model to multiple spoke sites for tuning of the central model on local data could allow vast improvements in…”
BackgroundWhat Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4828 · Cited by 1,514 · see citing works
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · see citing works
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906 · see citing works
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
Neumann, M., King, D., Beltagy, I., & Ammar, W. (2019) · BioNLP Workshop and Shared Task (BioNLP) · doi:10.18653/v1/w19-5034 · Cited by 772 · see citing works
Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M. R., Weissenborn, D., Krithara, A., Petridis, S., Polychronopoulos, D., et al. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0564-6 · Cited by 598 · see citing works
Deep learning in clinical natural language processing: a methodical review
Wu, S., Roberts, K. E., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz200 · Cited by 495 · see citing works
Murff, H. J., FitzHenry, F., Matheny, M. E., Gentry, N., Kotter, K., Crimin, K. S., Dittus, R. S., Rosen, A. K., Elkin, P. L., Brown, S. H., et al. (2011) · JAMA · doi:10.1001/jama.2011.1204 · Cited by 475 · see citing works
Pre-training of Graph Augmented Transformers for Medication Recommendation
Shang, J., Ma, T., Xiao, C., & Sun, J. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/825 · Cited by 311 · see citing works
Han, W., Zhang, Z., Zhang, Y., Yu, J., Chiu, C., Qin, J., Gulati, A., Pang, R., & Wu, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2059 · Cited by 290 · see citing works
Meystre, S. M., & Haug, P. J. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.11.004 · Cited by 200 · see citing works
Coden, A. R., Savova, G., Sominsky, I. L., Tanenblatt, M. A., Masanz, J. J., Schuler, K. K., Cooper, J., Guan, W., & de Groen, P. C. (2008) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.12.005 · Cited by 177 · see citing works
Li, F., Jin, Y., Liu, W., Rawat, B. P. S., Cai, P., & Yu, H. (2019) · JMIR Medical Informatics · doi:10.2196/14830 · Cited by 161 · see citing works
Deep Learning for Automated Extraction of Primary Sites From Cancer Pathology Reports
Qiu, J. X., Yoon, H., Fearn, P., & Tourassi, G. D. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2700722 · Cited by 146 · see citing works
Hierarchical attention networks for information extraction from cancer pathology reports
Gao, S. B., Young, M. T., Qiu, J. X., Yoon, H., Christian, J. B., Fearn, P., Tourassi, G. D., & Ramanthan, A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx131 · Cited by 141 · see citing works
Garcia, S., & Miller, S. (2019) · American Mathematical Society eBooks · doi:10.1090/mbk/121/79 · Cited by 122 · see citing works
Buckley, J. M., Coopey, S. B., Sharko, J., Polubriaginof, F. C. G., Drohan, B., Bellı, A. K., Kim, E. M. H., Garber, J. E., Smith, B. L., Gadd, M. A., et al. (2012) · Journal of Pathology Informatics · doi:10.4103/2153-3539.97788 · Cited by 113 · see citing works
Thoburn, K. K., German, R. R., Lewis, M., Nichols, P., Ahmed, F., & Jackson‐Thompson, J. (2007) · Cancer · doi:10.1002/cncr.22566 · Cited by 96 · see citing works
A frame semantic overview of NLP-based information extraction for cancer-related EHR notes
Datta, S., Bernstam, E. V., & Roberts, K. E. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103301 · Cited by 92 · see citing works
Pre-trained Language Model for Biomedical Question Answering
Yoon, W., Lee, J., Kim, D., Jeong, M., & Kang, J. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43887-6_64 · Cited by 77 · see citing works
Contextual Chatbot for Healthcare Purposes (using Deep Learning)
Kandpal, P., Jasnani, K., Raut, R., & Bhorge, S. (2020) · World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4) · doi:10.1109/worlds450073.2020.9210351 · Cited by 70 · see citing works
Extracting structured information from free text pathology reports.
Schadow, G., & McDonald, C. J. (2003) · PubMed · Cited by 67 · see citing works
Natural language processing in pathology: a scoping review
Burger, G., Abu–Hanna, A., de Keizer, N. F., et al. (2016) · Journal of Clinical Pathology · doi:10.1136/jclinpath-2016-203872 · Cited by 67 · see citing works
Tahayori, B., Chinniforoush, N., & Akhlaghi, H. (2020) · Emergency Medicine Australasia · doi:10.1111/1742-6723.13656 · Cited by 54 · see citing works
Sentiment Analysis Methods for HPV Vaccines Related Tweets Based on Transfer Learning
Zhang, L., Fan, H., Peng, C., Rao, G., & Cong, Q. (2020) · Healthcare · doi:10.3390/healthcare8030307 · Cited by 50 · see citing works
Artificial intelligence in medicine
Γενιτσαρίδη, Ε. (2020) · doi:10.12681/eadd/48548 · Cited by 48 · see citing works
Nguyen, A. N., Moore, J., O'Dwyer, J., & Philpot, S. P. (2015) · PubMed Central · Cited by 39 · see citing works
Automatic indexing of pathology data
Dunham, G. S., Pacak, M. G., & Pratt, A. W. (1978) · Journal of the American Society for Information Science · doi:10.1002/asi.4630290207 · Cited by 34 · see citing works
Machine learning to parse breast pathology reports in Chinese
Tang, R., Ouyang, L., Li, C., et al. (2018) · Breast Cancer Research and Treatment · doi:10.1007/s10549-018-4668-3 · Cited by 33 · see citing works
Question answering from natural language medical data bases
Grishman, R., & Hirschman, L. (1978) · Artificial Intelligence · doi:10.1016/0004-3702(78)90011-5 · Cited by 32 · see citing works
Schroeck, F. R., Patterson, O. V., Alba, P. R., et al. (2017) · Urology · doi:10.1016/j.urology.2017.07.056 · Cited by 32 · see citing works
Pattern-based information extraction from pathology reports for cancer registration
Napolitano, G., Fox, C., Middleton, R., & Connolly, D. (2010) · Cancer Causes & Control · doi:10.1007/s10552-010-9616-4 · Cited by 31 · see citing works
Automated extraction of Biomarker information from pathology reports
Lee, J., Song, H.-J., Yoon, E., et al. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0609-7 · Cited by 28 · see citing works
Alawad, M., Yoon, H., & Tourassi, G. D. (2018) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi.2018.8333408 · Cited by 26 · see citing works
Xu, D., Gopale, M., Zhang, J., Brown, K., Begoli, E., & Bethard, S. J. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa080 · Cited by 26 · see citing works
BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes
Peng, X., Long, G., Shen, T., Wang, S., Jiang, J. B., & Zhang, C. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00050 · Cited by 21 · see citing works
Identifying Metastases-related Information from Pathology Reports of Lung Cancer Patients.
Soysal, E., Warner, J. L., Denny, J. C., et al. (2017) · PubMed · Cited by 19 · see citing works
Yoon, H., Robinson, S., Christian, J. B., et al. (2018) · IEEE EMBS International Conference on Biomedical & Health Informatics (BHI) · doi:10.1109/bhi.2018.8333439 · Cited by 19 · see citing works
Aurélie Névéol, A., Zweigenbaum, P., & Processing, S. E. F. T. I. Y. S. O. C. N. L. (2018) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1667080 · Cited by 11 · see citing works
Hawhee, V. M., Jones, J. D., Stewart, S., et al. (2020) · Cancer Control · doi:10.1177/1073274820946794 · Cited by 4 · see citing works
Self-citationPattison, E. A., Denhalter, D. W., Patterson, O. V., et al. (2016) · The Journal of Urology · doi:10.1016/j.juro.2016.02.1267 · Cited by 2 · see citing works
Schroeck, F. R., Patterson, O. V., Alba, P. R., et al. (2017) · The Journal of Urology · doi:10.1016/j.juro.2017.02.987 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- [17]Nguyen M Structured pathology reporting for cancer from free text: lung cancer case study Electron J Health Informatics 2012 7 1 10.1109/TELSKS.2001.955803.S.Datta,etal.Journal · DOI
- [23]Devlin J Chang M Lee K Toutanova K BERT: pre-training of deep bidirectional transformers for language understanding Proceedings of NAACL-HLT 2019 2019 Proceedings of NAACL-HLT 2019 Jun 2 - 7, 2019 Minneapolis, Minnesota
- [28]Yang Z Dai Z Yang Y Carbonell J Salakhutdinov R Le Q XLNet: generalized autoregressive pretraining for language understanding ArXiv. Preprint posted online January 2, 2020.
- [29]Lan Z Chen M Goodman S Gimpel K Sharma P Soricut R ALBERT: a Lite BERT for self-supervised learning of language representations ArXiv. Preprint posted online February 9, 2020.
- [30]Liu Y Ott M Goyal N Du J Joshi M Chen D RoBERTa: a robustly optimized BERT pretraining approach ArXiv. Preprint posted online July 26, 2019.
- [31]Sanh V Debut L Chaumond J Wolf T DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter ArXiv. Preprint posted online March 1, 2020.
- [33]Raffel C Shazeer N Roberts A Lee K Narang S Matena M Exploring the limits of transfer learning with a unified text-to-text transformer ArXiv. Preprint posted online July 28, 2020.
- [34]Ju Y Zhao F Chen S Zheng B Yang X Liu Y Technical report on conversational question answering ArXiv Preprint posted online Septmber 29, 2019.
- [38]Huang K Altosaar J Ranganath R ClinicalBERT: modeling clinical notes and predicting hospital readmission ArXiv. Preprint posted online November 29, 2020.
- [45]Ma R Chen P Li G Weng W Lin A Gadepalli K Human-centric metric for accelerating pathology reports annotation ArXiv. Preprint posted online November 12, 2019.
- [47]Paszke A Gross S Massa F Lerer A Bradbury J Chanan G Killeen T Lin Z Gimelshein N Antiga L Desmaison A Köpf A Yang E DeVito Z Raison M Tejani A Chilamkurthy S Steiner B Fang L Bai J Chintala S PyTorch: an imperative style, high-performance deep learning library Proceedings of the 33rd International Conference on Neural Information Processing Systems 2019 Proceedings of the 33rd International Conference on Neural Information Processing Systems Dec 8-14, 2019 Vancouver, Canada 10.5555/3454287.3455008 · DOI
- [49]Jain A Meenachi N Venkatraman B NukeBERT: a pre-trained language model for low resource nuclear domain ArXiv. Preprint posted online August 19, 2020.
- [51]Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez A Attention is all you need Proceedings of the 31st Conference on Neural Information Processing Systems 2017 31st Conference on Neural Information Processing Systems Dec 4 - 9, 2017 Long Beach, CA, USA
- [53]Zhang Z Sabuncu M Generalized cross entropy loss for training deep neural networks with noisy labels Proceedings of the 32nd International Conference on Neural Information Processing Systems 2018 Proceedings of the 32nd International Conference on Neural Information Processing Systems Dec 3 - 8, 2018 Montréal Canada 10.5555/3327546.3327555 PMC11747755 39839708 · DOI
- [74]Settles B From theories to queries: active learning in practice JMLR Workshop Conference Proc 2011 16 1 18
Cited by
Sorted by citations
Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review
Chen, D., Alnassar, S. A., Avison, K. E., Huang, R. S., & Raman, S. (2025) · JMIR Cancer · doi:10.2196/65984 · Cited by 66 · see citing works
“For instance, Gao et al [44] reported that Blue-BERT did not outperform baseline nonLLM models in pan-cancer contexts, while Fang et al [22] and Mitchell et al (2022) [23] reported that the data extraction performance of BERT exceeded 90% accuracy in pan-cancer contexts.”
- Authors
- Mitchell, J. R., Szepietowski, P., Howard, R., Reisman, P., Jones, J. D., Lewis, P., Fridley, B. L., & Rollison, D. E.
- Affiliations
- Alberta Health Services (CA)
- Moffitt Cancer Center (US)
- Corresponding author
- Rollison, Dana E
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-03-23
- Volume
- 24
- Issue
- 3
- Pages
- e27210
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC8987958
- OpenAlex
- W3211439143
- Cited by
- 45
- References
- 53
- Citations by year
- 2026: 11 · 2025: 16 · 2024: 11 · 2023: 2 · 2022: 4 · 2021: 1
- FWCI
- 1.41
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- question answering, BERT, natural language processing, named entity recognition, fine-tuning, unsupervised pre-training, clinical text mining, electronic health records, clinical decision support
- MeSH terms
- Algorithms, Humans, Language, Medical Oncology, Natural Language Processing, Neoplasms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
