SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 401–500 of 3,596
SEARCHFORMER: Semantic patent embeddings by siamese transformers for prior art search
Vowinckel, K., & Hähnke, V. D. (2023) · World Patent Information · doi:10.1016/j.wpi.2023.102192 · Cited by 50
Diao, S., Xu, T., Xu, R., Wang, J., & Zhang, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.280 · Cited by 78
Transformer networks of human conceptual knowledge.
Bhatia, S., & Richie, R. (2024) · Psychological Review · doi:10.1037/rev0000319 · Cited by 59
Guo, X., Lu, S., Tang, Z., Bai, Z., Diao, L., Zhou, H., & Li, L. (2022) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2022.106776 · Cited by 42
Biomedical Event Extraction as Multi-turn Question Answering
Wang, X. D., Weber, L., & Leser, U. (2020) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2020.louhi-1.10 · Cited by 48
Document-Level Relation Extraction With Context Guided Mention Integration and Inter-Pair Reasoning
Zeng, D., Zhao, C., Jiang, C., Zhu, J., & Dai, J. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3316454 · Cited by 42
“In our CGM2IR implementation, we use cased BERT-base [28] or RoBERTa-large [34] the encoder on DocRED and cased SciBERT-base [35] on CDR and GDA.”
MethodIs it a good move? Mining effective tutoring strategies from human–human tutorial dialogues
Lin, J., Singh, S., Sha, L., Tan, W., Lang, D., Gašević, D., & Chen, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.09.001 · Cited by 57
A rule-free workflow for the automated generation of databases from scientific literature
Gilligan, L. P. J., Cobelli, M., Taufour, V., & Sanvito, S. (2023) · npj Computational Materials · doi:10.1038/s41524-023-01171-9 · Cited by 53
“Furthermore, there have been a number of BERT architectures adapted to outperform the conventional BERT model in various disparate NLP domains, including but not limited to the fields of general scientific literature [31], financial sentiment analysis [32], and biomedical text-mining [33].”
BackgroundGani, M. O., Ayyasamy, R. K., Sangodiah, A., & Fui, Y. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11842-1 · Cited by 50
“SciBERT (Beltagy et al., 2019), which was trained using scientific corpora, and BioBERT (Lee et al., 2020), which was trained using biomedical corpora, are examples of domain-specific embedding.”
MethodGeneralized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs
Yu, X., Liu, Z., Fang, Y., Liu, Z., Chen, S., & Zhang, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3419109 · Cited by 72
“In this section, we give the problem definition and introduce the background of GNNs.”
Narechania, A. A., Karduni, A., Wesslen, R. S., & Wall, E. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114820 · Cited by 49
Efficient Hierarchical Domain Adaptation for Pretrained Language Models
Chronopoulou, A., Peters, M., & Dodge, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.96 · Cited by 24
Adapting BERT to Implicit Discourse Relation Classification with a Focus on Discourse Connectives
Kishimoto, Y., Murawaki, Y., & Kurohashi, S. (2020) · Language Resources and Evaluation · Cited by 73
Goldberg, D. M. (2021) · Journal of Safety Research · doi:10.1016/j.jsr.2021.12.024 · Cited by 41
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
Luu, K., Khashabi, D., Gururangan, S., Mandyam, K., & Smith, N. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.435 · Cited by 41
Past and future uses of text mining in ecology and evolution
Farrell, M. J., Brierley, L., Willoughby, A. R., Yates, A., & Mideo, N. (2022) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2021.2721 · Cited by 50
“There also exist systematic inequalities in the representation and performance of NLP technologies across languages [91,100]: largely because of the historical dominance of English as the lingua franca of scientific publishing, current scientific language models are designed only for English texts [101,102].”
BackgroundLabels in a haystack: Approaches beyond supervised learning in biomedical applications
Yakimovich, A., Beaugnon, A., Huang, Y., & Özkırımlı, E. (2021) · Patterns · doi:10.1016/j.patter.2021.100383 · Cited by 50
Extracting a Knowledge Base of Mechanisms from COVID-19 Papers
Hope, T., Amini, A., Wadden, D., van Zuylen, M., Parasa, S., Horvitz, E., Weld, D. S., Schwartz, R., & Hajishirzi, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.355 · Cited by 45
Liu, Y., Checa, M., & Vasudevan, R. K. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad52e9 · Cited by 55
“In this regard, it is expected that LLMs more specifically trained on the scientific literature, with frameworks to manage the use of LLMs and dictate the standards for accuracy, would be significantly better at these tasks [53].”
BackgroundSE3M: A model for software effort estimation using pre-trained embedding models
Fávero, E. M. D. B., Casanova, D., & Pimentel, A. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106886 · Cited by 20
DATE: Detecting Anomalies in Text via Self-Supervision of Transformers
Manolache, A., Brad, F., & Burceanu, E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.25 · Cited by 59
Cross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40
“These abstracts are mostly, but not entirely, biomedical (Beltagy et al., 2019).”
BackgroundExploring Privileged Features for Relation Extraction with Contrastive Student-Teacher Learning
Xiaoyan, Z., Yang, M., Qu, Q., Xu, R., & Li, J. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3161584 · Cited by 51
“For the SciERC dataset, we follow the previous work and use SciBERT [58], which is pretrained on a large number of science papers, to encode sentences.”
MethodOn the importance of pre-training data volume for compact language models
Micheli, V., d’Hoffschmidt, M., & Fleuret, F. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.632 · Cited by 37
aeroBERT-NER: Named-Entity Recognition for Aerospace Requirements Engineering using BERT
Ray, A. T., Pinon-Fischer, O. J., Mavris, D. N., White, R. T., & Cole, B. F. (2023) · AIAA SciTech Forum · doi:10.2514/6.2023-2583 · Cited by 52
Exploring the use of ChatGPT in learning and instructing statistics and data analytics
Xing, Y. (2024) · Teaching Statistics · doi:10.1111/test.12367 · Cited by 48
ExpBERT: Representation Engineering with Natural Language Explanations
Murty, S., Koh, P. W., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.190 · Cited by 40
Automatic detection of COVID-19 vaccine misinformation with graph link prediction
Weinzierl, M., & Harabagiu, S. M. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103955 · Cited by 48
“This process of further pre-training has been shown to improve performance on down-stream tasks in various scientific [45], biomedical [46], and social media [47] domains.”
AnchiBERT: A Pre-Trained Model for Ancient Chinese Language Understanding and Generation
Tian, H., Yang, K., Liu, D., & Lv, J. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534342 · Cited by 58
JuriBERT: A Masked-Language Model Adaptation for French Legal Text
Douka, S., Abdine, H., Vazirgiannis, M., Hamdani, R. E., & Amariles, D. R. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.9 · Cited by 61
GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models
Rajpoot, P. K., & Parikh, A. P. (2023) · Workshop on Financial Technology and Natural Language Processing (FinNLP) · doi:10.18653/v1/2023.finnlp-2.5 · Cited by 40
Xu, Z., Xu, H., Lu, Z., Zhao, Y., Zhu, R., Wang, Y., Dong, M., Chang, Y., Lv, Q., Dick, R. P., et al. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3659600 · Cited by 69
“This work is inspired by the powerful language generation capabilities of LLMs [9, 17, 57, 58] and motivated by their lack of personal context awareness necessary to establish common ground.”
BackgroundInfluentialBoosting Transformers and Language Models for Clinical Prediction in Immunotherapy
Chen, Z., Balan, M. M., & Brown, K. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.32 · Cited by 43
Liu, Z., Li, H., Wang, H., Liao, Y., Liu, X., & Wu, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120435 · Cited by 43
COVIDSum: A linguistically enriched SciBERT-based summarization model for COVID-19 scientific papers
Cai, X., Liu, S., Yang, L., Lu, Y., Zhao, J., Shen, D. G., & Liu, T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.103999 · Cited by 52
Demir, G. B., Süküt, Y., Duran, G. S., Topsakal, K. G., & Görgülü, S. (2024) · European Journal of Orthodontics · doi:10.1093/ejo/cjae011 · Cited by 33
LegalDB: Long DistilBERT for Legal Document Classification
Bambroo, P., & Awasthi, A. (2021) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect49130.2021.9392558 · Cited by 48
“BERT-BASE has been outperformed by its variations like BioBERT [5], SciBERT [6], etc. which either have been further pre-trained or fine-tuned on domain-specific corpora.”
MethodMachine Learning and Health Science Research: Tutorial
Cho, H., She, J., de Marchi, D., El-Zaatari, H., Barnes, E. L., Kahkoska, A. R., Kosorok, M. R., & Virkud, A. (2024) · Journal of Medical Internet Research · doi:10.2196/50890 · Cited by 43
“For example, for medical tasks, “Med-BERT: pretrained contextualized embeddings on large-scale structured EHR for disease prediction” and “SciBERT: a pretrained language model for scientific text” may be of use, as they are trained on similar language as is used in medical contexts [60,61,66].”
Fine-Tuning BERT Model for Materials Named Entity Recognition
Zhao, X., Greenberg, J., An, Y., & Hu, X. T. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671697 · Cited by 47
Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks
Yan, Z., Yang, S., Liu, W., & Tu, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.467 · Cited by 58
MacAvaney, S., Mittu, A., Coppersmith, G., Leintz, J., & Resnik, P. (2021) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/2021.clpsych-1.7 · Cited by 38
“It is well known that systems perform better when they are trained on materials similar to the materials the system will run on (Alsentzer et al., 2019; Beltagy et al., 2019).”
BackgroundKjell, O., Giorgi, S., & Schwartz, H. A. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/293kt · Cited by 39
BERT models for Brazilian Portuguese: Pretraining, evaluation and tokenization analysis
de Souza, F. C., Nogueira, R. F., & Lotufo, R. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110901 · Cited by 50
Chandra, R., & Ranjan, M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0273476 · Cited by 50
Schindler, D., Bensmann, F., Dietze, S., & Krüger, F. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.835 · Cited by 55
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.-W. D., Wei, X., Arnold, A., & Ren, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.351 · Cited by 90
Rehearsal-free Continual Language Learning via Efficient Parameter Isolation
Wang, Z., Liu, Y., Ji, T., Wang, X., Wu, Y., Jiang, C., Ye, C., Han, Z., Wang, L., Shao, X., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.612 · Cited by 81
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
“The models used include GPT-J-6b, SciFive T5 [218], and an approach combining sequence-to-sequence and sciBERT [219] models.”
Semi-Supervised Hierarchical Graph Classification
Li, J., Huang, Y., Chang, H., & Rong, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3203703 · Cited by 44
“We collect 4666 Computer Science (CS) arXiv papers indexed by Microsoft Academic Graph (MAG) [50].”
MethodAviation-BERT: A Preliminary Aviation-Specific Natural Language Model
Chandra, C., Jing, X., Bendarkar, M. V., Sawant, K., Elias, L. R., Kirby, M. R., & Mavris, D. N. (2023) · AIAA AVIATION Forum · doi:10.2514/6.2023-3436 · Cited by 49
“SciBERT is a similarly pre-trained from scratch BERT model that was trained on scientific text corpus [17].”
BackgroundMethodAdapting transformer-based language models for heart disease detection and risk factors extraction
Houssein, E. H., Mohamed, R. E., Hu, G., & Ali, A. A. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00903-y · Cited by 41
“Pre-training from scratch using domain-specific corpora and vocabulary improves the performance of models SciBERT [76], PubMed-BERT [77], and Biolm [78].”
MethodTopoAct: Visually Exploring the Shape of Activations in Deep Learning
Rathore, A., Chalapathi, N., Palande, S., & Wang, B. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14195 · Cited by 50
Enhancing evidence-based medicine with natural language argumentative analysis of clinical trials.
Mayer, T., Marro, S., Cabrio, E., & Villata, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102098 · Cited by 36
“Interestingly here, the SciBERT cased model performs the best with a F1-score of .65.”
MethodInfluentialUsing BERT to identify drug-target interactions from whole PubMed
Aldahdooh, J., Vähä-Koskela, M. J. V., Tang, J. P., & Tanoli, Z. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04768-x · Cited by 42
Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks
Peng, B., Chersoni, E., Hsu, Y.-Y., & Huang, C. (2021) · Workshop on Economics and Natural Language Processing (ECONLP) · doi:10.18653/v1/2021.econlp-1.5 · Cited by 54
The Impact of Domain-Specific Pre-Training on Named Entity Recognition Tasks in Materials SciencePreprint
Walker, N., Trewartha, A., Huo, H., Lee, S.-H., Cruse, K., Dagdelen, J. M., Dunn, A., Persson, K. A., Ceder, G., & Jain, A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3950755 · Cited by 50
Distinguishing between fake news and satire with transformers
Low, J. F., Fung, B. C. M., Iqbal, F., & Huang, S.-C. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115824 · Cited by 41
“pre-training from scratch, such as SciBERT (pre-trained on scientific papers (Beltagy et al., 2019)), the pre-training in this work was not begun from scratch with randomly initialized weights but was instead started from distilbert-base-uncased weights. This decision was made out of concern for DistilBERT’s language…”
ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing
Nguyen, N., Phan, T., Nguyen, D.-V., & Nguyen, K. V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.315 · Cited by 51
Perspective: Large Language Models in Applied Mechanics
Brodnik, N. R., Carton, S., Muir, C., Ghosh, S., Downey, D., Echlin, M. P., Pollock, T. M., & Daly, S. H. (2023) · Journal of Applied Mechanics · doi:10.1115/1.4062773 · Cited by 38
D’Souza, J., Auer, S., & Ted Pedersen, A. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.44 · Cited by 52
Obayashi, T., Kodate, S., Hibara, H., Kagaya, Y., & Kinoshita, K. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac983 · Cited by 41
“We used a SciBERT pre-trained model (allenai/scibert scivocab uncased) (42) from Hugging Face’s Transformers library (43) for fine tuning of the binary classification of the positive and negative sentences (learning rate = 1e-05, epochs = 3).”
MethodBioMRC: A Dataset for Biomedical Machine Reading Comprehension
Pappas, D., Stavropoulos, P., Androutsopoulos, I., & McDonald, R. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.15 · Cited by 29
AraLegal-BERT: A pretrained language model for Arabic Legal text
Al‐Qurishi, M., Alqaseemi, S., & Souissi, R. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.31 · Cited by 47
Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs
Mavromatis, C., Ioannidis, V. N., Wang, S., Zheng, D., Adeshina, S., Ma, J., Zhao, H., Faloutsos, C., & Karypis, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43418-1_10 · Cited by 50
Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation
Yu, Y., Wu, F., Wu, C., Yi, J., & Liu, Q. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.368 · Cited by 18
“Even though this paradigm has achieved great success, it suffers from the known domain shift problem between the pre-training and down-stream corpus (Howard and Ruder, 2018; Lee et al., 2019; Beltagy et al., 2019).”
Nugent, T., Stelea, N., & Leidner, J. L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86967-0_12 · Cited by 53
Mahdaouy, A. E., Lamsiyah, S., Idrissi, M. J., Alami, H., Yartaoui, Z., & Berrada, I. (2026) · Journal of Network and Systems Management · doi:10.1007/s10922-025-10010-9 · Cited by 40
“To address this issue, domain-specific PLMs have been developed, such as BioBERT [34] for biomedical text and SciBERT [35] for scientific literature.”
ExplainaBoard: An Explainable Leaderboard for NLP
Liu, P., Fu, J., Xiao, Y., Yuan, W., Chang, S., Dai, J., Liu, Y., Ye, Z., & Neubig, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.acl-demo.34 · Cited by 35
“This is true both for classical tasks such as machine translation (Sutskever et al., 2014; Wu et al., 2016), as well as for new tasks (Lu et al., 2016; Rajpurkar et al., 2016), domains (Beltagy et al., 2019), and languages (Conneau and Lample, 2019).”
BackgroundNaturally occurring language as a source of evidence in suicide prevention
Resnik, P., Foreman, A., Kuchuk, M., Schafer, K. M., & Pinkham, B. (2020) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12674 · Cited by 42
Retraining a BERT Model for Transfer Learning in Requirements Engineering: A Preliminary Study
Ajagbe, M. A., & Zhao, L. (2022) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re54965.2022.00046 · Cited by 35
“Retraining dataset: Our retraining dataset contains 7 million words, which is comparable with the retraining dataset used for SCIBERT (1.14 million words) [15].”
MethodInfluentialCesario, E., Comito, C., & Zumpano, E. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127079 · Cited by 39
Annotating and Extracting Synthesis Process of All-Solid-State Batteries from Scientific LiteraturePreprint
Kuniyoshi, F., Makino, K., Ozawa, J., & Miwa, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07339 · Cited by 47
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation
Diao, S., Xu, R., Su, H., Jiang, Y., Song, Y., & Zhang, T. (2021) · doi:10.18653/v1/2021.acl-long.259 · Cited by 61
Training-free Lexical Backdoor Attacks on Language Models
Huang, Y., Zhuo, T. Y., Xu, Q., Hu, H., Yuan, X., & Chen, C. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583348 · Cited by 63
“To draw more public interest, the attackers can claim their published web API and extension has achieved state-of-the-art performance (e.g., SMART RoBERTa [27] in sentiment analysis) or the published model is unique in a specific domain, such as LEGAL-BERT [13] and SciBERT [8].”
MethodMiao, B. Y., Sushil, M., Xu, A., Wang, M., Arneson, D. V., Berkley, E., Subash, M., Vashisht, R., Rudrapatna, V. A., & Butte, A. J. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00244-3 · Cited by 37
“15,16 These pipelines can also map extracted concepts to existing biomedical vocabularies, such as MeSH categories, for standardisation and downstream analysis.”
BackgroundRaza, S., & Schwartz, B. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02117-3 · Cited by 41
Predicting article quality scores with machine learning: The U.K. Research Excellence Framework
Thelwall, M. A., Kousha, K., Wilson, P., Makita, M., Abdoli, M., Stuart, E., Levitt, J. M., Knoth, P., & Cancellieri, M. (2023) · Quantitative Science Studies · doi:10.1162/qss_a_00258 · Cited by 40
AcTune: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models
Yu, Y., Kong, L., Zhang, J., Zhang, R., & Zhang, C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.102 · Cited by 59
Multimodal learning on graphs for disease relation extraction
Lin, Y., Lu, K., Yu, S., Cai, T., & Žitnik, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104415 · Cited by 45
Advancing surgical VQA with scene graph knowledge
Yuan, K., Kattel, M., Lavanchy, J. L., Navab, N., Srivastav, V., & Padoy, N. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03141-y · Cited by 76
Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review
Rostam, Z. R. K., & Kertész, G. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3763002 · Cited by 42
Transferability of Natural Language Inference to Biomedical Question AnsweringPreprint
Jeong, M., Sung, M., Kim, G., Kim, D., Yoon, W., Yoo, J., & Kang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00217 · Cited by 42
Risk of bias assessment in preclinical literature using natural language processing
Wang, Q., Liao, J., Lapata, M., & Macleod, M. R. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1533 · Cited by 37
Utilizing BERT for biomedical and clinical text mining
Zhu, R., Tu, X., & Huang, J. X. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-819314-3.00005-7 · Cited by 40
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05006 · Cited by 34
“Except for the general domain, eforts have been proposed to extend PLMs in speciic domains such as: [71] trains CodeBERT in the programming language and [18] trains SciBERT on scientiic publications and biological sequence.”
BackgroundMethodInfluentialXue, Z., He, G., Liu, J., Jiang, Z., Zhao, S. X., & Lu, W. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103216 · Cited by 38
GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
Gao, Y., Xiong, Y., Wang, S., & Wang, H. (2022) · Applied Sciences · doi:10.3390/app122412942 · Cited by 45
Recycle-BERT: Extracting Knowledge about Plastic Waste Recycling by Natural Language Processing
Kumar, A., Bakshi, B. R., Ramteke, M., & Kodamana, H. (2023) · ACS Sustainable Chemistry & Engineering · doi:10.1021/acssuschemeng.3c03162 · Cited by 37
EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain
Lin, C., Miller, T., Dligach, D., Bethard, S. J., & Savova, G. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.21 · Cited by 55
Jin, R., Ye, Q., Wang, J., Cao, Z., Jiang, D., Wang, T., Kang, Y., Xu, W., Hsieh, C., & Hou, T. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae304 · Cited by 44
TaxoEnrich: Self-Supervised Taxonomy Completion via Structure-Semantic Representations
Jiang, M., Song, X., Zhang, J., & Han, J. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511935 · Cited by 34
“In our preliminary experiments, we found that SciBERT is better than other models in representing concept representations.”
MethodSong, D., Gao, S., He, B., & Schilder, F. (2022) · IEEE Access · doi:10.1109/access.2022.3190408 · Cited by 45
FinSoSent: Advancing Financial Market Sentiment Analysis through Pretrained Large Language Models
Delgadillo, J., Kinyua, J., & Mutigwe, C. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8080087 · Cited by 43
Aspect-based Document Similarity for Research Papers
Ostendorff, M., Ruas, T., Blume, T., Gipp, B., & Rehm, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.545 · Cited by 48
Protein Language Models: Applications and Perspectives
Leclercq, M., & Droit, A. (2025) · Journal of Proteome Research · doi:10.1021/acs.jproteome.5c00506 · Cited by 43
PaECTER: Patent-level representation learning using citation-informed transformers
Ghosh, M., Rose, M. E., Erhardt, S., Buunk, E., & Harhoff, D. (2026) · PLoS ONE · doi:10.1371/journal.pone.0358188 · Cited by 39
“They make use of the existing capabilities to model scientific language by taking SciBERT as a foundation and generate a training dataset with so-called triplets.”
InfluentialBanner: A Cost-Sensitive Contextualized Model for Bangla Named Entity Recognition
Ashrafi, I., Mohammad, M., Mauree, A. S., Nijhum, G. M. A., Karim, R., Mohammed, N., & Momen, S. (2020) · IEEE Access · doi:10.1109/access.2020.2982427 · Cited by 39
Gong, A., Qiao, Z., Li, X., Lyu, J., & Li, X. (2024) · Progress in Nuclear Energy · doi:10.1016/j.pnucene.2024.105474 · Cited by 35
Knowledge Graphs for COVID-19: An Exploratory Review of the Current Landscape
Chatterjee, A., Nardi, C., Oberije, C., & Lambin, P. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11040300 · Cited by 33
“[5] used nearest neighbors on RDF2vec embeddings, whereas [14], besides using SciBERT embeddings, also defined novel metrics such as topic and citation similarity.”
MethodCitation recommendation employing heterogeneous bibliographic network embedding
Ali, Z., Qi, G., Muhammad, K., Bhattacharyya, S., Ullah, I., & Abro, W. A. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06135-y · Cited by 36
“The SPECTER model is initialized by using SciBERT [56], which is trained on a large corpus of scientific documents, defined as follows.”
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.