SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,501–1,600 of 3,596
Aracena, C., Villena, F., Rojas, M. S., & Dunstan, J. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.22 · Cited by 10
Zheng, Y., Hao, A., & Luu, A. T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.813 · Cited by 10
Multi2Claim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking
Tan, N. Ö., Nguyen, T., Bensemann, J., Peng, A. Y., Bao, Q., Chen, Y., GAHEGAN, M. N., & Witbrock, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.194 · Cited by 20
BioPRO: Context-Infused Prompt Learning for Biomedical Entity Linking
Zhu, T., Qin, Y., Feng, M., Chen, Q., Hu, B., & Xiang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3331149 · Cited by 12
PRM-KGED: paper recommender model using knowledge graph embedding and deep neural network
Thierry, N., Bao, B., Ali, Z., Tan, Z., Chatelain, I. B. C., & Kefalas, P. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05162-7 · Cited by 9
“…of an academic article C p , k , SPECTER embeds the concatenated input text that contains the title, and the abstract, using the following equation: where Trans indicates the Transformer’s forward function that is SciBERT [12] over an input token [CLS] that is separated with [SEP] token.”
Semi-automatic staging area for high-quality structured data extraction from scientific literature
Foppiano, L., Tomoya, M. A. T. O., Terashima, K., Suárez, P. O., Tou, T., Sakai, C., Wang, W.-S., Amagasa, T., Takano, Y., & Ishii, M. (2023) · Science and Technology of Advanced Materials Methods · doi:10.1080/27660400.2023.2286219 · Cited by 8
“We prepared our experiment using the SciBERT [21] implementation, and we finetuned the model for our downstream NER task discussed in detail in [13].”
MethodDifference-Masking: Choosing What to Mask in Continued Pretraining
Wilf, A., Akter, S., Mathur, L., Liang, P. P., Mathew, S., Shou, M., Nyberg, E., & Morency, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.881 · Cited by 12
Beyond Good Intentions: Reporting the Research Landscape of NLP for Social Good
Adauto, F. G., Jin, Z., Schölkopf, B., Hope, T., Sachan, M., & Mihalcea, R. F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.31 · Cited by 13
Wu, D., Ahmad, W. U., & Chang, K.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.410 · Cited by 8
ConfliBERT-Spanish: A Pre-trained Spanish Language Model for Political Conflict and Violence
Yang, W., Alsarra, S., Abdeljaber, L., Zawad, N., Delaram, Z., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2023) · IEEE Congress on Information Science and Technology (CiSt) · doi:10.1109/cist56084.2023.10409883 · Cited by 8
“After building the corpus, we developed a Spanish-conflict-specific language model using the BERT-based structure, which has shown successful performance [9], [11], [13]–[17].”
BackgroundMethodModelling Text Similarity: A Survey
Mu, W., & Lim, K. H. (2023) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3625007.3627305 · Cited by 11
“However, when contextualised embedding starts to show great potential in various NLP tasks [55], many similarity schemes use contextualised embeddings as their main features.”
BackgroundGO2Sum: generating human-readable functional summary of proteins from GO terms
Giri, S. J., Ibtehaz, N., & Kihara, D. (2024) · npj Systems Biology and Applications · doi:10.1038/s41540-024-00358-0 · Cited by 8
Hirano, M., & Imajo, K. (2024) · IIAI International Congress on Advanced Applied Informatics (IIAI-AAI) · doi:10.1109/iiai-aai63651.2024.00059 · Cited by 10
“Back to the era of BERT, SciB-ERT [20], MedBERT [21], Japanese BERT 1 , and Japanese financial BERT [22] are proposed.”
BackgroundMethodThe Effect of Pretraining on Extractive Summarization for Scientific Documents
Gupta, Y., Ammanamanchi, P. S., Bordia, S., Manoharan, A., Mittal, D., Pasunuru, R., Shrivastava, M., Singh, M., Bansal, M., & Jyothi, P. (2021) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2021.sdp-1.9 · Cited by 10
“Without any intermediate pretraining, SCIBERT offers a small advantage over BERT on Pubmed and is statistically comparable to BERT on both SCITLDR tasks.”
BackgroundMethodInfluentialA Case-Based Reasoning Model Powered by Deep Learning for Radiology Report Recommendation.
Amador-Domínguez, E., Serrano, E., Manrique, D., & Bajo, J. (2021) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2021.08.011 · Cited by 8
Textomics: A Dataset for Genomics Data Summary Generation
Wang, M.-C., Liu, Z., & Wang, S. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2022.acl-long.335 · Cited by 8
Kato, S., Kanegami, K., & Kano, M. (2022) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2022.07.568 · Cited by 9
Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation
Gaschi, F., Plesse, F., Rastin, P., & Toussaint, Y. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892197 · Cited by 10
“BERT, and many of its proposed variations like RoBERTa [9], SciBERT [10], or BioBERT [11], are pre-trained and evaluated solely in English.”
FeedLens: Polymorphic Lenses for Personalizing Exploratory Search over Knowledge Graphs
Kaur, H., Downey, D., Singh, A., Cheng, E. Y.-Y., Weld, D. S., & Bragg, J. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545631 · Cited by 12
“The other model represents each paper with SPECTER embeddings, which are produced by a pre-trained language model (SciBERT [3]) fne-tuned on a citation-prediction objective and have been shown”
MethodMa, Y., Liu, J., Lü, W., & Cheng, Q. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103315 · Cited by 9
Syntax-Informed Self-Attention Network for Span-Based Joint Entity and Relation Extraction
Zhang, H., Zhang, G., & Ma, R. (2021) · Applied Sciences · doi:10.3390/app11041480 · Cited by 8
“In addition, for SciERC, the SciBERT [54] model will be adopted, which is trained on papers from the corpus of semanticsholar.org specifically.”
MethodD’Souza, J., & Auer, S. (2021) · Journal of Data and Information Science · doi:10.2478/jdis-2021-0023 · Cited by 16
Team Phoenix at WASSA 2021: Emotion Analysis on News Stories with Pre-Trained Language ModelsPreprint
Butala, Y., Singh, K., Kumar, A., & Shrivastava, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06057 · Cited by 13
“Domain specific language models also leads to a significant performance gain (Vaidhya and Kaushal, 2020; Lee et al., 2019; Beltagy et al., 2019).”
BackgroundMeta-research on COVID-19: An overview of the early trendsPreprint
Colavizza, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02961 · Cited by 9
UniRE: A Unified Label Space for Entity Relation ExtractionPreprint
Wang, Y., Sun, C., Wu, Y., Zhou, H., Li, L., & Yan, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.04292 · Cited by 13
“For fair comparison with previous works, we use three pre-trained language models: bert-base-uncased (Devlin et al., 2019), albert-xxlarge-v1 (Lan et al., 2019) and scibert-scivocab-uncased (Beltagy et al., 2019) as the sentence encoder and fine-tune them in training stage.”
BackgroundMethodComputational Job Market Analysis with Natural Language ProcessingPreprint
Zhang, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.18977 · Cited by 12
Abstractive Summarization Model for Summarizing Scientific Article
Ülker, M., & Özer, A. B. (2024) · IEEE Access · doi:10.1109/access.2024.3420163 · Cited by 11
Serialized EHR make for good text representationsPreprint
Chou, Z., Qin, Q., & Shi, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.13843 · Cited by 8
“SerialBEHRT marks an advancement in medical language modeling, building on the foundation established by SciBERT [Beltagy et al., 2019b].”
InfluentialLi, Y., Kong, C., Zhao, G., & Zhao, Z. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11337-0 · Cited by 7
“For instance, certain studies (Alfarghaly et al. 2021), among others adopt a direct approach, to initialize graph nodes by using SciBert (Beltagy et al. 2019) and utilizing pre-trained distilGPT2 (Li et al. 2023a; Sanh et al. 2019b) decoders.”
Automatic analysis of available source code of top artificial intelligence conference papers
Lin, J., Wang, Y., Yu, Y., Zhou, Y., Chen, Y., & Shi, X. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500358 · Cited by 7
COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews
Singh, S., Singh, M., & Goyal, P. (2021) · 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl52503.2021.00068 · Cited by 7
Schrader, T. P., Finco, M., Grünewald, S., Hildebrand, F., & Friedrich, A. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.11 · Cited by 7
Xu, C., Ding, J., Wang, B., Qiao, Y., Zhang, L., & Zhang, Y. F. (2024) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2024.100748 · Cited by 14
Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generation
Kang, S., Jin, B., Kweon, W., Zhang, Y., Lee, D., Han, J., & Yu, H. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703544 · Cited by 16
Ensembling approaches to citation function classification and important citation screening
Jiang, X. (2025) · Scientometrics · doi:10.1007/s11192-025-05265-7 · Cited by 9
“In addition, all of them relied on one way of encoding in the wide spectrum of modelling options, e.g., self-attention over contextualised word embeddings such as SciBERT, which made them inca-pable of utilising the benefits of different modelling methods.”
InfluentialBalakireva, L., & Klein, M. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702514 · Cited by 7
“We trained SciBERT model[3], optimized for scientific text, to capture contextual relationships and classify resource types.”
Agarwal, S., Wood, D. M., Murray, B. A. K., Wei, Y., Busaidi, A. A., Kafiabadi, S., Guilhem, E., Lynch, J., Townend, M., Mazumder, A., et al. (2025) · European Radiology · doi:10.1007/s00330-025-11500-9 · Cited by 7
SciRus: Tiny and Powerful Multilingual Encoder for Scientific Texts
Gerasimenko, N. A., Vatolin, A., Ianina, A., & Vorontsov, K. V. (2024) · Doklady Mathematics · doi:10.1134/s1064562424602178 · Cited by 9
“Starting from the trained SciBERT [1] used as an initialization model, SPECTER is pre-trained on the citation objective using cite and co-cite pairs of papers.”
CyberLLaMA: A fine-tuned large language model for cybersecurity named entity recognition
Zhang, H., Wu, T., Zhu, T., Wen, S., & Xiang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114183 · Cited by 12
Grounded knowledge-enhanced medical vision–language pre-training for chest X-ray
Deng, Q., Huang, Z., Wang, Y., Wang, Z., Wang, Z., Zhang, X., Dou, Q., Hui, Y. Y., & Hui, E. S. K. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108416 · Cited by 9
“SciBERT [33] was adopted as the report encoder, knowledge encoder, and entity encoder.”
Purnomo, R. D., Fatyanosa, T. N., & Data, M. (2025) · doi:10.1109/icnlp65360.2025.11108690 · Cited by 7
“To optimize BERT’s performance, researchers had trained it on domain-specific datasets, as demonstrated by BERT variants such as SciBERT [10] and BioBERT [11].”
McCarroll, N., McShane, P., O’Connell, E., Curran, K., Singh, M., McNamee, E., Clist, A., & Brammer, A. (2025) · SN Computer Science · doi:10.1007/s42979-025-04300-x · Cited by 7
“However, these ‘out-of-the-box’ models have been reported to under - perform in specialised domains as they are highly skewed towards generic language [30 , 31].”
From Symbolic to Neural and Back: Exploring Knowledge Graph–Large Language Model Synergies
Škrlj, B., Koloski, B., Pollak, S., & Lavrač, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-03028-3_11 · Cited by 9
Using Generative AI to Enhance Psychometric Scale Development in Market Research
Keane, D., & McNaughton, R. B. (2025) · International Journal of Market Research · doi:10.1177/14707853251384769 · Cited by 9
Kathuria, D., Aggarwal, S., Negi, A., Barthwal, R., Joshi, A., & Singh, N. (2025) · Food Chemistry X · doi:10.1016/j.fochx.2025.103360 · Cited by 12
“SciBERT, trained on scientific information, was mostly effective for mining literature on oxidative degradation, enzymatic processing, and solvent extraction (Beltagy et al., 2019).”
Liu, J., Song, X., Zhang, D., Thomale, J., He, D., & Hong, L. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1269 · Cited by 11
“In Round 2, the prompt was updated to include both the original metadata and the terms generated in Round 1.”
Scaling multimodal materials representation learning with synthetic narratives
Park, Y. J., Kumaran, M., Hsu, C.-W., Olivetti, E., & Li, J. (2026) · Nature Communications · doi:10.1038/s41467-026-76378-7 · Cited by 9
“To address this issue, Su et al. 33 used the weights ofthe GraphCL 34 , pre-trained on biomedical molecules, and SciBERT 35 , pre-trained on bio-molecularcorpora,astheinitialweightsofthegraphandtextencoder, respectively.”
InfluentialKnowledge-Guided Efficient Representation Learning for Biomedical Domain
Jha, K., Xun, G., Du, N., & Zhang, A. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467118 · Cited by 7
“Following the practice in contextual embedding literature [1, 2, 24], the range of hyperparameters are chosen to be the following: learning ratewithin the range [1e-5, 3e-5, 5e-5], batch size [10, 16, 32, 64] and epoch number [2–60].”
BackgroundMethodInfluentialTo be or not to be an Integer? Encoding Variables for Mathematical Text
Ferreira, D. M., Thayaparan, M., Valentino, M., Rozanova, J., & Freitas, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.76 · Cited by 7
Saeed, N., & Naveed, H. (2022) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2022.928530 · Cited by 8
Entity Coreference and Co-occurrence Aware Argument Mining from Biomedical Literature
Liu, B., Schlegel, V., Batista-Navarro, R., & Ananiadou, S. (2023) · Workshop on Computational Approaches to Discourse (CODI) · doi:10.18653/v1/2023.codi-1.6 · Cited by 9
Evidence > Intuition: Transferability Estimation for Encoder Selection
Bassignana, E., Müller-Eberstein, M., Zhang, M., & Plank, B. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.283 · Cited by 7
Bomgni, A. B., Abdala, D., Gurung, B. D. S., Nkenlifack, M., Gadhamshetty, V. R., & Gnimpiéba, E. Z. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385403 · Cited by 7
“SciBERT [2] addresses the limitations of generic language models in processing scientific literature due to the unique vocabulary, syntax, and structure of scientific texts.”
BackgroundSumBlogger: Abstractive Summarization of Large Collections of Scientific Articles
Zakkas, P., Verberne, S., & Zavrel, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_23 · Cited by 16
Quellcodekritik. Zur Philologie von Algorithmen
Bajohr, H., & Krajewski, M. (2024) · doi:10.52438/avaa1004 · Cited by 9
Domain Knowledge-Driven Relation Extraction Methods
Chen, B., & Yuan, G. (2024) · Journal of Data Science and Intelligent Systems · doi:10.47852/bonviewjdsis42022524 · Cited by 7
Zhang, Y., & Zhang, C. (2024) · Scientometrics · doi:10.1007/s11192-024-05048-6 · Cited by 7
“Consequently, we choose for SCIBERT as the pretrained language model for subsequent experiments.”
InfluentialShahzadi, I., Madni, T. M., Janjua, U. I., Batool, G., Naz, B., & Ali, M. Q. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01126-6 · Cited by 12
“Pre-trained transformers have frequent use on large corpus [24], and text-based transformers, such as BioBERT [53], SciBERT [54], and ClinicalBERT [55], are also commonly used in the healthcare field.”
Knowledge enhanced edge-driven graph neural ranking for biomedical information retrieval
Liu, X., Tan, J., & Dong, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125787 · Cited by 7
Torres, A. E., de Moura, E. S., da Silva, A. S., Nascimento, M. A., & Mesquita, F. (2026) · Natural language processing. · doi:10.1017/nlp.2026.10019 · Cited by 14
“We used SciBERT [34], a fine-tuned version of BERT for scientific texts, for datasets from the biomedical and materials science domains, and regular multilingual BERT for the other datasets.”
Dayarathne, R. N., Ranaweera, U., & Ganegoda, G. U. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-9255-9_26 · Cited by 7
“To give a brief idea of the vectorisation process; SPECTER will encode the abstracts using the SciB-ERT [2] transformer and then take the final representation of [CLS] token as the final representation of the abstract [4] as indicated in Fig.”
Automating Chapter-Level Classification for Electronic Theses and Dissertations
Banerjee, B., Ingram, W. A., & Fox, E. A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825418 · Cited by 8
“Early transformer-based language models such as BERT [9], SciBERT [10], and RoBERTa [11] handle text with short context length, whereas models such as BigBird Pegasus [12] and Longformer [13] are capable of handling up to a 4096 token length.”
Chatrinan, K., Noraset, T., & Tuarob, S. (2025) · Scientometrics · doi:10.1007/s11192-025-05233-1 · Cited by 7
“GAN-CITE leverages SciBERT (Beltagy et al., 2019), a pre-trained language model for scientific text, as its backbone classifier.”
InfluentialLi, W. (2025) · Education Sciences · doi:10.3390/educsci15020207 · Cited by 9
Wang, Y., Cao, P., Fang, H., & Ye, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107250 · Cited by 8
Safdar, M., Xie, J., Mircea, A., & Zhao, Y. F. (2025) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4068275 · Cited by 8
NLP and Neural Networks for Insider Threat Detection
Baghalizadeh-Moghadam, N., Neal, C., Cuppens, F., & Cuppens-Boulahia, N. (2024) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom63139.2024.00279 · Cited by 7
“2) SciBERT + LSTM Integration: SciBERT, pretrained on scientific literature, generates domain-specific embeddings that outperform general models like BERT in technical contexts [6].”
Kumawat, H., Sharan, A., & Verma, S. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.587 · Cited by 8
MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model
Reddy, K. S., Ragavenderan, N., Vasanth, K., Naik, G. N., Vishalakshi Prabhu, H., & Nagaraja, G. S. (2025) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v14.i3.pp2367-2378 · Cited by 7
“These models include SciBERT [8], ClinicalBERT [7], and BioBERT [4].”
Liu, R., Li, J., Zivkovic, M., & Abbasi, A. (2025) · MIS Quarterly · doi:10.25300/misq/2024/18573 · Cited by 10
Harnessing language models for computational literature review of emerging AI topics
Chung, J., Jeong, B., Park, Y.-J., Jeong, H., Lee, J., Yoon, J., & Choi, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104245 · Cited by 7
Knowledge transfer, knowledge gaps, and knowledge silos in citation networks
Cunningham, E., & Greene, D. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329302 · Cited by 9
“This allows us to analyse how research areas emerge, grow, merge, split or dissipate as the field evolves.”
ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research
Lin, J., Dai, L., Han, R., Sui, Y., Wang, R., Sun, X., Wu, Q., Feng, M., Liu, H., & Xiong, H. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737432 · Cited by 14
“The abstract and introduction sections should be combined into "abstract&introduction" - Method sections might also be called "methodology", "approach", "proposed method", etc.”
InfluentialUPB at SemEval-2020 Task 6: Pretrained Language Models for Definition Extraction
Avram, A.-M., Cercel, D.-C., & Chiru, C.-G. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.97 · Cited by 8
Layer-wise Guided Training for BERT: Learning Incrementally Refined Document Representations
Manginas, N., Chalkidis, I., & Malakasiotis, P. (2020) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2020.spnlp-1.7 · Cited by 11
Predicting Clinical Trial Results by Implicit Evidence Integration
Jin, Q., Tan, C., Chen, M., Liu, X., & Huang, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.114 · Cited by 9
Gene Ontology aided Compound Protein Binding Affinity Prediction Using BERT Encoding
Zhao, L., Xie, P., Hao, L., Li, T., & Wang, C. (2020) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9312985 · Cited by 7
“By applying the transfer learning technique, bioBERT [13], sci-BERT [14], and NCBIBERT [15] are trained for specific domains based on BERT on textual database such as PubMed abstract(https://www.”
MethodLeveraging Google BERT to Detect and Measure Innovation Discussed in News Articles
Chen, K., Cosgro, B., Domfeh, O., Stern, A., Korkmaz, G., & Kattampallil, N. A. (2021) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds52267.2021.9483744 · Cited by 7
Sarrouti, M., Abacha, A. B., & Demner‐Fushman, D. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00058 · Cited by 9
A Keyword Detection and Context Filtering Method for Document Level Relation Extraction
Kuang, H., Chen, H., Ma, X., & Liu, X. (2022) · Applied Sciences · doi:10.3390/app12031599 · Cited by 7
Seneviratne, S., Daskalaki, E., Lenskiy, A. A., & Suominen, H. (2022) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2022.clinicalnlp-1.3 · Cited by 7
GLADIS: A General and Large Acronym Disambiguation Benchmark
Chen, L., Varoquaux, G. P., & Suchanek, F. M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.152 · Cited by 7
Muthukkumaran, A., Raghunathan, S., Ravichandran, A., & Rengaswamy, R. (2023) · AIChE Journal · doi:10.1002/aic.18068 · Cited by 9
Researcher influence prediction (ResIP) using academic genealogy network
Kumar, D., Bhowmick, P. K., & Paik, J. H. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101392 · Cited by 8
Domain-Driven and Discourse-Guided Scientific Summarisation
Goldsack, T., Zhang, Z., Lin, C., & Scarton, C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_23 · Cited by 16
Gupta, S., Bouadjenek, M. R., & Robles‐Kelly, A. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110685 · Cited by 8
SciLander: Mapping the Scientific News Landscape
Gruppi, M., Smeros, P., Adalı, S., Castillo, C., & Aberer, K. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22144 · Cited by 7
Rescue Implicit and Long-tail Cases: Nearest Neighbor Relation Extraction
Wan, Z., Liu, Q., Mao, Z., Cheng, F., Kurohashi, S., & Li, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.113 · Cited by 13
ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts
Murthy, S., Lo, K., King, D., Bhagavatula, C., Kuehl, B., Johnson, S., Borchardt, J., Weld, D. S., Hope, T., & Downey, D. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.20 · Cited by 11
Self-citationArici, N., Gerevini, A., Olivato, M., Putelli, L., Sigalini, L., & Serina, I. (2023) · Future Internet · doi:10.3390/fi15080268 · Cited by 7
RAR-SB: research article recommendation using SciBERT with BiGRU
Thierry, N., Bao, B., & Ali, Z. (2023) · Scientometrics · doi:10.1007/s11192-023-04840-0 · Cited by 10
“In particular, the approach takes the title, abstract, author that published the papers, and field of study(FoS)/keywords as input data and generates the representations of academic articles using a pre-trained SciBERT (Beltagy et al., 2019).”
MethodInfluentialA machine learning-enabled open biodata resource inventory from the scientific literature
Imker, H. J., Schackart, K., Istrate, A.-M., & Cook, C. E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294812 · Cited by 9
Zhou, X., Fu, Q., Xia, Y., Wang, Y., Lu, Y., Chen, Y., & Chen, J. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3358169 · Cited by 7
“Given a document D , we flatten it and feed it into the PLMs for context encoding to obtain the symbol-level word segmentation feature: where Encoder is the PLMs encoder (e.g., SciBERT [19]) and H token is the hidden state of its last output layer.”
MethodPicha, S. G., Chanti, D. A., & Caplier, A. (2024) · International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC) · doi:10.5220/0012429600003657 · Cited by 7
“Notably, the pre-trained BERT model known as SciB-ERT (Beltagy et al., 2019) demonstrated better performance compared to the other BERT models, even outperforming the standard BERT model, which secured the second position in this evaluation.”
Investigating Zero- and Few-shot Generalization in Fact Verification
Pan, L., Zhang, Y., & Kan, M. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.34 · Cited by 13
Detection of Tortured Phrases in Scientific Literature
Martel, E., Lentschat, M., & Labbé, C. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.6 · Cited by 7
Pipelined biomedical event extraction rivaling joint learning
Wu, P., Li, X., Gu, J., Qian, L., & Zhou, G. (2024) · Methods · doi:10.1016/j.ymeth.2024.04.003 · Cited by 7
A framework for predicting scientific disruption based on graph signal processing
Yu, H., & Liang, Y. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103863 · Cited by 7
RoBERTaLexPT: um modelo RoBERTa jurídico pré-treinado com deduplicação para língua Portuguesa
Garcia, E., da Silva, N. F. F., Gomes, J., Albuquerque, H., de Souza, E. P. R., Siqueira, F., Lima, E., & Carvalho, A. (2024) · Linguamática · doi:10.21814/lm.16.2.457 · Cited by 8
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.