SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,401–1,500 of 3,596
APIE: An information extraction module designed based on the pipeline method
Jiang, X., Cheng, Y., Zhang, S., Wang, J., & Ma, B. (2023) · Array · doi:10.1016/j.array.2023.100331 · Cited by 9
Pre-training LLMs using human-like development data corpus
Bhardwaj, K., Shah, R. S., & Varma, S. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.30 · Cited by 9
MediBioDeBERTa: Biomedical Language Model With Continuous Learning and Intermediate Fine-Tuning
Kim, E., Jeong, Y., & Choi, M.-S. (2023) · IEEE Access · doi:10.1109/access.2023.3341612 · Cited by 13
Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps
Wei, Y., Zhang, T., Zhang, H., Zhong, T., Zhao, L., Liu, Z., Ma, C., Zhang, S., Shang, M., Du, L., et al. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385933 · Cited by 9
“From the methodological perspective, Chat2Brain employs ChatAUG as a data augmentation strategy (in Fig.2), utilizes SciBERT [27] as the text encoder, and employs the 3D convolutional layer as the brain activation map generator.”
MethodParallel sequence tagging for concept recognition
Furrer, L., Cornelius, J., & Rinaldi, F. (2022) · BMC Bioinformatics · doi:10.1186/s12859-021-04511-y · Cited by 10
A New Citation Recommendation Strategy Based on Term Functions in Related Studies Section
Chen, H. (2021) · Journal of Data and Information Science · doi:10.2478/jdis-2021-0022 · Cited by 9
Using neural networks to mine text and predict metabolic traits for thousands of microbes
Hackmann, T. J., & Zhang, B. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008757 · Cited by 9
Detect and Classify – Joint Span Detection and Classification for Health Outcomes
Abaho, M., Bollegala, D., Williamson, P. R., & Dodd, S. R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.686 · Cited by 15
“The dataset has supported clinicalNLP research tasks (Beltagy et al., 2019; Brockmeier et al., 2019).”
MethodLu, H., Li, L., Li, Z., & Zhao, S. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103874 · Cited by 9
BioPREP: Deep learning-based predicate classification with SemMedDB
Hong, G., Kim, Y., Choi, Y.-J., & Song, M. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103888 · Cited by 11
Relation Extraction using Multiple Pre-Training Models in Biomedical Domain
Hiai, S., Shimada, K., Watanabe, T., Miura, A., & Iwakura, T. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_060 · Cited by 9
Decouple knowledge from paramters for plug-and-play language modeling
Cheng, X., Lin, Y., Chen, X., Zhao, D., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.901 · Cited by 21
Deep Anomaly Detection by Residual AdaptationPreprint
Deecke, L., Ruff, L., Vandermeulen, R. A., & Bilen, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02310 · Cited by 9
Biomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 11
SciDr at SDU-2020: IDEAS -- Identifying and Disambiguating Everyday Acronyms for Scientific DomainPreprint
Singh, A., & Kumar, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08818 · Cited by 9
“…extended to using Transformers (Vaswani et al. 2017) based architecture, BERT (Devlin et al. 2018) as the backbone of the model; followed by SciBERT (Beltagy, Lo, and Cohan 2019), which too is a BERT-based model, but is pretrained on text from scientific research papers instead of Wikipedia corpus.”
Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text ClassificationPreprint
Lee, J., Kim, J., & Kang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10474 · Cited by 9
“trained LM with the task-relevant data before finetuning phase was proposed (Beltagy et al., 2019; Sun et al., 2019; Lee et al., 2020; Gururangan et al., 2020).”
MethodLegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-TrainingPreprint
Clavié, B., Gheewala, A., Briton, P., Alphonsus, M., Laabiyad, R., & Piccoli, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.00993 · Cited by 11
“Models such as BioBERT (Lee et al., 2019) and SciBERT (Beltagy et al., 2019) have greatly improved the state-of-the-art of NLP tasks related to biomedical and scientific text.”
BackgroundMethodLearning to Teach with Student FeedbackPreprint
Liu, Y., Sun, T., Qiu, X., & Huang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04641 · Cited by 12
Inteligencia artificial y salud
Oficina, C., de Ciencia, O., & Diputados, T. D. C. D. L. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.57952/tcsx-b678 · Cited by 10
Review of Extreme Multilabel ClassificationPreprint
Dasgupta, A., Lamba, P., Kushwaha, A., Ravish, K., Katyan, S., Das, S., & Kumar, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.05971 · Cited by 9
A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models
Rouhizadeh, H., Nikishina, I., Yazdani, A., Bornet, A., Zhang, B., Ehrsam, J., Gaudet-Blavignac, C., Naderi, N., & Teodoro, D. (2024) · Scientific Data · doi:10.1038/s41597-024-03317-w · Cited by 9
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Hsu, T.-Y., Huang, C.-Y., Rossi, R., Kim, S., Giles, C., & Huang, T.-H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.363 · Cited by 8
Basu, S., Chakraborty, S., Hassan, A., Siddique, S., & Anand, A. (2020) · Proceedings of the First Workshop on Scholarly Document Processing · doi:10.18653/v1/2020.sdp-1.15 · Cited by 8
Automating medical simulations
Gershov, S., Braunold, D., Spektor, R., Ioscovich, A., Raz, A., & Laufer, S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104446 · Cited by 8
Ensemble Text Summarization Model for COVID‐19‐Associated Datasets
Chellatamilan, T., Narayanasamy, S. K., Garg, L., Srinivasan, K., & Islam, S. M. N. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/3106631 · Cited by 9
Fonteyn, L., Manjavacas, E., Haket, N., Dorst, A. G., & Kruijt, E. (2024) · Linguistics Vanguard · doi:10.1515/lingvan-2022-0142 · Cited by 8
LERCause: Deep learning approaches for causal sentence identification from nuclear safety reports
Kim, J., Kim, J., Lee, A., Kim, J., & Diesner, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308155 · Cited by 8
“We build, test, and assess an architecture that integrates BERT-oriented language models, i.e., BERT, BioBERT (a BERT model optimized for biomedical text mining [8]), and SciBERT (a BERT model tailored to scientific literature [9]).”
BackgroundMethodInfluentialGuo, B., Meng, J., Zhao, D., Jia, X., Chu, Y., & Lin, H. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104717 · Cited by 8
Bridging text and crystal structures: literature-driven contrastive learning for materials science
Suzuki, Y., Taniai, T., Igarashi, R., Saito, K., Chiba, N., Ushiku, Y., & Ono, K. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ade58c · Cited by 9
“Our study builds on more recent advances in language modeling, specifically by adopting SciBERT [25] as the text encoder (section 2) and utilizing Llama 3 [26] for keyword extraction from title-abstract pairs in dataset generation (section 7.4).”
InfluentialKnowledge graphs in heterogeneous catalysis: Recent advances and future opportunities
Díaz, R., & Xin, H. (2025) · Chinese Journal of Chemical Engineering · doi:10.1016/j.cjche.2025.06.008 · Cited by 10
Rubio-Martín, S., García-Ordás, M. T., García, A. S., Pato, C. M. F., Crespo-Álvaro, A., & Benítez‐Andrades, J. A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3045 · Cited by 10
“…function, where l ( y i , y ˆ i ) represents the error function, and Ω ( f k ) penalizes model complexity to prevent overfitting: (5) i = 1 k • SciBERT (Beltagy et al., 2019): An adapted version of BERT (Bidirectional Encoder Representations from Transformers) that is specifically trained on…”
Shen, H., Cao, H., Wang, L., Ai, Z., & Dai, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114170 · Cited by 12
Wu, M., Sivertsen, G., Zhang, L., Qi, F., & Zhang, Y. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70004 · Cited by 10
“The classification boundaries observed in models such as fine-tuned Bio-BERT, SsciBERT, Ssci-SciBERT, and SentenceBERT are relatively distinct, consistent with their higher performance rankings.”
InfluentialBevendorff, J., Dementieva, D., Fröbe, M., Gipp, B., Greiner-Petter, A., Jussi Karlgren, C., Mayerl, M., Nakov, P., Panchenko, A., Potthast, M., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_21 · Cited by 15
Rai, N., Schumann, A. W., & Boyd, N. S. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00534 · Cited by 8
Strategic Management Knowledge Map via BERTopic (1980–2025): Evolution, Integration, and Application
Lai, K., Hsiao, C.-W., & Hsu, Y.-J. (2025) · Applied System Innovation · doi:10.3390/asi8050142 · Cited by 8
“To enhance accuracy, domain-specific models such as SciBERT [53] and PatentSBERTa are utilized.”
Multiple agroecological practices use and climate change mitigation. A review
Wezel, A., Marchetti, A., Nichenametla, C. K., Boughamoura, O., Kamilia, K., & Bàrberi, P. (2025) · Agronomy for Sustainable Development · doi:10.1007/s13593-025-01048-9 · Cited by 8
“On this final dataset, we compared the model previously used to different types of transformers such as a BERT (Devlin et al. 2018), a RoBERTa (Liu et al. 2019), a sciBERT (Beltagy et al. 2019), and a DeBERTa-v3 (He et al. 2021).”
Relation-Faceted Graph Pooling with LLM Guidance for Dynamic Span-Aware Information Extraction
Baek, H.-Y., Choi, J., Seo, J., Jin, X., Lee, D., & Oh, B. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761243 · Cited by 10
“For token encoding, we adopt DeBERTa [14] for ACE05 and CoNLL04, and SciBERT [1] for SciERC.”
Using language models to label clusters of scientific documents
Murray, D., Ni, C., Gu, W., & Hubbard, T. (2025) · Scientometrics · doi:10.1007/s11192-025-05445-5 · Cited by 8
“Vectors were generated using SPECTER (Cohan, Feldman, Beltagy, Downey, & Weld, 2020), a fine-tuned variant of SciBERT (Beltagy, Lo, & Cohan, 2019) that takes as input the title and abstract of each document and generates a dense vector representation.”
CiteBART: Learning to Generate Citations for Local Citation Recommendation
Çelik, E. Y., & Tekır, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.89 · Cited by 13
BiBLoX realtime science trend mapping MLdriven forecasting
Kesgin, K., & Özer, D. Z. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00693-z · Cited by 8
Ko, Y. S., Parkinson, J., & Wang, W. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag014 · Cited by 9
“Similarly, ProteinDT introduced the Contrastive Language and Protein pre-training (ProteinCLAP) approach to align the representations between SciBert [42] embeddings of Swiss-Prot annotations and ProtBert [23] embeddings.”
Generative Artificial Intelligence Models: A Survey
Almuammar, M., Alsultan, K., & Altukhaim, Y. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11546-1 · Cited by 10
Text Mining in Bibliometrics and Science Mapping: A Methodological Review
Misuraca, M. (2026) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.70066 · Cited by 10
“Recent advances in transformer-based language models, including domain-specific variants such as SciBERT (Beltagy et al. 2019), BioBERT (Lee et al. 2019), and Galactica (Taylor et al. 2022), have further expanded the representational capacity of bibliometric text analysis by enabling the extraction of contextualized d…”
Argumentation Mining in Scientific Literature: From Computational Linguistics to Biomedicine.
Accuosto, P., Neves, M., & Saggion, H. (2021) · Repositori digital de la UPF (Universitat Pompeu Fabra) · Cited by 8
BLAR: Biomedical Local Acronym Resolver
Hogan, W. R., Baeza, Y. V., Katsis, Y., Baldwin, T., Kim, H., & Hsu, C. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.14 · Cited by 8
Jiang, M., D’Souza, J., Auer, S., & Downie, J. S. (2021) · International Journal on Digital Libraries · doi:10.1007/s00799-021-00313-y · Cited by 8
“[7] have developed SciBert which are Bert models trained on scholarly publications.”
InfluentialRule-Enhanced Active Learning for Semi-Automated Weak Supervision
Kartchner, D., An, D. N., Ren, W., Zhang, C., & Mitchell, C. S. (2022) · AI · doi:10.3390/ai3010013 · Cited by 8
“[6,7] have extended benefits to new domains.”
Intent-Controllable Citation Text Generation
Jung, S.-Y., Lin, T., Liao, C., Yuan, S., & Sun, C. (2022) · Mathematics · doi:10.3390/math10101763 · Cited by 18
BEEDS: Large-Scale Biomedical Event Extraction using Distant Supervision and Question Answering
Wang, X. D., Leser, U., & Weber, L. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.28 · Cited by 14
AAEBERT: Debiasing BERT-based Hate Speech Detection Models via Adversarial Learning
Okpala, E., Cheng, L., Mbwambo, N. M. J., & Luo, F. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00053 · Cited by 13
Badenes-Olmedo, C., & Corcho, Ó. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104382 · Cited by 12
Learning to rank query expansion terms for COVID-19 scholarly search
Khader, A., & Ensan, F. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104386 · Cited by 8
Leveraging Domain Knowledge for Inclusive and Bias-aware Humanitarian Response Entry Classification
Tamagnone, N., Fekih, S., Contla, X., Orozco, N., & Rekabsaz, N. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/690 · Cited by 8
Zaki, M. F. M., Jan, A., Krishnan, N. M. A., & Mauro, J. C. (2023) · MRS Bulletin · doi:10.1557/s43577-023-00560-1 · Cited by 9
“71 Zhao et al. following a similar methodology of pretraining and fine-tuning, 61,64,69,72 created BERT-a based language model, namely, OpticalBERT, which is trained on research papers related to optical materials.”
MethodBetter with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks
Xu, J., Huang, R., Jiang, X., Cao, Y., Yang, C., Wang, C., & Yang, Y. (2023) · PubMed · doi:10.48550/arxiv.2311.01038 · Cited by 20
“Regarding the pre-training datasets, arxiv, dblp and patents-main are citation networks collected from [2], [37] and [9], respectively.”
BackgroundExploring the Impact of Corpus Diversity on Financial Pretrained Language Models
Choe, J., Noh, K., Kim, N., Ahn, S., & Jung, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.138 · Cited by 8
“This trend has been extended to further pretraining of PLMs on domain-specific corpus in various fields such as biomedical (Lee et al., 2020), scientific (Beltagy et al., 2019), and clinical (Alsentzer et al., 2019) domains.”
BackgroundPadilla-Cuevas, J., Reyes‐Ortiz, J. A., Rasgado, A. D. C., Mora-Gutiérrez, R. A., & Bravo, M. (2024) · Applied Sciences · doi:10.3390/app14167031 · Cited by 9
Multi-Task Multi-Attention Transformer for Generative Named Entity Recognition
Mo, Y., Liu, J., Tang, H., Wang, Q., Xu, Z., Wang, J., Quan, X., Wu, W., & Li, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3458796 · Cited by 9
Kabal, O., Harzallah, M., Guillet, F., & Ichise, R. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.436 · Cited by 9
Ozawa, K., Suzuki, T., Tonogai, S., & Itakura, T. (2024) · Science and Technology of Advanced Materials Methods · doi:10.1080/27660400.2024.2406219 · Cited by 12
Predicting Tumor Type and Residual Status of Suprasellar Lesions Using Indian Discharge Summaries
Nair, P. C., Gupta, D., Devi, B. I., Kanjirangat, V., & Deepak, P. (2024) · IEEE Access · doi:10.1109/access.2024.3460976 · Cited by 9
Advancing Molecular Graph-Text Pre-training via Fine-grained Alignment
Li, Y., Fang, Y., Zhang, M., & Shi, C. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736834 · Cited by 13
“We use SciBERT [3], which has been pre-trained on texts from the chemical and biological domains, as our text encoder, denoted as 𝑓 bert .”
InfluentialGaliani, S., Gálvez, R. H., & Nachman, I. (2024) · Economic Inquiry · doi:10.1111/ecin.13261 · Cited by 10
LmRaC: a functionally extensible tool for LLM interrogation of user experimental results
Craig, D. B., & Drăghici, S. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae679 · Cited by 9
“In particular, LLMs and their precursors (Beltagy et al. 2019, Lee et al. 2020, Gu et al. 2021, Luo et al. 2022) have rapidly moved toward human levels of competence in biology.”
Chen, Q., Zhou, W., Cheng, J., & Yang, J. X. (2024) · Applied Sciences · doi:10.3390/app142411529 · Cited by 9
Towards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddings
Sengupta, S., Heaton, C., Cui, S., Sarkar, S., & Mitra, P. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821824 · Cited by 11
“Although BERT, trained on open-domain text such as Wikipedia, is proficient in MRC [3], it requires massive amounts of unlabelled medical corpora to work well on medical QA (BioBERT [6] and SciBERT [7] adopt this strategy).”
Douard, N., Samet, A., Giakos, G., & Cavallucci, D. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7010007 · Cited by 9
“For example, Beltagy et al. [29] introduced SciBERT, a pretrained language model based on BERT, trained on scientific text.”
Constantino, I., Kojaku, S., Fortunato, S., & Ahn, Y. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00349 · Cited by 9
Discontinuous named entities in clinical text: A systematic literature review
Alhassan, A., Schlegel, V., Aloud, M., Batista-Navarro, R., & Nenadić, G. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104783 · Cited by 11
Umeike, R., Getty, N., Xia, F., & Stevens, R. L. (2025) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi60581.2025.10981159 · Cited by 8
Evolution of ESG-focused DLT research: An NLP analysis of the literature
Cruz, W. H., Tylinski, K., Moore, A. P., Roche, N., Vadgama, N., Treiblmaier, H., Shangguan, J., Tasca, P., & Xu, J. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.7 · Cited by 8
“…offers a scalable and adaptable approach, positioning it as an essential tool for researchers, policymakers, and practitioners to navigate the growing complexity of publications in DLT and related disciplines to understand its evolution further and assess the impact of potential changes.”
IIITBH-IITP@CL-SciSumm20, CL-LaySumm20, LongSumm20
Reddy, S. M., Saini, N., Saha, S., & Bhattacharyya, P. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.27 · Cited by 12
Advances in scientific literature mining for interpreting materials characterization
Park, G., & Pouchard, L. C. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abf751 · Cited by 8
Zaharia, G.-E., Cercel, D.-C., & Dascălu, M. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.77 · Cited by 8
Probing Pre-Trained Language Models for Disease Knowledge
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.266 · Cited by 13
Full-Abstract Biomedical Relation Extraction with Keyword-Attentive Domain Knowledge Infusion
Zhu, X., Zhang, L., Du, J., & Xiao, Z. (2021) · Applied Sciences · doi:10.3390/app11167318 · Cited by 8
“In addition to the base BERT, two of its variants, BioBERT [29], and SciBERT [30] have emerged with a stronger embedding capability to work with scientific publications and gain popularity in the RE task [31–34].”
Pattern-Based Acquisition of Scientific Entities from Scholarly Article Titles
D’Souza, J., & Auer, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91669-5_31 · Cited by 8
DataHunter: A System for Finding Datasets Based on Scientific Problem Descriptions
Färber, M., & Leisinger, A.-K. (2021) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3460231.3478882 · Cited by 8
“In preliminary research [8], we implemented several text classification methods combined with text representation methods: (1) classification based on tfidf similarity; (2) classification based on the BM25 score; (3) random forest; (4) logistic regression; (5) Gaussian, multinomial and complement naïve Bayes; (6) conv…”
MethodLi, J., Wei, Q., Ghiasvand, O., Chen, M., Lobanov, V. S., Weng, C., & Xu, H. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00095 · Cited by 9
Drug knowledge discovery via multi-task learning and pre-trained models
Li, D., Xiong, Y., Hu, B., Tang, B., Peng, W., & Chen, Q. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01614-7 · Cited by 9
“…there are likewise several pre-trained models that follow BERT architecture leveraging domain-specific knowledge effectively from a large set of unannotated biomedical texts (e.g., PubMed abstracts, clinical notes), such as SciBERT [11], BioBERT [12], NCBI BERT [13], Clinical BERT [14, 15].”
BackgroundGupta, K. C., Ahmad, A., Ghosal, T., & Ekbal, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91669-5_34 · Cited by 8
Sboev, A. G., Selivanov, A., Александрович, М. И., Борисович, Р. Р., Gryaznov, A. V., Sboeva, S., & Rylkov, G. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6010010 · Cited by 8
PSRMTE: Paper submission recommendation using mixtures of transformer
Nguyen, D. H., Huynh, S. T., Dinh, C. V., Huynh, P. T., & Nguyen, B. T. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117096 · Cited by 8
LAME: Layout-Aware Metadata Extraction Approach for Research Articles
Choi, J., Kong, H., Yoon, H.-M., Oh, H., & Jung, Y. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.025711 · Cited by 8
Detecting and analyzing missing citations to published scientific entities
Lin, J., Yao, Y., Song, J., & Shi, X. (2022) · Scientometrics · doi:10.1007/s11192-022-04334-5 · Cited by 8
“We can use BLEU (Papineni et al, 2002) and SciBERT (Beltagy et al, 2019) in comparison as a distinct case in point.”
BackgroundMethodInfluentialA Syntax-enhanced model based on category keywords for biomedical relation extraction
Liu, X., Tan, J., Fan, J., Tan, K., Hu, J., & Dong, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104135 · Cited by 9
Paul, D., Chakdar, D., Saha, S., & Mathew, J. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3187342 · Cited by 8
“The SciBERT is pre-trained on scientific documents.”
MethodInfluentialData-Centric and Model-Centric Approaches for Biomedical Question Answering
Yoon, W., Yoo, J., Seo, S., Sung, M., Jeong, M., Kim, G., & Kang, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13643-6_16 · Cited by 10
Emati: a recommender system for biomedical literature based on supervised learning
Kart, Ö., Mestiashvili, A., Lachmann, K., Kwasnicki, R., & Schroeder, M. (2022) · Database · doi:10.1093/database/baac104 · Cited by 8
“SCI-BERT is a pretrained language model that has the same architecture as BERT but is trained on a large corpus of scientific text (25).”
MethodA Multi-Gate Encoder for Joint Entity and Relation Extraction
Xiong, X., Liu, Y., Liu, A., Gong, S., & Li, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18315-7_11 · Cited by 10
Deep representation learning of scientific paper reveals its potential scholarly impact
Jiang, Z., Lin, T., & Huang, C. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101376 · Cited by 8
ADPG: Biomedical entity recognition based on Automatic Dependency Parsing Graph
Yang, Y., Lin, H., Yang, Z., Zhang, Y., Zhao, D., & Huai, S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104317 · Cited by 8
Scientific Paper Recommendation System
Singh, R., Gaonkar, G., Bandre, V., Sarang, N., & Deshpande, S. (2023) · IEEE International Conference for Convergence in Technology (I2CT) · doi:10.1109/i2ct57861.2023.10126196 · Cited by 8
“The SciBert[25] network processes the tokenized information to generate meaningful latent embedding of the paper’s abstract.”
BackgroundMethodInfluentialBiomedical Relation Extraction Using Dependency Graph and Decoder-Enhanced Transformer Model
Kim, S., Yoon, J., & Kwon, O. (2023) · Bioengineering · doi:10.3390/bioengineering10050586 · Cited by 10
“…(NSP), where the model is trained to predict whether one sentence follows another, as shown in REVIEW We initialized the model with SCIBERT [23] for drug-related relationship extraction in order to leverage the domain specific knowledge and then fine-tuned all of the parameters using…”
MethodInfluential#ChronicPain: Automated Building of a Chronic Pain Cohort from Twitter Using Machine Learning
Lakamana, S., Guo, Y., Guo, Y., Leslie, A., Okunromade, O., Yang, Y., Gonzalez‐Polledo, E., Perrone, J., McKenzie-Brown, A. M., & Sarker, A. (2023) · Health Data Science · doi:10.34133/hds.0078 · Cited by 9
“SciBERT: This model is trained on a large corpus of scientific data and with the same model structure as BERT [35].”
SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
Bashlovkina, V., Matthews, R., Kuang, Z., Baumgartner, S., & Bendersky, M. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599907 · Cited by 9
“such as scientific literature [6], biomedical text [24, 37], and electronic health records [54], some of the most popular social media language models [4, 5, 14, 26, 38, 56] are explicitly pretrained on data from a single social media platform.”
BackgroundSchopf, T., Gerber, E., Ostendorff, M., & Matthes, F. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_113 · Cited by 13
“We use SciBERT (Beltagy et al., 2019), SPECTER (Cohan et al., 2020), and DeCLUTR (Giorgi et al., 2021) in their base -versions as published by their authors without any fine-tuning on our corpus.”
MethodInfluential
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.