SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 3,001–3,100 of 3,596
Self-Supervised Fine-Tuning for Neural Expert Finding
Subagdja, B., Dan, S., & Tan, A. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00014
“SciBERT [21] is trained on scientific text to address scientific language understanding due to the lack of high-quality, large-scale labeled scientific data.”
InfluentialGeneration-Based Few-Shot BioNER via Local Knowledge Index and Dual Prompts
Li, W., Wang, H., Li, W., Zhao, J., & Sun, Y. (2025) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-025-00709-3
“In biomedical NLP, models such as BioBERT [25], CasANGCL [26], and SCIBERT [27] have successfully leveraged domain-specific knowledge to improve text-mining performance.”
Lin, H.-Y., Chang, Y., Yang, P., & Huang, T.-Y. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111094
Data Optimization for LLMs: A Survey
Wu, O. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.36227/techrxiv.174776562.20873028/v1 · Cited by 4
Lange, M., Gogarty, E., Martyn, M., Braude, P., Fayez, F., Koutsouleris, N., Howes, O. D., Twumasi, R., & Carter, B. (2025) · BMC Digital Health · doi:10.1186/s44247-025-00170-8
“There are already available a multitude of specifically medical literature trained models, e.g., BioBERT [57] or SciB-ERT [58], to name only a few.”
Deep Learning for Inverse Polymer Design
Liu, G., Inae, E., & Jiang, M. (2025) · Synthesis lectures on data mining and knowledge discovery · doi:10.1007/978-3-031-84732-5_3
Demers, C. (2025) · Proceedings of the Annual Conference of CAIS / Actes du congrès annuel de l ACSI · doi:10.29173/cais2020
“Successful IE models have combined language models with deep neural networks, such as BERT variants (e.g., SciBERT [Beltagy et al., 2019], RoBERTa [Liu et al., 2019]), Convolutional Neural Networks, and Long Short-Term Memory Networks (Wang et al., 2022).”
Multi-purpose technology commercialization recommender system with large-scale Korean language model
Noh, K., Hwang, H., Lim, Y., Oh, C., Kim, S., Lee, E., Choi, Y., Kim, S., Choi, H., & Song, K. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07341-4
“For instance, SciBERT [19] and BioBERT [3] are trained on scientific and biomedical corpora, respectively, and have achieved strong results in their domains.”
Chen, J., & Li, C. (2025) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait64916.2025.11010778
“…through natural language processing techniques, such as the application of domain adaptation models such as SciBERT in semantic novelty detection [6]; the latter uses graph neural networks to capture knowledge diffusion patterns in citation relationships, such as the practice of GraphSAGE in…”
A Hybrid Framework for Predicting Citation Harm from Retracted Papers
Zhao, J., Li, H., & Tang, S. (2025) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1145/3728725.3728790
“For generating these text embeddings, we considered two prominent Language Models: SPECTER [15] and SciBERT [16].”
InfluentialOptimizing LLaMa2 Fine-Tuning with LoRA and Quantization for Medical Question Answering
Bui, T.-T., Truong, D., & Ngo, L. T. (2025) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-90194-2_34
Exploring the Potential of Large Language Models in Differential Abundance AnalysisPreprint
Franco-Alba, R., Goryanin, I., & Goryanin, I. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.04.657820
Gao, J., Cao, J., Yu, H., Zhang, Y., & Fang, Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113831
Elevating Clinical Semantics: Contrastive Pre-Training Beyond Cross-Entropy in Discharge Summaries
Kim, S., & Jung, Y. (2025) · Applied Sciences · doi:10.3390/app15126541 · Cited by 1
Sriram, R., Sundaram, S. S., & Sophie, S. L. (2025) · IAES International Journal of Robotics and Automation (IJRA) · doi:10.11591/ijra.v14i2.pp273-280
“…with 110 million parameters with uncased tokens, also pre-trained with Books Corpus (800M words) [19] and English Wikipedia (2.5 B words) BioBERT [20] It is pre-trained with PubMed abstracts and Central articles SciBERT [21] It is pre-trained with academic corpus from the semantic scholar…”
InfluentialWITHDRAWNPreprint
(2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ujyec_v3
Darling (v2.0): Mining disease-related databases for the detection of biomedical entity associations
Baltoumas, F. A., Karatzas, E., Venetsianou, N. K., Aplakidou, E., Giatras, K., Chasapi, M. N., Chasapi, I. N., Iliopoulos, I., Iconomidou, V. A., Trougakos, I. P., et al. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.06.025
“Pretrained language models such as BioBERT [16], ClinicalBERT [17], and SciBERT [18] have been developed to capture the nuances of biomedical and clinical language, improving performance on tasks like NER, relation extraction, and question answering.”
Big data and machine learning-based future chemical management strategy
Liu, W., Wang, H., He, J., Bai, X., Bi, G., & Chen, J. (2025) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/tb-2025-0221
Sun, M. (2025) · Arts Culture and Language · doi:10.61173/1fghre60
DianGPT: Bridging Precision and Efficiency for a Domain-Specific Question-Answering System
Chen, X., Jiang, Y., Pei, H., & Hei, X. (2025) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/hpsc66065.2025.00017 · Cited by 1
“Key limitations include insufficient domain knowledge integration, generic responses lacking contextual relevance, and limited customization options [1].”
Deep Learning Methods for Biomedical Named Entity Recognition: A Comprehensive Review
Sabitha, S., & Pillai, A. S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-78937-3_11
Choose Your Words Wisely: Domain-Adaptive Masking Makes Language Models Learn Faster
Kohli, V. S., Monis, A., & Mamidi, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95841-0_40
Bi-Encoder-Based Approach to Biomedical Document-Level Entity Recognition and Relation Extraction
Nie, P., Lin, M., Ning, J., Yang, Z., & Wang, L. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3582790
“In this study, BioECR utilises PubMedBERT as the base model, which outperforms BioBERT [39], SciBERT [34], and BlueBERT [40] in biomedical text understanding owing to its domain-specific design and optimised pre-training strategies [16].”
CSFJRE: Convolutional Shared Feature Network for Joint Entity and Relation Extraction
Hao, Z., Zhao, J., Lu, K., & Ding, L. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6954-7_16
LLDRE: A DocumentLevel Relationship Extraction Model Based on Enhanced Content Dependency
Liu, M., Li, X., Zhang, P., & He, X. (2024) · doi:10.1145/3703935.3704099
“The overall modeling cannot avoid the noise interference of irrelevant information on key [2] 65.1 82.5 LSR[7] 64.8 82.2 DHG[19] 65.9 83.1 GLRE[13] 68.5 ∗ ATLOP[21] 69.4 83.9 CGM2IR[18] 73.8 84.7 DGIN[15] 70.37 84.52 SKAMRR[20] 74 (2) Compared with Transformer based ATLOP and other models, this…”
InfluentialIRIS: Rapid Curation Framework for Iterative Improvement of Noisy Named Entity Annotations
Yano, K., Miwa, M., & Ananiadou, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_6
Cross-Domain Named Entity Recognition: A Resource-Efficient Transfer Learning Approach
Balducci, G., Fersini, E., & Messina, E. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_10
Gowhar, S., Kempaiah, P., Kamath, S. S., & Sugumaran, V. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96473-2_8
Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study
Kim, K. W., Islamaj, R., Kim, J.-D., Boudin, F., & Aizawa, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_13
“It has been widely used in various model evaluations, such as BioBERT and SciBERT [14,3], which allows us to evaluate the significance and practical applicability of our proposed method.”
Improving Summarization Quality with Citation Networks in Scientific Documents
Nguyen, Q.-A., Doan, T.-T., Ha, Q.-T., & Tran, M.-V. (2024) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse63888.2024.11063598
“Besides the OpenAI Ada method described in Section III-A , we experimented with two additional embedding methods: TFIDF and SciBERT [28].”
How can machine learning inform about chemical risks in circular textiles?
Bour, A., Melnyk, K., Hunka, A. D., Vanacore, E., Palmqvist, A., Bui, T., & Syberg, K. (2025) · Integrated Environmental Assessment and Management · doi:10.1093/inteam/vjaf088
Yuan, W., Meng, F., Gou, Y., Wang, Y., & Xu, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_6
Zhiyu, T., Shuang, L., Hanxi, L., & Yajing, L. (2025) · International Conference on Scientometrics and Informetrics (ISSI) · doi:10.51408/issi2025_090
Kashgouli, S., Malek, A., Masouleh, S. S. M., Naseri, S., Esteban, K., Black, R., Freund, M. S., Dasog, M., & Fatih, K. (2025) · ChemRxiv · doi:10.26434/chemrxiv-2025-4pk4w
Prompting encoder models for zero-shot classification: a cross-domain study in Italian
Auriemma, S., Miliani, M., Madeddu, M., Bondielli, A., Passaro, L., & Lenci, A. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09853-0
“Pre-trained LMs have proven to be effective in NLP tasks related to specific domains, whether they were trained from scratch [9, 10], or further pre-trained on domain data [6, 11, 12] with a Masked Language Modeling (MLM).”
MicrobioRel: A Set of Datasets for Microbiome Relation ExtractionPreprint
Khettari, O. E., Batteux, D., Quiniou, S., & Chaffron, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.03.666357
Advanced NLP Techniques for Enhancing ChatGPT Performance
Roy, B., Roy, P., Ghosh, S., Paul, T., & Paul, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-9841-8.ch001
Fartit, H., El-allaly, E.-D., & Bekri, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-91334-1_36
Dong, F., Zhang, H., Ren, Y., Li, W., Li, Y., Cui, G., & Liu, P. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70437
“adaptability in scientific research 16 and water management, 17,18 highlighting LLMs' wide application prospects.”
Tapajós, G., de Melo, T., Guedes, E. B., & da Siva, F. S. (2025) · Revista Eletrônica de Iniciação Científica em Computação · doi:10.5753/reic.2025.6077
Patenting and Information DisclosurePreprint
Wang, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5387619
Liu, Q., Wang, Y., Mu, L., & Li, J. (2025) · Discover Computing · doi:10.1007/s10791-025-09675-2
“Furthermore, some application field names have been changed, such as application field 5, 6, 15, and 19, even the LLM’s temperature is set to 0.”
Leveraging Pre-Trained Models for Abstractive Summarization of Computer Science Research Papers
Pepita, P., Hendra, J. S., Wulandhari, L. A., Nabiilah, G. Z., & Novika, A. D. (2025) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict65714.2025.11100819
“Research in pretrained model for scientific paper also completed by Beltagy et al. (2019), where they used BERT-Base and SciBERT models on scientific papers on biology, computer science, and multi field.”
Modification of the method for modeling the thematic environment of terms using the LDA approach
Золотарев, О., & А., Ю. В. (2025) · Computational nanotechnology · doi:10.33693/2313-223x-2025-12-2-19-27
Static and Dynamic Contextual Embedding for AutoML in Text Classification Tasks
Safikhani, P., & Broneske, D. (2025) · doi:10.1109/icnlp65360.2025.11108687
“These include BioBERT [15], [16] and Taiyi [17] for biomedical NLP, LegalPro-BERT [19] for legal NLP, FinBERT [20] in the financial domain, and GermaBERT [21] and Camem-BERT [22] for language-specific NLP.”
A Review of Automatic Text Summarization Technology Research
Liu, Y., Xiao, X., Zhang, J., & Zhou, S. (2025) · doi:10.1109/icnlp65360.2025.11108451
“Beltagy et al. put forth SciBERT, a BERT-based pre-trained language model, as a means of addressing the dearth of high-quality, massively labeled scientific data [16] .”
Accelerating Insight Discovery in Large Biomedical Text with Scalable Processing FrameworkPreprint
Kim, D., Hauptman, M., & Patrick, M. T. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.14.670384
Kverne, C., Monteverdi, F., Polyzou, A., Lisetti, C., & Bhimani, J. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--56157
Research on an Improved Box Plot Key Point Detection Algorithm Based on YOLOv8
Wu, Q., Yang, Y., Luo, J., & Yan, Y. (2024) · International Academic Exchange Conference on Science and Technology Innovation (IAECST) · doi:10.1109/iaecst64597.2024.11117897
“Modern data mining techniques, such as SciBERT[1] and BioBERT[2], have improved the efficiency of literature search through text mining, but cannot handle graphical data.”
Differential diagnosis making with large language models and probabilistic logic program
Galitsky, B. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-30046-2.00010-7
Topic-dependent relation prediction in argument mining: ternary classification
Schmidt, F. M., Gottifredi, S., & García, A. J. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02573-5
Analyzing we-sentences in Scientific Writing to Predict Peer Review Outcomes
López-Reyes, G. A., Ortega-Mendoza, R. M., Pérez-Cortes, O., Calderón-Suárez, R., & Castro-Espinoza, F. A. (2025) · IEEE Access · doi:10.1109/access.2025.3606415
Improving BioBERT Performance in Multi-Level Enzyme Classification
Henry, M. M., Kenneth, C., & Pardamean, B. (2025) · International Seminar on Intelligent Technology and Its Applications · doi:10.1109/isitia66279.2025.11137503
“Its general understanding and semantic alignment with GO biomedical descriptors justifies the selection over SciBERT [14] and PubMedBERT [15], which are pretrained only on a specific scientific corpus.”
Wu, R., Matsuno, H., & Ge, Q. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.02.673688
“Domain-specific language models such as BioBERT 31 and SciBERT 32 , trained on biomedical and scientific corpora, can extract and integrate knowledge dispersed across literature and databases.”
InfluentialForestry Text Classification Based on DeepSeek in Forestry Domain Hint Engineering
Guo, Z. (2025) · Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS) · doi:10.1109/acctcs66275.2025.00020
“It was launched by Google in 2018 and the code is open source[10].”
Lee, J., & Kim, S. (2025) · IEEE Access · doi:10.1109/access.2025.3607395
DNGR: Deep Neural Graph-Based Recommendation System for Scholarly Paper Retrieval
Sathya, M., Ramu, A., Velkumar, K., & Bhavani, M. (2025) · International Journal of Basic and Applied Sciences · doi:10.14419/dzzstd42
Xia, W., Zhong, Y., Gong, L., Yang, Y., Zhao, S., & Cui, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_25
LegSegSC: A Silver-Standard Rhetorical Role Labeled Dataset of Indian Supreme Court Judgments
Sharma, S., Singh, P. P., & Verma, A. (2025) · SN Computer Science · doi:10.1007/s42979-025-04359-6
“To evaluate the effectiveness of different transformer-based architectures, we fine-tuned multiple pre-trained language models, including DistillBERT [36], LegalBERT [37], Legal RoBERTa [9], ERNIE 2.0 [38], SCI-BERT [39].”
InfluentialMultidisciplinary End-to-End Document-Level Relation Extraction from Scientific Literature
Delaunay, J., Tran, H. T. H., González-Gallardo, C.-E., Bordea, G., Sidère, N., Doucet, A., & de Viron, O. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04627-7_15
A fragment-aware model for novel psychoactive substances analysis with uncertainty quantification
Liu, P., Guo, J., Xie, J., Li, L., & Tao, J. (2025) · Journal of Pharmaceutical and Biomedical Analysis · doi:10.1016/j.jpba.2025.117132
The word and the way: strategies for domain-specific BERT pre-training in German medical NLP
He, H. Y., Frei, J., & Scheible, R. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03728-2 · Cited by 3
“A similar method is employed by SciBERT [12], which retains the original BERT configuration but substitutes the initial general corpora with 1.14 million scientific articles randomly chosen from Semantic Scholar.”
InfluentialMNM: Multi-level Neuroimaging Meta-analysis with Hyperbolic Brain-Text Representations
Baek, S., Lee, J., Sim, J., Jeong, M., & Kim, W. H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04927-8_38
“Text2Brain [15] addresses this by using SciB-ERT [2] embeddings and convolutional neural networks to generate 3D activation maps, though it faces challenges with longer texts.”
Exploring the Learning Styles of "Semi-Humans": A Comparative Study of Nine Large Language Models
Yang, Y., Wang, T., Shen, D., Yao, F., & Huang, S. (2025) · International Conference on Computer Technology Applications (ICCTA) · doi:10.1109/iccta65425.2025.11166219
“Beltagy et al. notably observed that LLMs not only enhance textual processing accuracy but also significantly augment deep linguistic comprehension and knowledge extraction capabilities [17].”
Multi-View Multi-Label Online Learning for Education Topic Recommendation
Zhu, C., Li, H., Sun, Y., Wang, J., Pan, Y., & Han, L. (2025) · doi:10.1109/icaie64856.2025.11158002
“2) Generation of sentence-level semantic view: We use a scientific documents-based bidirectional encoder representations from transformers (SciBERT) [8] to generate the embeddings of the articles and in SciBERT, we use the sentences of the article as input to generate the vector representation of…”
العبدلي, ح. ع. ع., & أخواجه, ع. م. (2025) · مجلة دراسات الخليج والجزيرة العربية. · doi:10.34120/jgaps.v51i198.3475
RAG-BioQA: A Retrieval-Augmented Generation Framework for Long-Form Biomedical Question Answering
Panchumarthi, L. Y., Saleti, S., Gudari, S. P., Negi, A., Upadhya, H. K., & Budime, P. R. (2026) · International Conference on Emerging Trends in Advancements and Applications of Computational Intelligence Techniques (ETAACT) · doi:10.1109/etaact69135.2026.11541950
“On the generative side, BioGPT [28] achieved 81% accuracy on PubMedQA with 1.5 billion parameters, while BioMedLM [29] scaled to 2.7 billion parameters with a domain-specific tokenizer, reaching 57.3% on MedQA.”
InfluentialResearch Direction and Science Evaluation: The Role of Coherence and Alignment
Ayoubi, C., Barbosu, S., Pezzoni, M., & Visentin, F. (2025) · Journal of Economics & Management Strategy · doi:10.1111/jems.70009 · Cited by 1
Zhang, Y., Zhai, H., Zhang, Q., Zhai, Q., Bian, R., Liu, B., Bu, X., & He, X. (2025) · IEEE Access · doi:10.1109/access.2025.3618907
Author Name Disambiguation Based on Multi Relationship Fusion and Representation Learning
Cui, H., Huang, Q., Yang, J., Liu, R., Hu, K., & Dong, G. (2025) · International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) · doi:10.1109/caibda65784.2025.11182797
“The OAG-BERT [20] and SciBERT [21] models are selected as baseline models for comparison.”
Explain Before Classify: Contrastive Rationale Distillation for Academic Opinion Recognition
Zhang, M., Zhang, Z., Wang, Y., Li, Y., Lin, X., & Wang, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3453-1_21 · Cited by 1
Sparse Keyword Data Analysis Using Bayesian Pattern Mining
Jun, S. (2025) · Computers · doi:10.3390/computers14100436
“In scientific text analysis, Beltagy et al. (2019) developed pretrained language models such as SciBERT, which showed strong performance not only in classification tasks but also in entity and relation extraction, closely connected to rule mining and keyword association [40].”
Detecting Expert-Written Comments in Stack Exchange
Musku, H., Ritchie, A., Jedidi, N., Leekha, R., & VanDam, C. (2025) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec67600.2025.11196502
Meudec, R., Dockès, J., Ghayem, F., Wassermann, D., & Thirion, B. (2026) · Imaging Neuroscience · doi:10.1162/imag.a.1367 · Cited by 1
“Text-2Brain ( Ngo et al., 2021) leveraged language models ( Beltagy et al., 2019) to capture semantic relationships between terms, enabling the encoding of any query.”
Preprint Archives as Active Scholarly Communities: An Augmentation FrameworkPreprint
Lissack, M. R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5626649
A Foundation Model pre-trained on Serialized EHR RepresentationsPreprint
Qin, Q., Chou, Z., & Li, S. (2025) · doi:10.36227/techrxiv.176127291.14956389/v1
DiffTMR: Diffusion-based Hierarchical Alignment for Text-Molecule Retrieval
Wang, C., Zhou, D. M., Liu, T., Lin, J., Zhou, Y., & Yang, A. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755668
“Implementation Details : We employed the pretrained SciB-ERT [1] model as the text encoder (maximum sequence length of 256) and a three-layer GCN [37] as the molecular graph encoder (output dimension of 300).”
InfluentialAprendizado Profundo para Detecção de Movimentos Retóricos
de Jesus, B. V. V., & Cândido, A. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37824
Jamil, H. M. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3773278
“In parallel, domain-specific models like SciBERT [4] demonstrate how pre-trained language models tailored to specialized domains can enhance KG construction and extension, which is especially relevant for fields such as biomedical informatics.”
Taslim, T., Handayani, S., Walhidayat, W., & Toresa, D. (2025) · Digital Zone Jurnal Teknologi Informasi dan Komunikasi · doi:10.31849/digitalzone.v16i2.28375
Gupta, K. K., Dey, S., & Mukhopadhyay, T. (2025) · Materials horizons · doi:10.1007/978-981-95-2660-4_10
Software Defects Prediction: Source Code Is All You Need
Ba, T. A., Touré, F., & Faghihi, U. (2025) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c65679.2025.00029
“To fine-tune the DeepSeek-Coder model, we used LLaMA [55] based architecture for sequence classification 4 [59].”
Measuring Semantic Coherence of RAG-Generated Abstracts Through Complex Network Metrics
Gana, B., Palma, W., Lucay, F. A., Missana, C., Abarza, C., & Allende‐Cid, H. (2025) · Mathematics · doi:10.3390/math13213472
Structuring and analyzing scientific texts based on academic genealogy
Marinosyan, A. K., & Anikieva, M. A. (2025) · International Conference on Engineering Management of Communication and Technology (EMCTECH) · doi:10.1109/emctech65814.2025.11220572
Exploring Generative AI Models for Enhancing Text Summarization
Khan, R., Sharma, S., & Upadhyay, D. (2025) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic66080.2025.11212151
“Within academia, LED_Large[31], Pegasus[25] and SciBERT[32], and similar models have been developed for the purpose of summarizing some highly complex domains which further indicates the necessity of tailored AI model application for the same.”
Zhang, M., Wang, Y., Wang, Y., & Zhang, Z. (2025) · IEEE International Conference on Big Data Analytics (ICBDA) · doi:10.1109/icbda65366.2025.11210932 · Cited by 1
“The models included in our comparison are BERT [9], SciBERT [24], DeBERTa [11] and AgriBERT.”
Pattern Prediction: A Comprehensive Review of Evaluation Methods
Berquedich, A., Ajallouda, L., Zellou, A., & Chihi, I. (2025) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v19i21.56747
Batch Aggregation: An Approach to Enhance Text Classification with Correlated Augmented DataPreprint
Hui, C., & Wen, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.05020
“…areas, the common approach is to acquire a substantial amount of annotated data in the desirable domain (Alsentzer et al., 2019; Paul et al., 2023; Beltagy, Lo, and Cohan, 2019) or in general improve existing model architectures and train it from scratch (Liu et al., 2019; Lan et al., 2019; Clark…”
Sadouki, K., & Kornyshova, E. (2025) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat67162.2025.00061
“Three domain-adapted transformer encoders, I4BERT-Base [21], SciBERT, and RoBERTa-Large [22], were evaluated, each coupled with a span-classification head for simultaneous entity and relation extraction.”
Enhanced Biomedical Relation Extraction with Semantic and Descriptive Knowledge
Liu, Y., & He, L. J. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229399
“We compare SDK-BRE with the following baseline meth-ods: Most baseline methods [3], [5]–[7], [16], [18], [30] are classification methods based on distributed representations, which involve performing | Y | + 1-way classification on top of biomedical pre-trained language models; QARE [7] and NBR…”
Desenvolvimento de um Índice de Infração de Patentes com Apoio de Inteligência Artificial Multimodal
Morais, A. M. D., & Dias, C. G. (2025) · Navus - Revista de Gestão e Tecnologia · doi:10.22279/navus.v16.2133 · Cited by 1
“…corpus de artigos acadêmicos nas áreas biomédica e de ciência da computação, o SciBERT mostra-se mais adequado do que o BERT padrão para tarefas de PLN envolvendo literatura científica — como classificação de sentenças, extração de entidades e análise semântica de textos técnicos (Beltagy, 2019).”
Ataei, S., & Butler, G. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.13.688294
A hidden network of gene relationships unifies behavioral and molecular researchPreprint
Connell, D. R., Gaiteri, C., & Tasaki, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.14.688547
“…scientific models, we added four more models based on BERT’s architecture: Allen Institute for AI’s SciBERT uncased ( scibert_scivocab_uncased ) [10], Specter [17], and newer Specter2 [8], and PubMedNCL [18] which is a fine-tune of Microsoft’s BiomedBert—a model similar to SciBERT but…”
A Human-Centered AI Agent Framework with Large Language Models for Academic Research Tasks
Chen, Z., Chen, J., Wang, Y. G., & Shen, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3343-5_28 · Cited by 1
Domain Knowledge Empowered Language Models: A SurveyPreprint
Palikhe, A., Wang, Z., Hu, S., Yin, Z., Zhao, X., Liu, J., Cai, Y., & Zhang, W. (2025) · doi:10.36227/techrxiv.176403151.11257350/v1 · Cited by 1
Transformer Based Argumentation Mining in English Language: BERT for Encoding and T5 for Decoding
Singh, R. B., Sinha, S., & Singh, A. (2025) · Journal of Emerging Technologies and Innovative Research · doi:10.56975/jetir.v12i11.572270
Dataset Similarity Estimation Using LLM-Based Metadata Embeddings
Ito, K., & Matsubara, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4861-3_11
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.