SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 3,401–3,500 of 3,596
Miao, Y., Luo, S., & Zhan, K. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3432-5_31
Cheng, H., Liu, L., Cui, T., & Pan, H. (2026) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2026.103096
Causal Backdoor Adjustment for Citation Intent Classification
Song, Y., Wan, L., Zhang, Z., Wang, D., & Zhuang, F. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2868-3_9
Soltani, A., Bala, A., Nkashama, D. K., Tardif, P.-M., Bahnasse, A., Frappier, M., & Kabanza, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-20732-6_1
Vitale, J. D. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8070207
“[30], LEGAL-BERT [31], SciBERT [32], AgriBERT [33], BERTOverflow [34], Gururangan RoBERTa [29].”
Tleubayeva, A., Meiramkhanov, T., Şeker, Ş. E., & Boluk, P. S. (2026) · IEEE International Conference on Smart Information Systems and Technologies (SIST) · doi:10.1109/sist61674.2026.11596111
“Models from the SentenceTransformers family, including all-mpnet-base-v2 , have demonstrated strong performance in semantic search and clustering and are commonly used as high-quality general-purpose baselines [13].”
SpaceBio Knowledge Hub: A LiteratOmics Platform for Microgravity and Space Biology ResearchPreprint
Silva, J. C. F., Vieira, A., Donahey, M. S. C., Silva, S. M. D. C., Veloso, T., Lopes, A., Sexson, N., Barker, R., Porterfield, D. M., Silva, C. A., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.13.737239
From Data Science to Artificial Intelligence
Fogel, G. B. (2026) · Springer eBooks · doi:10.1007/978-3-032-11023-7_1
Integrating Heterogeneous Knowledge for Enhanced Recommendation with Large Language Models
Petruzzelli, A., Musto, C., de Gemmis, M., Lops, P., & Semeraro, G. M. (2026) · ACM Transactions on Information Systems · doi:10.1145/3831695
“All the aforementioned evaluation metrics are computed utilizing the clayrs framework [38].”
Hao, Z., Duan, Y., & Wang, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/91215
“[35] (73.1%), BioLinkBERT-Large [36] (74.1%), BioBERT [37] (73.1%), and AlpaPICO [38] (64.8%).”
InfluentialThe rise of methods-driven science: How Methods Papers Reshaped the Research Landscape
Krauß, A., Rosenfeld, A., & Bornmann, L. (2026) · EMBO Reports · doi:10.1038/s44319-026-00873-7 · Cited by 1
“We evaluated this model against modern alternatives, including SciBERT (Beltagy et al, 2019) and large language models such as GPT-4.”
Context-Aware Dual-Stream Cross-Attention for Sentence Classification in Biomedical Abstracts
Rostam, Z. R. K., & Kertész, G. (2026) · IEEE International Symposium on Intelligent Systems and Informatics (SISY) · doi:10.1109/sisy71351.2026.11738425
“…conditional random fields, while more recent methods have leveraged pre-trained language models (PLMs) such as BERT [8] and its variants (SciBERT [9], BioBERT [10], PubMedBERT [11], and Bi-oLinkBERT [12]), which have shown strong results across a range of biomedical text analysis tasks [13],…”
Aydoğan, M., Yıldırım, S., & Yıldız, T. (2026) · IEEE Access · doi:10.1109/access.2026.3715915
Pilli, R. M. R., Koch, J., & Wright, C. (2026) · IEEE International Conference on AI and Data Analytics (ICAD) · doi:10.1109/icad69378.2026.11609131
“Domain-Adaptive Pretraining : Domain-specific BERT variants (BioBERT [10], Legal-BERT [11], SciBERT [12]) established that continued pretraining improves downstream performance when domain vocabulary differs from general text.”
Li, H., Kong, Y., & Wang, M. (2026) · International Conference on Machine Learning and Intelligent Systems Engineering (MLISE) · doi:10.1109/mlise70044.2026.11607601
“Using Qwen3-8B as the fixed augmentation source, we further compare four encoders under sequence constraints—BERT-base [36], RoBERTa-base [37], SciBERT [38], and DeBERTa-v3-base [39]—to examine the moderating role of encoder choice.”
Kaur, R., Ginige, J. A., & Obst, O. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03631-w
“ClinicalBERT, pre-trained on clinical notes, and SciBERT [21], pre-trained on scientific texts, are also tailored for medical text summarisation.”
Clinical Text De-Identification Beyond PHI Detection: A Scoping Review
Sariyar, M. (2026) · J — Multidisciplinary Scientific Journal · doi:10.3390/j9030022
Wang, W., Lu, M., & Huang, X. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02848-5
“Therefore, we use SciBERT for paper content representation [54], which shows excellent performance in semantic representation tasks for scientific literature [55].”
Semantic Embedding and Profile-Based Ranking for Automated Reviewer Recommendation
Bintang, A. L., & Nurhaida, I. (2026) · SinkrOn · doi:10.33395/sinkron.v10i3.16208
TCellAlign: Cross-study T-cell Populations Alignment with Nomenclature-Guided Multi-Agent WorkflowPreprint
Xie, P., Zhou, R., Ou, Z., Zhang, J., Zhou, X., Sun, X., Lu, J., & Ma, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.24093
“First, biomedical text-mining models such as BioBERT [24], SciBERT [4], and PubMedBERT [14] support entity recognition, relation extraction, and other structured knowledge extraction tasks from scientific literature.”
CPC-aware neural patent retrieval with LLM-assisted query expansion
Elmi, Z., Aydın, N., Engin, S., Besli, R., & Elmi, S. (2026) · Journal of Data Information and Management · doi:10.1007/s42488-026-00165-8
“…scientific-domain, sentence-level, and cross-lingual representations for downstream classifica - tion and retrieval tasks (Devlin et al. 2019; Beltagy et al. 2019; Reimers and Gurevych 2019; Conneau et al. 2020; Jiang et al. 2020; Karthikeyan et al. 2020 ). word similarity, subword-level…”
Sebe, C., Ferret, O., Névéol, A., Esmailoghli, M., Leser, U., & Cohen‐Boulakia, S. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag565
“Encoder-based approaches were evaluated using Transformer-based encoders [e.g. SciBert (Beltagy et al. 2019), ModernBERT (Warner et al. 2025)] coupled with a BiLSTM-CRF architecture (Huang et al. 2015) (see Fig.”
Information extraction from literature for ORR catalyst in fuel cell
Htet, H., Hirano, M., Ali, A. A., Sasaki, Y., & Asahi, R. (2026) · Computational Materials Science · doi:10.1016/j.commatsci.2026.114958
Tan, F., Liu, G., & Zhao, R. (2026) · IEEE Access · doi:10.1109/access.2026.3718020
Matos, N., & Júnior, M. C. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32023-0_19
Rehman, T., Sanyal, D. K., & Chattopadhyay, S. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-19215-8_15
Guan, C., Ren, J., & Lim, T. (2026) · Analytics · doi:10.3390/analytics5030027
Shi, D., Chen, Z., Song, K., & Han, Y. (2026) · Chinese Journal of Chemical Engineering · doi:10.1016/j.cjche.2026.07.012
Anzà, B., Garofalo, S. F., Massarelli, E., Massai, D., & Fino, D. (2026) · Food Reviews International · doi:10.1080/87559129.2026.2711120
Ayadi, M. G. (2026) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-026-00843-y
“Methodological Evolution: The field has progressed from simple term-frequency methods (TF-IDF) through graph-based statistical methods (TextRank, RAKE) and super - vised sequence labeling (CRF, LSTM-CRF, BiLSTM-CRF) to transformer-based methods, which encompass both fine-tuned sequence labeling (BioBERT, ClinicalBERT,…”
InfluentialXunZi, an AI biologist, reveals disease-modifying targets
Huang, X., Qin, J., Tang, F., Yang, C., Xu, L., Wei, J., Liu, D., Chen, K., Zhang, C., Chen, M., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01769-6
Jin, S., Song, G., Xiang, H., Fu, Z., Zhang, X., Pei, J., Zhang, T., Fan, W., Wang, Q., & Zhang, D. (2026) · Journal of Energy Chemistry · doi:10.1016/j.jechem.2026.07.066
CiteFuncRanker: an LLM-based pairwise ranking framework for multi-functional citation analysis
Wang, Y., Ruan, X., Lyu, D., & Cheng, Y. (2026) · Scientometrics · doi:10.1007/s11192-026-05764-1
“Table 10 summarizes representative supervised classification results on ACL-ARC and SciCite, including BiLSTM-based models (Cohan et al., 2019) and SciBERT-based models (Lahiri et al., 2023).”
PreSL: Document Level Pre-training of Scientific Literature via Self-supervised Learning
Wang, L., Qiao, Z., Xiao, M., Chen, M., Du, Y., Cui, W., & Zhou, Y. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8447-5_2
Rethinking Software Ethics in the Age of Artificial Intelligence
Nigsch, K. (2026) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-032-31732-2_7
Qian, J., Liang, X., Que, L., Pang, Z., Chen, S., & Zhao, F. (2026) · IEEE Access · doi:10.1109/access.2026.3721388
INDUS-SDE: A Language Model for Scientific Content Curation and Discovery
Pantha, N., Awale, S., Kuruvanthodi, V., Simran, K. C., Ramasubramanian, M., Davis, C., Praveen, B., Foshee, E., Bhattacharjee, B., Bugbee, K., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818847
Mitigating data imbalance in scholarly networks using topic models and social network analysis
Noor, M. A., Sheppard, J. W., & Clark, J. A. (2026) · Social Network Analysis and Mining · doi:10.1007/s13278-026-01630-3
“We use SciBERT as the embedding model and fine-tune it on the publication corpus via masked language modeling to adapt the embeddings to domain-specific terminology.”
InfluentialDing, R. F., Ding, Z., Ferreira, R. P., Chen, Y., & Chen, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3819013
RECO-Rank: A Compensatory Multi-Evidence Ranking Model for Cross-Domain Reviewer RecommendationPreprint
Chen, J., Xu, J., & Fang, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9558981/v1
Kim, M.-S., Rhee, J.-W., Lee, S.-H., Park, S.-H., & Sung, T.-E. (2026) · Scientometrics · doi:10.1007/s11192-026-05734-7
Compass: Navigating Global Marine Lead Data Integration through Expert-Guided LLM Agent
Liu, Y., Lü, B., Jin, M., Sang, Z., Jiang, S., Zhou, L., Wang, X., Zhou, C., & Zhang, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3819001
“ChemAgent [35] and SciAgents [14] leverage tool-use and ontological knowledge graphs for multi-agent reasoning, while other approaches explore self-updating mechanisms and automated environment generation [19, 30].”
Patil, S. J. S. A. V. (2026) · Dandao Xuebao/Journal of Ballistics · doi:10.52783/dxjb.v38.516
Yan, F., Zheng, D., Yu, F., & Kang, J. (2026) · Applied Sciences · doi:10.3390/app16167956
A Transformer-Based Approach to Automated Drug Information and Interaction Prediction
Chitteti, C., Kumar, K. V. S., Dharani, M., Suchithra, M., Kavya, K., & Dhanaraj, E. (2026) · OPJU International Technology Conference on Smart Computing for Innovation and Advancement in Industry (OTCON) · doi:10.1109/otcon69332.2026.11629633
“Transformer models like BioELMo [11], SciBERT [12], and domain-specific fine-tuning approaches [13] have shown promise in medical NLP.”
Data and Knowledge Dual-Driven Text Embedding Approaches: A Survey
Chen, X., 詹佳青, Lin, J., Liu, H., Zhang, Q., Chen, J., & Chen, H. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3721883
“BioBERT[80]andSciBERT[81]arepre-trainedonbiomedicalandscientificcorpora,respectively.”
Parameter-Efficient Multi-task Learning for Biomedical Abbreviation Disambiguation
Manikandan, V., Wen, D., & Dewan, M. A. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35387-0_12
TARL: Multi-task adaptive representation learning for secure encrypted traffic analysis
Ge, M., Liu, L., Zhang, Z., Wang, H., Yu, X., Xu, Y., Feng, R., & Hu, Z. (2026) · Computer Networks · doi:10.1016/j.comnet.2026.112646
Han, S., & Kim, J. (2026) · Geoscience Data Journal · doi:10.1002/gdj3.70095
“SciBERT, trained on more than 1.1 million scientific papers and abstracts, has been shown to improve representation of domain-specific terminology and contextual meaning in scientific text (Beltagy et al. 2019).”
InfluentialA hierarchical attributed graph rag framework for biomedical literature retrieval
Rani, M., Mishra, B. K., Thakker, D., & Khan, J. A. (2026) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae9af2
LLM-Assisted Reviewer Assignment via Auditable Expertise Matching
Bagheri, F., Buscaldi, D., & Recupero, D. R. (2026) · IEEE Access · doi:10.1109/access.2026.3724622
Wei, Y., Guo, F., Zhou, Q., Fu, Z., Ding, M., Yang, K., & Zhang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.15624
“Early approaches relied on lexical matching [23], while scientific-domain encoders adapted pretrained language models to scientific corpora [4].”
Adebusuyi, O., Ibekwe, J. P., Otubu, R., Ajadi, A., Aderinto, T., Sola-Alao, O. E., Babalola, A. V., Obagaye, J., & Lukman, B. O. (2026) · doi:10.2196/preprints.109367
Kong, D., Funabiki, N., Kyaw, H. H. S., Kotama, I. N. D., & Zhu, Z. (2026) · Future Internet · doi:10.3390/fi18080442
“Other studies further showed that pre-training on scientific text can improve performance on scientific natural-language processing tasks [32].”
Joshi, N. (2026) · Research Square · doi:10.21203/rs.3.rs-10330873/v1
Semantic Axis-Guided Diffusion for Data Augmentation in Academic Text Analysis
Li, H., Wang, M., & Tan, Z. (2026) · IEEE International Conference on Big Data and Artificial Intelligence (BDAI) · doi:10.1109/bdai70753.2026.11655153
“…baseline, isotropic diffusion augmentation, and the proposed semantic axis diffusion augmentation — using five pre-trained encoders as backbone models, namely BERT [23], RoBERTa [24], SciBERT [25], DeBERTa [26], and Qwen-Embedding [27], under the 8-shot low-resource experimental setting.”
Chen, Y., & Popescu, M. (2026) · medRxiv · doi:10.64898/2026.08.20.26360740
Service-Based RAN User Plane Decoupling and Orchestration via ComBERT for AI AgentServices
Ding, H., Shangyuan, D., Sun, X., Guo, X., Yuan, C., Tian, L., Zhang, S., & Jin, J. (2026) · Sensors · doi:10.3390/s26175318
“Recently, domain-adaptive pre-training, as represented by SciBERT and related approaches, has demonstrated significant advantages in understanding specialized corpora [20,21].”
LLM assisted writing deserves empirical evaluationPreprint
Feng, X. Z., Lin, Y., Zhang, Y., Weng, C., & Peng, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.22124
“We grouped papers as pre-LLM or post-LLM, using the public release of ChatGPT in late 2022 as a practical boundary [4], and classified post-LLM papers into LLM-independent and LLM-assisted using a SciBERT classifier [5].”
Intelligent Researcher Recommendation Based on Hybrid GraphRAG and Semantic Knowledge Graph Fusion
Kong, X., Liu, Y., 王体朋, Zhao, K., Yu, X., Yu, H., Yanfang, M., & Zhao, Z. (2026) · International Conference on Electronics and Communication, Network and Computer Technology (ECNCT) · doi:10.1109/ecnct70535.2026.11661455
“SciBERT, pretrained on large-scale scientific corpora, has been shown to capture scientific terminology and academic expressions more effectively than general text models in scientific natural language processing tasks [11] .”
Structure-Aware GPU Scheduling with Cycle Fusion and Feedback Adaptation for Multi-Task Training
Guo, L., Xu, G., Zhang, T., Ding, M., Yang, L., & Li, Y. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661263
“Notably, the backward pass generally imposes a significantly higher computational burden than the forward pass [11], [12].”
Kong, X., Liu, Y., 王体朋, Zhang, J., Zhao, K., Yu, X., Yu, H., & Zhao, Z. (2026) · International Conference on Electronics and Communication, Network and Computer Technology (ECNCT) · doi:10.1109/ecnct70535.2026.11661478
TriboBERT: A pre-training language model tailored for tribology
He, W., & Lu, Z. (2026) · Tribology International · doi:10.1016/j.triboint.2026.112639
İlgen, B., Tolias, Y. S., Kühnert, D., Papadopoulou, P., Westerlund, M., Heider, D., Ladewig, K., & Hattab, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30912
“In literature mining and knowledge extraction , domain-specific language models such as BioBERT [13], SciBERT [14], PubMed-BERT [15], and BioGPT [16] automate the discovery of variant–disease links and biomarkers from millions of oncology publications, with systems such as LitVar [17], PubTator…”
Nooraldaim, A. S., Abushama, A. A., Ahmed, H., Mohamed, M., Salih, W. E. M., Ali, A. E. A. O., Mergany, N. N., Lai, X., Zhu, X., & Wang, J. (2026) · Scientific Reports · doi:10.1038/s41598-026-67923-x
Chen, H., Dong, H., Zhu, Y., Hong, Y., Niu, L., & Ye, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.29526
“For specialized domains, researchers further developed domain-specific language models such as SciBERT and BioBERT, achieving substantial improvements in named entity recognition and relation extraction tasks (Beltagy et al., 2019; Lee et al., 2020).”
Understanding before verifying: Claim normalization for automated citation verificationPreprint
He, Y., Wu, M., Deng, S., & Zhang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30145
Scaling Biomedical Text-Mining: Transformers, GenAI, and Drug Discovery
Agostinho, S. P., Santos, C., Ramalhão, D., Moreira, I. S., & Rosário‐Ferreira, N. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5408-8_4
Babaiha, N. S., Geißler, S., Simon, M.-C., Hofmann‐Apitius, M., & Jacobs, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30393
The Role of N-ary Relations in Representing Biomedical Complexity
Conceição, S. I. R., & Couto, F. M. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5408-8_15
SciRep: A Ranking-Aware Representation Model for Scientific Text
Fu, B., & Wang, J. (2026) · Applied Sciences · doi:10.3390/app16178722
Text-guided flow matching enables sample-efficient crystal structure generationPreprint
Li, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.01076
Intra-document Disambiguation of Math Identifiers for Formula Grounding
Asakura, T. (2024) · Institutional Repositories DataBase (IRDB) · doi:10.15083/0002013240
Yagahara, A., Yokohama, N., Uesugi, M., Aizawa, M., Ando, D., Ishikawa, T., Sudo, M., Nishimoto, N., Tanikawa, T., Aoki, Y., et al. (2026) · Radiological Physics and Technology · doi:10.1007/s12194-026-01133-y
“This interpretation is consistent with previous studies showing that the performance of domain-specific pretraining is affected not only by vocabulary design but also by corpus composition and the representations learned from in-domain text [22, 43, 44].”
Table-Based Text Parsing Methods for Polymer Science: A Comparative Study
howe, R., & Cole, J. M. (2026) · ACS Omega · doi:10.1021/acsomega.6c05295
Arts, S., Melluso, N., & Veugelers, R. (2025) · Lirias (KU Leuven)
Mikhaylov, D. V., & Emelyanov, G. M. (2026) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661826700045
Koç, C., & Özyurt, F. (2026) · Drug Metabolism and Disposition · doi:10.1016/j.dmd.2026.100404
Skrypnyk, P. (2026) · Rowan Digitals Works (Rowan University)
Hoz-M, J. D. L., Solano, H. L., Campuzano, M. J., Delahoz-Domínguez, E., & Acosta-Vega, R. K. (2026) · Oceans · doi:10.3390/oceans7050077
Eckel, G. (2023) · The Atrium (University of Guelph)
MythoBERT 1.0: A pre-trained language model for Hindu mythology
Paul, A., & Das, D. (2026) · Sādhanā · doi:10.1007/s12046-026-03290-7
“Similarly, SciBERT, introduced by [3], extends BERT for scientific literature, improving performance on tasks within the scientific domain.”
Identifying Studies on Culture Medium Optimization and Extracting Associated Composition Data
Yoda, S., Tsukagoshi, H., Sasano, R., Ukai, Y., & Sato, Y. (2026) · Journal of Natural Language Processing · doi:10.5715/jnlp.33.1681
Babac, M. B. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01092-1
“BERT established the masked-language-modelling paradigm for transferable bidirectional encoding (Devlin et al., 2019), while BioBERT and SciBERT showed that domain-specific pre-training improves specialised understanding in biomedical and scientific text (Lee et al., 2020; Beltagy et al., 2019).”
Adhikari, S. (2026) · arXiv.org · doi:10.48550/arxiv.2609.14572
“A specialized named-entity recognition model [15] trained on CS and mathematical text labels concepts such as merge sort or NP-hard and recognizes LaTeX-formatted notation.”
Latent Chain-of-Thought Reasoning with Adaptive Scratchpad for Citation Intent Classification
Song, Y., Li, Y., Zhang, Z., Wang, D., & Zhuang, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_32
Author Name Disambiguation via Integrating Semantic and Relational Information--Name Disambiguation
Yuan, H., Liu, S., & Yuan, C. (2026) · International Conference on Digital Society and Artificial Intelligence (DSAI) · doi:10.1145/3833541.3833560
“Compared to the traditional BERT model, the Sci-BERT model [15] is pre-trained on a large corpus of academic papers; consequently, its performance on academic paper classification tasks is significantly superior to that of the traditional BERT model.”
Instant local citation recommendation via non-sampling embedding of dynamic historical citations
Li, H., & Zhang, J. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.135125
Медианные значения критериев качества генерации обзоров при сравнении с альтернативными подходами
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.4
Кузнецов, И. (2026) · Международный научно-исследовательский журнал · doi:10.60797/irj.2026.171.79
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.1
Основные параметры конфигурации системы при проведении эксперимента
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.2
Медианные значения критериев качества генерации обзоров для рассматриваемых моделей и подходов
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.171.79.3
Kapoor, S., & Verma, D. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-40338-5.00001-9
XRAG-CJG: A Trust-Aware and Explainable Retrieval Framework for Reviewer Recommendation
Gunawardana, H., & Silva, T. (2026) · Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon71835.2026.11691368
LLM-Based Text Classification with SEP-Embedded Contextual Semantic Features
Li, H., Wang, M., Xie, J., Liu, Q., & Xing, C. (2026) · International Conference on Machine Learning Technologies (ICMLT) · doi:10.1109/icmlt69916.2026.11688314
“6-10B [24], Qwen3-8B [25], Llama 3.2-10B [26], and Mistral-8B [27] — are selected as baseline models to validate the classification performance differences between no semantic feature embedding and embedding different semantic encoders (BERT, SciBERT [28], RoBERTa, Qwen-embedding [29]).”
InfluentialJaya, V. K., Ferawati, K., & Nirwana, M. B. (2026) · International Conference on Smart Data, Intelligence, and Analytics (ICoSDIA) · doi:10.1109/icosdia70683.2026.11690464
“SciBERT is a pre-trained language model trained using the BERT architecture, which is an encoder only transformer, with the aim of addressing the problem of a lack of high quality, large scale labeled scientific data [17].”
Zhang, L., Wan, Z., Cheng, M., Shi, S., & Gou, Z. (2026) · Scientometrics · doi:10.1007/s11192-026-05813-9
“Other databases, such as Semantic Scholar and Scite, have also implemented similar functions, predominantly relying on supervised learning algorithms for citation classification (Beltagy et al., 2019; Cohan et al., 2019).”
A data-driven comparative analysis on citation intents across scientific disciplines
Phan, T.-A., Nam, S., & Jung, J. J. (2026) · PLoS ONE · doi:10.1371/journal.pone.0358327
“However, the existing literature on citation intents in bibliometrics has only predominantly centered on two key directions: i ) defining robust taxonomies and establishing benchmark datasets [1–6]; and ii ) developing novel methodologies to enhance automated classification performance [3,4,7–19].”
Yamauchi, K., & Katsurai, M. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846040
“Brack et al. [4] replaced the word embedding layer of HSLN with SciB-ERT [1], an encoder pretrained on scientific text, and evaluated the resulting model across four scientific domains.”
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.