SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 3,101–3,200 of 3,596
Montfrond, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.05179
“Domain-specific adaptations such as BioBERT, trained on biomedical corpora (Lee et al., 2020), and SciBERT, trained on scientific publications (Beltagy et al., 2019), demonstrate that fine-tuning BERT on specialized datasets yields significant improvements in accuracy.”
Maiti, A., Panth, P., & Satapathy, S. C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.10435
“Our specific contributions are as follows: SciBERT-Based Anomaly Detection: We employ a moving-window probability assessment system that uses allenai/scibert scivocab uncased [8].”
InfluentialGroenendyk, M., & Ivan, L. (2025) · Scientometrics · doi:10.1007/s11192-025-05508-7
“Recent developments in large language models have demonstrated the ability to identify patterns in academic texts that can detect dataset citations, even when inconsistent or incomplete (Beltagy et al., 2019).”
Zhou, Q., Yang, Q., Li, X., Zhou, F., & Zhao, J. (2025) · International Conference on Electronic Information Engineering and Computer Science (EIECS) · doi:10.1109/eiecs67708.2025.11283490
“Recent advancements in natural language processing (NLP) have spotlighted the efficacy of pre-training large-scale language models as a precursor to task-specific fine-tuning [19] .”
GTSP: A Unified Framework for Graph Pretraining and Task-Specific Prompting
Lu, F. (2025) · International Conference on Artificial Intelligence, Automation and High Performance Computing (AIAHPC) · doi:10.1109/aiahpc66801.2025.11290070
“Inspired by the idea of pre-training in the field of natural language processing (NLP) [18, 19], a bunch of pre-training paradigms have emerged in the graph domain [1, 20, 21].”
Seneviratne, S. (2024) · ANU Open Research (Australian National University) · doi:10.25911/arxy-hr63 · Cited by 2
Dablain, D. A., Huang, L., & Sepulvado, B. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6936879
A Comprehensive Attempt to Research Statement GenerationPreprint
Wu, W., & Li, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.14339
“To extract scientific terms, we use the NER model Scibert 6 (Beltagy et al., 2019), which is pretrained on scientific corpus and tuned on the NER dataset SciERC (Luan et al., 2018).”
MethodComputer vision and deep-learning tools for automatic data extraction from chemical reaction schemes
Wilary, D. M. (2024) · Apollo (University of Cambridge) · doi:10.17863/cam.111098
Öztürk, Y., Bordignon, F., Labbé, C., & Portet, F. (2024) · HAL · doi:10.5281/zenodo.14142700
Web Page Classification using Unsupervised and Semi-Supervised Clustering Technique
Pandey, H., Elßner, T., & Kersten, J. (2024) · elib (German Aerospace Center) · doi:10.5281/zenodo.13882804
Data-efficient methods for information extraction
Yaseen, U. (2023) · doi:10.5282/edoc.31511
Shi, M., Janowicz, K., Liu, Z., & Currier, K. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.cosit.2024.23
Saier, T. (2024) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000170262
Named Entity Recognition for Specific Domains - Take Advantage of Transfer Learning
Torge, S., Hahn, W., Manjunath, L., & Jäkel, R. (2021) · International Journal of Information Science and Technology · doi:10.57675/imist.prsm/ijist-v6i3.189
Psarrou, K. (2025) · DSpace - NTUA (National Technical University of Athens) · doi:10.26240/heal.ntua.29264
What, When, and Where Do You Mean? Detecting Spatio-Temporal Concept Drift in Scientific Texts
Shi, M., Janowicz, K., Liu, Z., Karimi, M., Majić, I., & Fortacz, A. (2025) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2025.16
Additional file 1 of Charting mobility patterns in the scientific knowledge landscape
Singh, C. K., Tupikina, L., Lécuyer, F., Starnini, M., & Santolini, M. (2024) · Figshare · doi:10.6084/m9.figshare.26683300
Interpreting patient case descriptions with biomedical language models
Alghanmi, I. (2023) · ORCA Online Research @Cardiff (Cardiff University)
Learning Label Relationships Using Deep Neural Networks for Hierarchical Text Classification
방진현 (2023) · Seoul National University Open Repository (Seoul National University)
A Case for Pre-trained Language Models in Systems Engineering
Lim, S. C. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 5
Cross-domain Relation Extraction
Bassignana, E. (2024) · IT University Of Copenhagen (IT University of Copenhagen)
Damgard, J., & Löfgren, E. (2025) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Automating Text Naturalness Evaluation of NLG SystemsPreprint
Çano, E., & Bojar, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13268
Information Extraction Beyond Sentence Boundary
Youmi, M. A., & Ma, Y. (2024) · Institutional Repositories DataBase (IRDB)
Automated Information Warfare: For and Against Saturation Attacks
Low, J. F. (2023) · eScholarship@McGill (McGill) · doi:10.82308/4782 · Cited by 1
Knowledge graph embedding enhancement using ontological knowledge in the biomedical domain
Niccolai, L. (2023) · AMS Degree Thesis (University of Bologna)
Subtopic-oriented biomedical summarization using pretrained language models
Xia, T. (2023) · AMS Laurea Institutional Research Repository (University of Bologna)
計算機によって支援されたピアレビューにおける引用目的の役割に関する研究
Basuki, S. (2023) · Institutional Repositories DataBase (IRDB)
Advances in Natural Language Processing for Business
Borchert, P. (2024) · Lirias
Issifu, A. M. (2022) · Dspace Repository (Marmara Üniversitesi)
Evolution of ESG-focused DLT Research: An NLP Analysis of the LiteraturePreprint
Hernandez, W., Tylinski, K., Moore, A. P., Roche, N., Vadgama, N., Treblmaier, H., Shangguan, J., Tasca, P., & Xu, J. (2023) · UCL Discovery (University College London) · Cited by 1
An Investigation of Context-Driven Caption Generation
YANG, Z., & Yang, Z. (2025) · Institutional Repositories DataBase (IRDB)
Farzana, S. M., Fröbe, M., Hendriksen, G., Granitzer, M., Hiemstra, D., Potthast, M., & Zerhoudi, S. (2024) · Radboud Repository (Radboud University)
Pretraining Large Language Models for Finnish
Luukkonen, R. (2025) · UTUPub (University of Turku)
Alonso Casero, Á. (2021) · Hispana · Cited by 19
TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretraining
Zhang, R., Somayajula, S. A., & Xie, P. (2024) · PubMed · doi:10.48550/arxiv.2410.10006
“…to pretrain a BERT model on legal documents to improve performance on legal NLP tasks (Chalkidis et al., 2020), while SciBERT leverages a large multi-domain corpus of scientific publications for further pretraining, enhancing its effectiveness on scientific NLP tasks (Beltagy et al., 2019).”
BackgroundIdentification of Potential Co-citation Linkages via Context-Aware Citation Network Embeddings
Eto, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-06136-2_20
Analyzing and Predicting Cooperation Behavior in Bibliometric Data
Koopmann, T. (2025) · Online Publication Service of Würzburg University (Würzburg University) · doi:10.25972/opus-42514
Foussat, A.-S., Guigue, V., Sauvion, N., Bossy, R., & Nédellec, C. (2025) · HAL
Large Language Models for Biological Knowledge Extraction
Wang, H., & Li, M. (2025) · Computational Molecular Biology · doi:10.5376/cmb.2025.15.0016
Generating Synthetic Training Data for Named Entity Recognition
Ocklind, V. (2025) · Digitala vetenskapliga arkivet (Diva) (Karlstad University)
Automated Detection and Categorization of Sexism in Multimodal Social Network Content
Fritzsche, F. (2025) · MOnAMi (Hochschule Mittweida Hochschulbibliothek)
TGDK, Volume 3, Issue 3, Complete Issue
Informatik, S. D. L.-Z. F. (2025) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/tgdk.3.3
Doniparthi, G. (2025) · Publication Server of Kaiserslautern University of Technology (Kaiserslautern University of Technology) · doi:10.26204/kluedo/9361
Tracking Citation Practices Across Disciplines: A SourceMapper Analysis of MICUSP
Kane, M. (2025) · The Journal of Writing Analytics · doi:10.37514/jwa-j.2025.8.1.01
ArcBERT: An LLM-based Search Engine for Exploring Integrated Multi-Omics MetadataPreprint
Doniparthi, G., Pandhare, S. B., Deßloch, S., & Mühlhaus, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.15365
“Domain-specific LLMs such as BioBERT [10] and SciBERT [11] have demonstrated effectiveness in tasks like question answering and text mining.”
A Comprehensive Evaluation of Llama 3 for Text Classification TasksPreprint
Amjadi, A., TaghipourEivazi, S., Arasteh, B., & Kusetoğulları, H. (2025) · Research Square · doi:10.21203/rs.3.rs-8354244/v1
Li, H., Qiao, H., Wu, S., Ren, Y., Sun, W., & Zheng, J. (2025) · IEEE International Conference on Unmanned Systems (ICUS) · doi:10.1109/icus66297.2025.11293982
“However, domain-specific adaptations like BioBERT [6], SciBERT [7], and specialized variants such as FF-BERT [8] for disaster classification demonstrate the value of domain-optimized pre-training while revealing gaps in addressing military-specific needs such as equipment attribute modeling (”cruising…”
The Knowledge Cosmos: An Immersive Platform for Interdisciplinary Research Discovery
McGail, A., Tajani, R., Sridhar, N., & Li, J. (2025) · IEEE VIS Arts Program (VISAP) · doi:10.1109/visap.2025.00007
“To produce these the authors begin with embeddings from the SciBERT [2] model, which has been trained to predict missing words in scientific text and whether one sentence follows another.”
A Text-Enhanced Statistical Learning Framework for Robust Frontier Research Identification
Chen, H., Du, J., & Xue, Z. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-1106-8_16
ANAVIT: Enhancing Document-Level Relation Extraction with Anaphor Nodes and Visual Transformation
Pham, A. D. V., Thi, Q.-T. P., & Dang, T. H. (2025) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse68178.2025.11309552
“SciBERT [21] provided domain-specific pre-trained models that significantly improved biomedical NLP tasks.”
Trend Prediction of NLP Research based on few-shot in-context Learning
Shao, M.-Q., Liu, R., Li, W.-S., & Liu, Y. (2025) · International Conference on Wavelet Analysis and Pattern Recognition · doi:10.1109/icwapr66259.2025.11306652
“However, most of the existing studies focus on the application of pre-trained models in text generation and basic understanding tasks [20].”
Can AI Predict Scientific Research? a Case Study on the NLP Domain
Shao, M.-Q., Liu, R., Li, W.-S., & Liu, Y. (2025) · International Conference on Wavelet Analysis and Pattern Recognition · doi:10.1109/icwapr66259.2025.11306695
“The document representation approach by Le and Mikolov [10] facilitates nuanced semantic clustering, while LLMs such as GPT-3 [11] show remarkable capabilities in reasoning across documents.”
Sosnin, I. M., Reznikova, A. R., & Frolova, A. V. (2025) · Reviews on advanced materials and technologies · doi:10.17586/2687-0568-2025-7-4-203-210
Lyu, Z., & Ke, Q. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.22524
“Recent advances in multimodal learning[26] suggest that fusing citation embed-dings with text-based representations [27, 28] might be able to better capture the topology of science entities.”
Adversarial Attack on Cyber Threat Intelligence System
R, G. L. E., Arikkat, D. R., Vinod, P., A., R. R. K., & Deepa Reddy, K. (2025) · International Conference on Contemporary Computing · doi:10.1109/ic366947.2025.11290182
“OSTIS employed CySecBERT [16] for title-based classification and SciBERT [17] for description-based classification.”
Literature Mining System for Nutraceutical Biosynthesis: From AI Framework to Biological InsightPreprint
Sun, X., Sarmah, N., & Guo, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.22225
Abdelhamed, R., Essgaer, M., Agaal, A., & shibani, A. S. M. S. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-00232-7_42
LLM-Based Abstractive Summarization of PubMed Articles
Ho, E., Heidari, M., Zad, S., Boroojeni, K. G., & Headden, C. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0433-6_14
Efficient Tuning Framework for Resource- Constrained Biomedical Question Answering
Wang, B., Du, Y., Jin, X., & Wang, Z. (2026) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3650547
Kumar, S., Arora, H., Ghosal, T., & Ekbal, A. (2026) · Scientometrics · doi:10.1007/s11192-025-05529-2 · Cited by 1
“We show three variants of our model i.e. JDAS-SBERT, JDAS-SPECTRE and SCIBERT (Beltagy et al., 2019) variant of JDAS ie: JDAS-SCIBERT, the difference between them being the choice of transformer model for creating the encodings one makes use of Google’s BERT (Devlin et al., 2018)) while the other makes use of Allen AI…”
Analyzing and Improving Cross-lingual Knowledge Transfer for Machine TranslationPreprint
Stap, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.04036
Zaidi, S. S. A., Sharif, A. Z., Shahbaz, M. H., & Urooj, S. (2025) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit67061.2025.11333701
“ML techniques such as SVMs, Random Forests, and KNN have been applied using features like amino acid composition, dipeptide frequency, and physicochemical properties [17].”
Makohon, I., Najafi, M., Wu, J. F., Brochhausen, M., & Li, Y. (2026) · Journal of Computational Biology · doi:10.1177/15578666251411390 · Cited by 1
“To better understand why CoT prompting outperforms both CoT-KG approaches in aligning ground-truth and generated outputs, we applied UMAP (McInnes et al., 2018) along with the transformer-based model SciBERT (Beltagy et al., 2019) to visualize noun embeddings across different prompting strategies.”
Function-based Patent Landscape Analysis for Diabetes Technologies: Insights from the U.S. and China
Wu, Z., & Limpanadusadee, J. (2025) · International Journal of Contents · doi:10.5392/ijoc.2025.21.4.116
Recent Advances in Large Language Models
Sharma, P., & Jain, S. K. (2025) · International Conference on Computational Intelligence and Computing Applications (ICCICA) · doi:10.1109/iccica67008.2025.11337670
“In scientific research, models Galactica [5] and SciBERT [38] assist in literature review and summarization of scientific papers respectively.”
AROMSUK, T., Nootyaskool, S., & Netisopakul, P. (2025) · doi:10.1109/icitee66631.2025.11338426
AROMSUK, T., Netisopakul, P., & Nootyaskool, S. (2026) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2025dap0003
Imanov, O. Y. L., Yilmaz, T., & Kulali, D. U. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.14862
Adaptive Fine-Tuning for Multi-Class Classification Over Software Requirement Data
Yıldırım, S., Çevik, M., & Basar, A. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00078
Detecting AI-Generated Scientific Abstracts Using Galactica and Graph Neural Networks
Pathak, A., Sharma, D. K., Agrawal, H., Chawuthai, R., & Petchhan, J. (2025) · International Conference on Information Systems and Computer Networks (ISCON) · doi:10.1109/iscon65210.2025.11341445
“Fine-tuned language models such as RoBERTa [19], De-BERTa [21], and SciBERT [18] have been employed to distinguish human-and AI-generated text based on learned semantic patterns.”
Adaptive Fine-Tuning for Multiclass Classification Over Software Requirement Data
Yıldırım, S., Çevik, M., & Basar, A. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00039
“Based on Siamese network structures, SBERT can produce semantically meaningful and independent embeddings of the sentences, making it a suitable candidate for a baseline method.”
KEEL: Knowledge Engineering for Educators Layer
Konstantinov, A. M., Avdyushina, A. E., Solodkaya, M. A., Pakhorukova, A., Ivanov, A., & Danilenko, I. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-14674-8_41
Fine-Tuning SciBERT for Citation Intent Classification: An Empirical Study
Firdaus, J., & Ruskanda, F. Z. (2025) · IEEE International Conference on Data and Software Engineering (ICoDSE) · doi:10.1109/icodse68111.2025.11351694
“While our final score is comparable to that reported in the original SciBERT paper [8], our work provides a transparent optimization roadmap for a modern baseline and a deeper analysis of the task’s specific challenges, particularly regarding data augmentation and semantic ambiguity.”
Intelligent Extraction Technology of Security Attributes for Chip Manual
Huang, Y., Wang, Y., Liang, S., Yang, T., & Feng, Y. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343005
“…biomedical corpora to capture domain-specific semantics, thereby addressing the lexical and syntactic peculiarities of biomedical literature [8]. b) Data Augmentation: Data Augmentation is a widely adopted technique in machine learning and deep learning, aimed at enhancing dataset diversity…”
CLEAR: Contextual Learning for Enhanced Scientific Articles Recommendation
Elayadi, K., & Yanes, N. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342815
“This section reviews six systems — EILEEN [11], RECSA [12], ARziGO [13], an LSTM-based approach [7], CiteSeerX [14], and SciBERT-based systems [8] — focusing on content-based modeling, behavioral context, semantic enrichment, and interpretability.”
Park, N., Kim, S. A., Jaekwang, K. I. M., & Shim, J. (2026) · Electronics · doi:10.3390/electronics15030553
Leveraging Clinical Notes with Medical NER for Treatment Recommendation
Saleem, M. A., Raza, S. A., & Haider, S. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343709
“[11] : SciSpacy’s SciBERT model with a biomedical vocabulary of ~785,000 terms.”
Condori-Luna, G. F., Oliveros, D. A. V., & Reis, M. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15984-7_27
Bomgni, A. B., Donald, F. N., Aryal, S., Gurung, B. D. S., Gadhamshetty, V., & Gnimpiéba, E. Z. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356949
Enhancing Research Paper Title Generation Using GPT-2 for Natural Language Processing
Rokade, S., Biradar, R., Patil, K., Anigol, S., & Bagewadi, S. (2025) · International Conference for Advancement in Technology (ICONAT) · doi:10.1109/iconat66879.2025.11362623
“The goal of domain-specific models like SciBERT [6] and systems like TLDR generators [3] is to improve communication and accessibility in academic work.”
Utility-Based Preference Training for Effective Synthetic Text Classification
Gwak, J., & Jung, Y. (2026) · Mathematics · doi:10.3390/math14030507
Optimizing Research Topic and Researcher Clustering Through Big Data and Graph Theory
Bolla, S. R., & Narsepalle, K. R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-95-2878-3_35
Insardi, A., & Gradvohl, A. L. S. (2026) · Scientometrics · doi:10.1007/s11192-026-05784-x
“Sections "Phase I: construction of SciBERT-SolarPhysics-Search as a retrieval infrastructure", "Phase II: application-layer evidence-space reconstruction and temporal mirroring", and "Phase III: grounded analytical synthesis under structural constraints" describe the three experimental phases.”
InfluentialHao, Z., Duan, Y., Wang, Y., & Qian, H. (2025) · doi:10.2196/preprints.81833
DecoderTCR: Compositional Pretraining and Entropy-Guided Decoding for TCR-pMHC InteractionsPreprint
Lai, B., Englund, M., Bharanikumar, R., Nocedal, I., Davariashtiyani, A., Perera, J. F., & Khan, A. A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.04.703820 · Cited by 1
“Domain-adaptive pre-training on scientific (Beltagy et al., 2019) and biomedical (Lee et al., 2020) text improves down-stream performance.”
Kashtriya, P., & Singh, P. (2025) · International Journal of Advances in Signal and Image Sciences · doi:10.29284/538mtv24
Leveraging Large Language Models (LLMs) for Service Optimization in Beyond 5G (B5G) Networks
Muralidharan, S., Woo, W., & Lee, N. (2025) · Springer eBooks · doi:10.1007/978-3-032-06418-9_15
Multi-Granular Encoder-Decoder Architecture for Explainable Fact-Checking via Multi-Task Learning
Nathani, M., Misra, R. N., & Mukhopadhyay, S. (2025) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1109/aimlsystems67835.2025.11387032
Types of Embeddings and their Application in Intellectual Academic Genealogy
Marinosyan, A. K. (2026) · Russian Digital Libraries Journal · doi:10.26907/1562-5419-2026-29-1-240-261
Domain-Specific Health Text Generation Through Low-Rank Adaptation of a Transformer Architecture
Debnath, K., Das, S., Handique, M., Paul, A., Thangjom, L. S., & Arakeri, M. P. (2025) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon66050.2025.11375045
“…BERT [2] Encoder (Transformer) General NLP 0.11 Pretraining, QA, NER Full BioBERT [4] Encoder (BERT) Biomedical 0.11 NER, QA, RE Full SciBERT [5] Encoder (BERT) Scientific Literature 0.11 NER, QA, Classification Full BioGPT [6] Decoder (GPT-2) Biomedical 0.35 Generation, Inference Full…”
InfluentialHsu, T. T. Y., & Lu, O. H. T. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2623468
Small Models, Big Impact: Rethinking Scale of LLMs in Cybersecurity
Villagomez‐Bernabe, B., & Devereux, B. J. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11390922
“This staged approach aims to reduce the risk of overőtting to potentially suboptimal initial embeddings [7].”
Heterogeneous Multi-Head Attention Fusion Network for Enhanced Q&A Reasoning
Sun, J., Yu, L., & Ma, L. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00186
“We select the corresponding pre-trained LMs on different datasets such as SapBERT-Base , PubmedBERT [25] , BioBERT-base and BioBERT-Large [26] , BioRoBERTa-Base [27] , SciBERT [28] , BERT-Base [29] , and Roberta-Large.”
Specialised natural language processing methods for modern human resource management
Decorte, J.-J. (2025) · Ghent University Academic Bibliography (Ghent University)
Manage, P. D., Wu, Y., Zhang, J., & Li, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_37 · Cited by 1
Document-Level Relation Extraction with Multi-scale and Non-bridge Reasoning
Wang, Z., Wang, Y., & Zhao, W. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_27
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.