SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,701–1,800 of 3,596
Generating and visualizing trace link explanations
Liu, Y., Lin, J., Anuyah, O., Metoyer, R., & Cleland‐Huang, J. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510129 · Cited by 6
“In this study, we used SciBERT[9] as our language model, which has been pre-trained with massive papers in difference science domains.”
MethodGenerating Fluent Fact Checking Explanations with Unsupervised Post-Editing
Jolly, S., Atanasova, P., & Augenstein, I. (2022) · Information · doi:10.3390/info13100500 · Cited by 23
Frisoni, G., Italiani, P., Boschi, F., & Moro, G. (2022) · International Conference on Data Science, Technology and Applications (DATA) · doi:10.5220/0011354900003269 · Cited by 10
A marker collaborating model for entity and relation extraction
Yi-zhao, W., Chen, Y., Qin, Y., Huang, R., & Tang, R. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.08.038 · Cited by 7
Bijari, K., Zoubi, Y., & Ascoli, G. A. (2022) · Brain Informatics · doi:10.1186/s40708-022-00174-4 · Cited by 6
“These models are difficult to generalize, because the entity categorical needs differ across domains, and for many specific domains no suitable datasets are publicly available [6, 20].”
BackgroundPorciello, J., Lipper, L., & Ivanina, M. (2023) · Frontiers in Sustainable Food Systems · doi:10.3389/fsufs.2022.1013701 · Cited by 6
“…(BERT), SciBERT and named-entity recognition with BERT are highly adept at capturing the context-dependent meaning of words even before additional training for other tasks that require expert input in the form of training data (Devlin et al., 2018; Beltagy et al., 2019; Luoma and Pyysalo, 2020).”
BackgroundMethodA Joint Domain-Specific Pre-Training Method Based on Data Enhancement
Gan, Y., Lu, G., Su, Z., Lei, W., Zhou, J., Jiang, J., & Chen, D. (2023) · Applied Sciences · doi:10.3390/app13074115 · Cited by 7
Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis
Qiu, C., Wu, W. M. C., Zhang, X. F., & Wang, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.659 · Cited by 9
Türkmen, H., Dikenelli, O., Eraslan, C., Çallı, M. C., & Özbek, S. S. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.22 · Cited by 8
Question Answering Versus Named Entity Recognition for Extracting Unknown Datasets
Younes, Y., & Scherp, A. (2023) · IEEE Access · doi:10.1109/access.2023.3309148 · Cited by 7
Graph-Enriched Biomedical Entity Representation Transformer
Sakhovskiy, A., Semenova, N., Kadurin, A., & Tutubalina, E. V. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42448-9_10 · Cited by 6
Gonzalez, A. D. D., Hughes, K. S., Yue, S., & Hayes, S. T. (2023) · International Conference on Information System and Data Mining (ICISDM) · doi:10.1145/3603765.3603771 · Cited by 6
“The model can be fine-tuned on task-specific datasets to perform tasks, such as named entity recognition and relation extraction [14, 27, 28], which are two critical tasks to construct domain-specific knowledge graphs.”
BackgroundHuang, X., Zhu, P., Chen, Y., & Ma, J. (2023) · Scientometrics · doi:10.1007/s11192-023-04825-z · Cited by 7
“SciBERT is a transformer model proposed by (Beltagy et al., 2020).”
BackgroundMethodInfluentialDeep dive into language traits of AI-generated Abstracts
Kumar, V., Bharti, A., Verma, D., & Bhatnagar, V. (2024) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3632410.3632471 · Cited by 8
“Semantic features include sentence-sentence similarity using SciBert [3], sentence-title similarity and named entities in the text.”
MethodOn the Use of Language Models for Function Identification of Citations in Scholarly Papers
Ikoma, T., & Matsubara, S. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.15 · Cited by 7
Investigating the Impact of Syntax-Enriched Transformers on Quantity Extraction in Scientific Texts
Bölücü, N., Rybinski, M., & Wan, S. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.1 · Cited by 6
Automatic Aspect Extraction from Scientific Texts
Marshalova, A., Bruches, E. P., & Batura, T. V. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-67008-4_6 · Cited by 6
PatentTransformer: A Framework for Personalized Patent Claim Generation.
Lee, J.-S. (2019) · Cited by 6
Concept Wikification for COVID-19
Lymperopoulos, P., Qiu, H., & Min, B. (2020) · Workshop on NLP for COVID-19 (NLP-COVID19) · doi:10.18653/v1/2020.nlpcovid19-2.29 · Cited by 6
Avram, A.-M., Zaharia, G.-E., Cercel, D.-C., & Dascălu, M. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.65 · Cited by 6
Elkaref, M., & Hassan, L. (2021) · Social Media Mining for Health Research and Applications Workshop and Shared Tasks (SMM4H) · doi:10.18653/v1/2021.smm4h-1.16 · Cited by 6
“Gattepaille (2020) showed that simply tuning a base BERT (Devlin et al., 2019) model could achieve strong results, even beating ensemble meth-ods that rely on tranformers pretrained on more academic texts such as SciBERT (Beltagy et al., 2019), BioBERT (Lee et al., 2020) or ensembles of them, while approaching the per…”
BackgroundMethodDevelopment of International Agricultural Trade Using Data Mining Algorithms-Based Trade Equality
Yan, L. (2021) · Mobile Information Systems · doi:10.1155/2021/5046244 · Cited by 6
Empathy and Hope: Resource Transfer to Model Inter-country Social Media Dynamics
Yoo, C. H., Palakodety, S., Sarkar, R., & KhudaBukhsh, A. R. (2021) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2021.nlp4posimpact-1.14 · Cited by 10
ECNUICA at SemEval-2021 Task 11: Rule based Information Extraction Pipeline
Lin, J., Ling, J., Wang, Z., Liu, J., Chen, Q., & He, L. J. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.185 · Cited by 7
Towards extraction of theorems and proofs in scholarly articles
Mishra, S., Pluvinage, L., & Senellart, P. P. (2021) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/3469096.3475059 · Cited by 6
Improving hierarchical product classification using domain-specific language modelling
Brinkmann, A., & Bizer, C. (2021) · MADOC (University of Mannheim) · Cited by 6
An Interactive Fusion Model for Hierarchical Multi-label Text Classification
Zhao, X., Zhao, L., Zhang, X., Wang, J., Chen, T., Ju, Z., Wang, C., Zhang, C., & Zhan, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17189-5_14 · Cited by 6
Sentiment Drift Detection and Analysis in Real Time Twitter Data Streams
Susi, E., & Shanthi, A. P. (2022) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.032104 · Cited by 7
Chen, C.-Y., Tien, K.-Y., Cheng, Y.-H., & Lee, L. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.107 · Cited by 6
Biomedical Information Extraction for Disease Gene PrioritizationPreprint
Parmar, J., Koehler, W. C., Bringmann, M., Volz, K. S., & Kapicioglu, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05188 · Cited by 6
BagBERT: BERT-based bagging-stacking for multi-topic classificationPreprint
Rakotoson, L., Letaillieur, C., Massip, S., & Laleye, F. A. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05808 · Cited by 6
“We used 3 BERT models with vocabularies from PubMed (7), on clinic data (8) and SciBERT on CORD-19 (9,10) .”
SimCLAD: A Simple Framework for Contrastive Learning of Acronym DisambiguationPreprint
Ли, Б., Xia, F., Weng, Y., Huang, X., & Sun, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14306 · Cited by 6
“This architecture of the SciBERT follows the same architecture as BERT to capture the well-formed representation of the scientific data.”
BackgroundMethodInfluentialProceedings of The 4th Workshop on e-Commerce and NLP
(2021) · doi:10.18653/v1/2021.ecnlp-1 · Cited by 6
Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research PapersPreprint
Brack, A., Hoppe, A., Buschermöhle, P., & Ewerth, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06008 · Cited by 6
“In contrast to Jin and Szolovits [34], we utilise SciBERT [6] as word embeddings and evaluate the approach on abstracts as well as full papers.”
BackgroundMethodEffective transfer learning for low-resource natural language understanding
Liu, Z. (2022) · doi:10.14711/thesis-991013106358603412 · Cited by 9
Galiani, S., Gálvez, R. H., & Nachman, I. (2023) · National Bureau of Economic Research · doi:10.3386/w31295 · Cited by 6
Procko, T., Davidoff, A., Elvira, T., & Ochoa, O. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4594836 · Cited by 8
Measuring the Commercial Potential of Science
Masclans, R., Hasan, S., & Cohen, W. M. (2024) · National Bureau of Economic Research · doi:10.3386/w32262 · Cited by 9
A Multidisciplinary Multimodal Aligned Dataset for Academic Data Processing
Song, H., Xu, H.-Y., Wang, Z.-K., Wang, Y.-F., & Li, J. (2025) · Scientific Data · doi:10.1038/s41597-025-04415-z · Cited by 4
Prasad, D., Pimpude, M., & Alankar, A. (2024) · arXiv (Cornell University) · doi:10.13140/rg.2.2.15272.34561 · Cited by 6
Yu, J., Tang, Y., Feng, K., Liang, L., Zhang, Q., Ding, K., & Chen, H. (2026) · Scientific Data · doi:10.1038/s41597-026-06594-9 · Cited by 9
“Scientific texts are often dense with technical terminology, implicit assumptions, multi-modal data representations, and tightly interlinked concepts that require deeper contextual comprehension (Beltagy et al., 2019; Mann et al., 2020).”
Drinkall, F., Rahimikia, E., Pierrehumbert, J. B., & Zohren, S. (2024) · arXiv · doi:10.48550/arxiv.2404.18543 · Cited by 6
“This insight led to the development of domain-specific language models such as BioBERT (Lee et al., 2019b), SciBERT (Beltagy et al., 2019), FinBERT (Yang et al., 2020), and more recent models like BloombergGPT (Wu et al., 2023).”
MethodLarge Language Model Agent for Modular Task Execution in Drug Discovery
Ock, J., Meda, R. S., Badrinarayanan, S., Aluru, N. S., Chandrasekhar, A., & Barati Farimani, A. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c02454 · Cited by 5
“We employed SciBERT 56 as the embedding encoder because it is pretrained on a large corpus of scientific publications from Semantic Scholar, capturing richer representations of biomedical and technical terminology than general-domain encoders such as RoBERTa.”
SciND: a new triplet-based dataset for scientific novelty detection via knowledge graphs
Gupta, K., Ahmad, A., Ghosal, T., & Ekbal, A. (2024) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00386-x · Cited by 5
“We initialize the SciBERT [25] layer with the pre-trained weight from allenai / scibert _ sci v ocab _ uncased .”
“Paper, Meet Code”: A Deep Learning Approach to Linking Scholarly Articles With GitHub Repositories
Puangjaktha, P., Choetkiertikul, M., & Tuarob, S. (2024) · IEEE Access · doi:10.1109/access.2024.3399767 · Cited by 7
Heterogeneous hypergraph learning for literature retrieval based on citation intents
Shi, K., Liu, K., & He, X. (2024) · Scientometrics · doi:10.1007/s11192-024-05066-4 · Cited by 5
“For citing and referenced literature, the intent type between them is defined as 𝜏 ( v i ) , 𝜙 ( e ) , 𝜏 ( v j ) = SciBERT ( v i , v j ) .”
InfluentialEnhancing Financial Domain Adaptation of Language Models via Model Augmentation
Tanabe, K., Hirano, M., Matoya, K., Imajo, K., Sakaji, H., & Noda, I. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825292 · Cited by 6
Liad, B., Berman, E., Jagodnik, K. M., & Bartal, A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3565553 · Cited by 6
“Following the network integration, in Step 4, we calculated feature embedding on each node type individually, utilizing our fine-tuned SciBERT model [19].”
InfluentialEnhanced Biomedical Named Entity Recognition Using SpaCy and BERT Models
Kumar, M., Singh, P., & Kashtriya, P. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.446 · Cited by 10
Zhang, C., Xie, L., & Wei, S. (2025) · Library & Information Science Research · doi:10.1016/j.lisr.2025.101353 · Cited by 5
“The abstraction network then feeds these input texts into SciBERT for text embedding (Beltagy et al., 2019).”
Research Knowledge Graphs: The Shifting Paradigm of Scholarly Information Representation
Zloch, M., Dessì, D., D’Souza, J., Castro, L. J., Zapilko, B., Karmakar, S., Mathiak, B., Stocker, M., Otto, W., Auer, S., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94578-6_8 · Cited by 10
Li, X., Zhao, D., Meng, J., & Lin, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128342 · Cited by 6
AI-empowered health coaching for university students: A mixed-method process evaluation
Bojić, I., Ong, Q. C., Ito, S., Liu, J., Lawate, A., Palaiyan, M., Nair, E., Lwin, M. O., Theng, Y. L., Chia, M., et al. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110271 · Cited by 6
Quantum gate control pulse optimization based on the Adam algorithm
Yang, M., Feng, Y., Lu, B., Zhao, H., Ma, G. L., & Wang, L. (2025) · Quantum Information Processing · doi:10.1007/s11128-025-04791-w · Cited by 6
Wu, Y. (2025) · Soil Advances · doi:10.1016/j.soilad.2025.100065 · Cited by 5
Liu, X., Rastegari, S., Huang, Y., Cheong, S. C., Liu, W., Zhao, W., Tian, Q., Wang, H., Guo, Y., Zhou, S., et al. (2025) · Methods · doi:10.1016/j.ymeth.2025.07.004 · Cited by 6
A novel framework with ComMAND: A combined method for author name disambiguation
Rodrigues, N. S., & Ralha, C. G. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104304 · Cited by 5
Deng, X., Wang, X., & Stevenson, M. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744614 · Cited by 10
“However, existing scientific fact-checking systems primarily operate at the abstract level, adopting methodologies similar to general fact-checking [49, 50, 68, 72, 75, 97, 100, 102, 111, 119], albeit incorporating domain-specific models such as BioSentVec [21] and SciB-ERT [11].”
Li, T., Wang, J., Zhang, Y., Li, S., & Chen, L. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736829 · Cited by 8
“Only output one word as the answer: Our motivation for experiments with the LLMs is to offer a potential way to take advantage of the LLMs for citation classification, as well as to shed light on the possible performance.”
Lázaro, F. L., Santos, L. F. F. M., Valério, D., & Melício, R. (2025) · Applied Sciences · doi:10.3390/app15179403 · Cited by 5
Liu, Y., & Luo, H. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103820 · Cited by 9
Doneva, S. E., de Viragh, S., Hubarava, H., Schandelmaier, S., Briel, M., & Ineichen, B. V. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10031 · Cited by 7
Wang, M., Liu, H., Zhang, M., & Ikram, R. M. A. (2025) · Atmosphere · doi:10.3390/atmos16091102 · Cited by 7
“These procedures ensured trans-parency, reproducibility, and broad applicability, with repeated-run consistency and expert validation confirming the reliability of the generated topic labels and summaries [30].”
Weissenbacher, D., Zhao, X., Priestley, J. R. C., Szigety, K. M., Schmidt, S. F., O’Connor, K., Soysal, E., Roberts, K. E., Qi, J., Luo, L., et al. (2025) · Database · doi:10.1093/database/baaf051 · Cited by 7
“We evaluated several BERT models to use as the first layer of the system, including BioBERT [ 5 ], SciBERT [ 27 ], PubMedBERT [ 25 ], and ClinicalBERT [ 28 ].”
Hu, H., Stirrat, H. J., Alayli, A., Saeki, A., & Huang, Y. (2025) · ACS Omega · doi:10.1021/acsomega.5c03612 · Cited by 8
Koon, Y. L., Tan, H. X., Teo, D. C. H., Neo, J. W., Ang, P. S., Loke, C. W. P., Tham, M. Y., Tan, S. H., Soh, B. L. S., Foo, P. Q. B., et al. (2025) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.70100 · Cited by 7
Oyelade, O. N., Wang, H., & Rafferty, K. J. (2025) · ACM Computing Surveys · doi:10.1145/3774628 · Cited by 10
“In the science domain, an LMs named SciBERT has been researched and is also based on improving BERT [ 74 ].”
Does scientific writing converge to U.S. english? evidence from generative AI-assisted publications
Filimonović, D., Rutzer, C., Macher, J. T., & Weder, R. (2026) · Scientometrics · doi:10.1007/s11192-026-05774-z · Cited by 8
“For each publication, we measure linguistic similarity to a benchmark corpus of U.S. scientific writing using SciBERT text embeddings 1 (Beltagy et al., 2019).”
InfluentialData engineering for AI applications in energetic materials
Zang, X., Feng, Q., Zhu, P., Ye, Y., & Shen, R. (2025) · FirePhysChem · doi:10.1016/j.fpc.2025.11.010 · Cited by 6
Parsa, M., Cumani, R., Fam, H. J. A., & Tawbe, B. (2025) · Natural Resources Research · doi:10.1007/s11053-025-10564-0 · Cited by 7
“To address this challenge, our domain-specific language model was fine-tuned on a large and diverse corpus of geoscientific literature, which increases the likelihood of encountering and learning meaningful representations of rare terms (Beltagy et al., 2019).”
Custom GPTs to aid in compliance checking for reporting standards in academic publishing
Mohapatra, R., & Härtung, T. (2026) · ALTEX · doi:10.14573/altex.2601011 · Cited by 5
SciWING– A Software Toolkit for Scientific Document Processing
Kashyap, A. R., & Kan, M. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.13 · Cited by 6
Variable Instance-Level Explainability for Text Classification.
Chrysostomou, G., & Αλέτρας, Ν. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 5
Carniel, T., Cazenille, L., Dalle, J., & Halloy, J. (2022) · Bioinspiration & Biomimetics · doi:10.1088/1748-3190/ac9208 · Cited by 7
“of the Bidirectional Encoder Representations from Transformers (BERT) model [7] specifically trained on scientific literature called SciBERT [8].”
MethodAutomated patent classification for crop protection via domain adaptation
Christofidellis, D., Lehmann, M. M., Luksch, T., Stenta, M., & Manica, M. (2023) · Applied AI Letters · doi:10.1002/ail2.80 · Cited by 5
Inline Citation Classification Using Peripheral Context and Time-Evolving Augmentation
Gupta, P., Atri, Y. K., Nagvenkar, A. S., Dasgupta, S., & Chakraborty, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33383-5_1 · Cited by 7
Self-supervised scientific document recommendation based on contrastive learning
Tan, S., Zhang, T., Zhao, S. G., & Zhang, Y. (2023) · Scientometrics · doi:10.1007/s11192-023-04782-7 · Cited by 6
“Lu et al. (2023) enhanced semantic information through SciBERT and WordNet (Miller, 1995) for citation recommendation in papers.”
MethodNewton, A. J. H., Chartash, D., Kleinstein, S. H., & McDougal, R. A. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05397-8 · Cited by 5
“Using SPECTER embeddings [21] solved both of these challenges as this pre-trained, SciBERT-powered [23] model places similar papers (e.g. those that have COVID-19 immune signatures) near each other in the embedding space without requiring any custom training.”
MethodInfluentialFine-Tuning Language Models for Scientific Writing Support
Mücke, J., Waldow, D., Metzger, L., Schauz, P., Hoffman, M., Lell, N., & Scherp, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40837-3_18 · Cited by 10
Zhang, Z., Fang, M., Wu, R. C., Zong, H., Huang, H., Tong, Y., Xie, Y., Cheng, S., Wei, Z., Crabbe, M. J. C., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/48115 · Cited by 7
“For the biomedical domain, we adopted 10 popular models with different pretraining corpus and parameter settings: BioM-ALBERT [33], BioM-ELECTRA [33], PubMedELECTRA [34], BioMed-RoBERTa [35], RoBERTa-PubMed [36], BioBERT [9], SciBERT [37], BioClinicalBERT [38], BlueBERT [39], and PubMedBERT [40].”
Gorelik, A. J., Gorelik, M. G., Ridout, K. K., Nimarko, A. F., Peisch, V. D., Kuramkote, S. R., Low, M., Pan, T., Singh, S., Nrusimha, A., et al. (2023) · Nature Mental Health · doi:10.1038/s44220-023-00109-w · Cited by 6
“Both BERT and SciBERT break up text into tokens representing words or subwords on the basis of a tokenization approach used in each transformer-based model.(40,41) However, BERT is trained on generalpurpose text corpus (book corpus and Wikipedia), whereas SciBERT is trained on scientific literature.”
MethodGuedes, G. B., & da Silva, A. E. A. (2023) · doi:10.5121/csit.2023.131923 · Cited by 7
“SciBERT, from [6], was chosen because it is a BERT-based language model for the scientific domain.”
MethodZheng, X., Wang, X., Luo, X., Tong, F., & Zhao, D. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05601-9 · Cited by 5
“For instance, SciBERT [23] and BioM-BERT [46] achieve F1-score of over 80% on a pre-processed CHEMPROT dataset different from that provided by Peng et al. [28], while [15] reports that SciBERT achieves 75.24% of F1-score on the same dataset to evaluate BioEGRE.”
BackgroundMethodInfluentialResearch on the structure function recognition of PLOS
Liu, J., Zhao, Z., Wu, N., & Wang, X. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1254671 · Cited by 5
A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets
Noguti, M. Y., Vellasques, E., & Oliveira, L. S. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394189 · Cited by 5
“The result goes in the opposite direction to other experiments conducted with the specialization of language models, for example with scientific vocabulary [12] or fine-tuned with biomedical corpora [13].”
ResultDSTEA: Improving Dialogue State Tracking via Entity Adaptive pre-training
Lee, Y., Kim, T., Yoon, H., Kang, P., Bang, J., & Kim, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111542 · Cited by 5
Vandana, C. P., Adithya, D., Kedilaya, D. D., Gondkar, S. S., & Halhalli, S. (2024) · International Conference on Artificial Intelligence and Machine Learning Applications Theme: Healthcare and Internet of Things (AIMLA) · doi:10.1109/aimla59606.2024.10531558 · Cited by 5
HOSSemEval-EB23: a robust dataset for aspect-based sentiment analysis of hospitality reviews
Doan, T. T., Tran, T. Q., Le, D. T., Tran, A. H., Nguyen, A. T., Le, T. H. A., Doan, T. N. T., Huynh, S. T., & Nguyen, B. T. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19518-9 · Cited by 5
“This modification involves utilizing Transformers models such as XLNet [25], ELECTRA [26], Sci-BERT [27], ConvBERT [28] and DistilBERT [29] in place of the BERT model.”
Recent Advances in Large Language Models: An Upshot
Rithika, R. (2024) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.5.0624.1403 · Cited by 6
Liu, X., Qin, X., Xu, C., & Li, H. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02259-4 · Cited by 8
“…h ( l − 1 ) j represents the output representation of node c j in the ( l − 1 ) -th layer ( h ( 0 ) j represents the output of the original SciBERT [36] encoder), W ( l ) and b ( l ) represent the training matrix and bias of the l -layer GCN, respectively, and ReLU (∗) represents the Rectified…”
Large models in transportation infrastructure: a perspective
Du, Y. (2024) · Intelligent Transportation Infrastructure · doi:10.1093/iti/liae007 · Cited by 11
Mining Chemical Process Information from Literature for Generative Process Design: A Perspective
Schweidtmann, A. M. (2024) · Systems and Control Transactions · doi:10.69997/sct.184704 · Cited by 5
Tang, X., Du, X., Wang, Q., & Wu, J. (2024) · Scientometrics · doi:10.1007/s11192-024-05157-2 · Cited by 6
“These vectors serve as representations of knowledge elements, facilitating diverse tasks such as abstract generation, question answering, literature recommendations, and more (Beltagy et al., 2019; Maheshwari et al., 2021).”
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Trippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 6
Pattern-based engineering of Neurosymbolic AI Systems
Ekaputra, F. J. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100855 · Cited by 6
Towards Cross-Modal Text-Molecule Retrieval with Better Modality Alignment
Song, J., Zhuang, W., Lin, Y., Zhang, L., Li, C., Su, J., He, S., & Bo, X. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821722 · Cited by 8
“A. Model Architecture a) Encoders: As a common practice, we use SciBERT [16] stacked with a linear layer as our text encoder.”
Leveraging Multimodal Retrieval-Augmented Generation for Cyber Attack Detection in Transit Systems
Munir, M. B., Cai, Y., Khan, L. R., & Thuraisingham, B. M. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00046 · Cited by 5
“There are already versions of BERT [23] that have been fine-tuned for similar tasks, and these could be adapted for our purposes as well.”
Kumar, A., Jami, H. C., Bakshi, B. R., Ramteke, M., & Kodamana, H. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109011 · Cited by 5
Puy, A., Bacon, E., Carmona, A., Flinders, S., Gefen, D., Khanjani, M., Larsen, K. R. T., Lachi, A., Linga, S. N., Piano, S. L., et al. (2025) · iScience · doi:10.1016/j.isci.2025.112184 · Cited by 5
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.