SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,201–2,300 of 3,596
TaxonGPT: Taxonomic Classification Using Generative Artificial IntelligencePreprint
Huang, H., Li, T., Wang, Z., Seldon, D. S., & Rodrigo, A. G. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.28.618575 · Cited by 4
Jefry, W., Hussain, F. K., & Al-Doghman, F. (2023) · Preprints.org · doi:10.20944/preprints202508.0753.v1 · Cited by 4
SCIRGC: Multi-Granularity Citation Recommendation and Citation Sentence Preference AlignmentPreprint
Li, X., & Chen, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.20103 · Cited by 4
“…into the paper’s association, but this method is limited by the high computational cost of GCN. [31] innovate transformer encoding, introduce paragraph attributes, and optimize sorting by scibert[32] to improve citation scores, where the prefetching reordering strategy improves citation scores.”
Assessing the relevance of R&D funding towards societal goals
Aristodemou, L., Appelt, S., Beuzekom, B. V., & Galindo‐Rueda, F. (2025) · OECD science, technology and industry working papers · doi:10.1787/bafcdc7b-en · Cited by 3
S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text UnderstandingPreprint
Wang, H., Guo, L., Huo, P., Lin, X., Yuan, Y., Jiang, J., & Liu, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.00264 · Cited by 3
Windisch, P., Dennstädt, F., Koechli, C., Förster, R., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · doi:10.1101/2024.07.09.24310155 · Cited by 2
“4,5 In clinical research, randomized controlled trials (RCTs) are the gold standard for testing the effectiveness of an intervention.”
BackgroundOelen, A., Jaradeh, M. Y., & Auer, S. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-97207-2_2 · Cited by 2
On the Rigour of Scientific Writing: Criteria, Analysis, and Insights
James, J., Xiao, C., Li, Y., & Lin, C. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.380 · Cited by 2
Segundo, L. B. D. S., Teixeira, V., Pedrini, H., & Dias, Z. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-65775-8_14 · Cited by 2
ZHANG, Y., & Ma, Q. (2022) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2022edp7034 · Cited by 3
Rae, A. R., Mork, J. G., & Demner‐Fushman, D. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24722 · Cited by 3
“SciBERT (Beltagy et al., 2019) is a BERT model with a domain-specific vocabulary that has been pre-trained on a corpus of scientific documents to improve performance on downstream scientific natural language processing tasks.”
MethodPrompt Learning with Structured Semantic Knowledge Makes Pre-Trained Language Models Better
Zheng, H.-T., Xie, Z., Liu, W., Huang, D., Wu, B., & Kim, H. (2023) · Electronics · doi:10.3390/electronics12153281 · Cited by 6
“Biomedical PLMs are typically built by adapting a general-domain PLM to the biomedical domain with the same model architecture and training objectives, as exemplified by BioBERT [31], BlueBERT [32], SciBERT [33], ClinicalBERT [34], and PubMedBERT [35].”
MethodEbert, L., Huettemann, S., & Mueller, R. M. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.076 · Cited by 4
Evaluation and Domain Adaptation of Similarity Models for Short Mathematical Texts
Steinfeldt, C., & Mihaljević, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66997-2_14 · Cited by 8
Gerasimov, I. V., Mehrabian, A., Kc, B., Alfred, J., & McGuire, M. P. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678676 · Cited by 2
“It was further shown that for science specific domains, LLMs trained on the domain specific corpora such as SCIBERT [13] produce better accuracy on the domain specific applications.”
BackgroundBiomedical Relation Extraction Using LLMs and Knowledge Graphs
Chellagurki, P., Kumaru, S. P. K., Peela, R. R., Yeluri, N., Rojas, C., & Jetcheva, J. G. (2024) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice62917.2024.00015 · Cited by 3
“Fine-tuning these models with biomedical text has been shown to lead to improved performance when performing biomedical NLP tasks in the context of BERT, e.g. [6]–[8].”
BackgroundMethodSciHyp: A Fine-Grained Dataset Describing Hypotheses and Their Components from Scientific Articles
Vasu, R., Sarasua, C., & Bernstein, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77847-6_8 · Cited by 2
Temporal Extraction and Retrieval
Alonso, O., & Spaniol, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674137 · Cited by 2
Experimenting with modeling-specific word embeddings
López, J. A. H., Durá, C., & Cuadrado, J. S. (2024) · Software & Systems Modeling · doi:10.1007/s10270-024-01250-5 · Cited by 2
Gorshkov, S., Ignatov, D. I., Chernysheva, A. Y., Goiko, V. L., & Kashpur, V. V. (2024) · IEEE Access · doi:10.1109/access.2024.3521857 · Cited by 3
Capponi, G., & Frenken, K. (2025) · Research Policy · doi:10.1016/j.respol.2025.105194 · Cited by 2
Entity and Relationship Extraction with High-quality Spans and Enhanced Marker Strategies
Li, Q., Yao, N., & Zhou, N. (2025) · Machine Intelligence Research · doi:10.1007/s11633-024-1515-3 · Cited by 2
Three-stage document-level entity relation extraction
Lu, B., Wang, X., Ming, W., & Wang, X. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07068-2 · Cited by 2
“…in the pre-trained model is mainly relied on to save and utilize the context information, and the long-distance relationship is implicitly captured without generating the document graph, so as to make more comprehensive use of the large amount of information in the document Beltagy et al (2019).”
Using join learning of multi-level features for automated essay scoring
Li, S., & Gao, Y. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11131-7 · Cited by 3
Zhao, X., Wang, X., Liu, Y., Chen, H., & Guo, R. (2025) · Scientometrics · doi:10.1007/s11192-025-05279-1 · Cited by 3
“SciBERT accurately represents academic terms (Beltagy et al., 2019).”
Extracting Information in a Low-Resource Setting: Case Study on Bioinformatics Workflows
Sebe, C., Cohen‐Boulakia, S., Ferret, O., & Névéol, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91398-3_21 · Cited by 2
“More specifically, we used this model with the SciBERT language model [2], more specifically trained on scientific articles.”
Native language identification from text using a fine-tuned GPT-2 model
Nie, Y. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2909 · Cited by 3
“SciBERT ( Beltagy, Lo & Cohan, 2019 ): Designed speci fi cally for scienti fi c text, SciBERT leverages a large corpus of scienti fi c literature to improve performance on domain-speci fi c tasks, making it particularly effective for identifying languages used in scienti fi c contexts.”
Wu, R., & Ardakani, S. P. (2025) · Electronics · doi:10.3390/electronics14112276 · Cited by 2
Improving Drug-Drug Interaction Extraction from Biomedical Literature Using Deep Multimodal Fusion
Nguyen, B.-N., Tran, B.-H., Can, D.-C., Do, T.-H., & Le, H.-Q. (2025) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-90194-2_36 · Cited by 2
MedSiML: A Multilingual Approach for Simplifying Medical Texts
Jain, H. A., Patel, C., Umatiya, R., Mistry, S., Krishna, A., & Beheshti, A. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6606-5_12 · Cited by 2
Hafid, S., Schellhammer, S., Kartal, Y. S., Παπαστεργίου, Θ., Dietze, S., Bringay, S., & Todorov, K. (2025) · ACM Transactions on the Web · doi:10.1145/3746170 · Cited by 2
“SciBERT [12]: Among top-performing models for several scientific-text classification tasks [12].”
Zhou, Z., Zeng, Z., Jiang, W., Zhu, Y., Mao, J., Yuan, Y., Xia, M., Zhao, S., Yao, M., & Chen, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-05499-8 · Cited by 2
AdaGram in Python: An AI Framework for Multi-Sense Embedding in Text and Scientific Formulas
Arokiaraj, A. J., Ibrahim, S., Then, A., Ibrahim, B., & Peter, S. (2025) · Mathematics · doi:10.3390/math13142241 · Cited by 2
Lai, R., Wu, W., Zou, L., Liao, F., Wang, Z., & Mi, H. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9080198 · Cited by 2
Yang, S., Wu, L., Wu, B., & Song, Y. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70012 · Cited by 2
“Various refined versions of BERT, such as SciBERT and Sentence-BERT, cater to specific functions and requirements (Beltagy et al., 2019; Reimers & Gurevych, 2019).”
El-allaly, E.-D., Oubelkacem, A., & Bourray, H. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00214-8 · Cited by 2
Montero‐Parodi, J. J., Sánchez, R. M. R., García, J. A., & Fdez-Valdivia, J. (2025) · Expert Systems · doi:10.1111/exsy.70131 · Cited by 2
“These sentences are encoded by using SciBert (Beltagy et al. 2019).”
Attend to What I Say: Highlighting Relevant Content on Slides
Mariam, K. M. M., & Jawahar, C. V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04627-7_2 · Cited by 2
AI-driven knowledge-data graph for tracking global arbovirus research
Ming, Z., Huang, Y., Hu, C., Peng, J., Sun, W., Guo, W., Tang, S., Huangfu, X., Li, Z., Li, X., et al. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01256-w · Cited by 2
“Inspired by domain-specific adaptations of BERT, such as BioBERT [19] and SciBERT [20], ArboBERT was fine-tuned on corpora specific to arboviral research to capture the unique vocabulary and contextual relationships within this domain.”
Yao, Z., Chen, S., Zhang, S., & Zhang, Z. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05765-0 · Cited by 3
“Sentence-BERT (Beltagy et al., 2019), which derives fi xed-sized sentence embeddings through a pooling operation on BERT outputs, is selected to quantify content similarity.”
Smarter Crowdsourcing With NLP and Attention Mechanisms for Task Complexity Prediction
Munir, Y., Umer, Q., Aslam, M. W., Faheem, M., & Hakami, H. A. (2025) · IEEE Access · doi:10.1109/access.2025.3615745 · Cited by 2
Ciubotaru, B.-I. (2025) · International Conference on Control Systems and Computer Science · doi:10.1109/cscs66924.2025.00063 · Cited by 3
“In academic contexts, LLMs assist with literature reviews by summarizing research papers and identifying connections between studies [43].”
Discourse-Aware Language Representation
Zhang, Z., Ouyang, S., & Zhao, H. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3621229 · Cited by 2
“ThenotableexamplesareBioBERT[32], SciBERT[33],Clinical-BERT[34],andDialoGPT[35].”
Lee, S., & Kim, R. E. (2025) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.70101 · Cited by 3
“Herein, the proposed framework adopted SciBERT, which is a pretrained BERT model specifically designed for scientific text data and known for its superior performance in handling academic and scientific text-related tasks (“scibert-scivocab-cased”) (Beltagy et al., 2019).”
Zhang, S., Li, X., Liu, Y., Bi, P., & Hu, T. (2025) · Symmetry · doi:10.3390/sym17111793 · Cited by 2
Pretraining and evaluation of BERT models for climate research
Poleksić, A., & Martinčić-Ipšić, S. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07740-5 · Cited by 2
“Beltagy et al. [5] use the original BERT BASE architecture from Devlin et al. [6] to train the SciBERT model.”
A DistilBERT-based hierarchical text classification for traffic analysis
Minh, Q. T., & Thai, D. T. (2025) · International Journal of Cognitive Computing in Engineering · doi:10.1016/j.ijcce.2025.10.008 · Cited by 2
Shang, D., Duan, Y., Bai, P., & Xie, J. (2025) · Scientometrics · doi:10.1007/s11192-025-05472-2 · Cited by 2
Makrinsky, K. I., Shendrikov, V. P., Makhonko, A., Merkushkin, D., & Batishchev, O. V. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7040152 · Cited by 2
“The emergence of self-attention mechanisms and the subsequent development of large language models (LLMs) have revolutionized our ability to process and extract information from scientific literature [10–14].”
EchoGraph system for automated quality assessment of echocardiography reports
Chao, C., Delbrouck, J.-B., Asadi, M., Banerjee, I., Farina, J., Galasso, F. I., Mahmoud, A., Abbas, M. T., Wang, Y.-C., Arsanjani, R., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02140-w · Cited by 4
Bharath, M., Raman, R. K. S., & Alankar, A. (2025) · Progress in Materials Science · doi:10.1016/j.pmatsci.2025.101636 · Cited by 6
Extractive versus Generative Language Models for Political Conflict Text Classification
Brandt, P. T., Alsarra, S., D'Orazio, V. J., Heintze, D., Khan, L., Meher, S., Osorio, J., & Sianan, M. (2025) · Political Analysis · doi:10.1017/pan.2025.10027 · Cited by 2
Azmi, R., Chenal, J., Diop, E. B., Abdem, S. A. E., Bounabi, M., Hlal, M., Adraoui, M., & Chafiq, T. (2026) · City and Environment Interactions · doi:10.1016/j.cacint.2026.100296 · Cited by 2
A citation recommendation model employing knowledge graph embedding
ali, Z., Qi, G., Hussain, S., Ullah, I., Khalid, S., Mohammed, A. A. Q., Ullah, I., Malik, A., & Kefalas, P. (2026) · Soft Computing · doi:10.1007/s00500-025-10928-x · Cited by 3
Recommending university-industry collaboration: a topic-institution graph-based solution
Huang, L., Cao, X., Ren, H., Wang, G., & Wang, Y. (2026) · Scientometrics · doi:10.1007/s11192-025-05516-7 · Cited by 2
PharmaMap-LLM: Fine-Tuning Large Language Models for Drug-Drug Interaction (DDI) Analysis
Raveendran, K., Sumanathilaka, D. K., Thiruthanigesan, K., Retnakumar, P., & Sivapiragasam, Y. (2025) · International Conference on Advancements in Computing (ICAC) · doi:10.1109/icac69156.2025.11361526 · Cited by 4
Yengejeh, A. A., Tang, L., Bridge, C. M., & Kundu, C. (2026) · Informatics · doi:10.3390/informatics13020024 · Cited by 2
Handling Publication Imbalance for Effective Community Detection in Scholarly Networks
Noor, M. A., Sheppard, J., & Clark, J. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-13821-7_32 · Cited by 2
MathMex-V2: A Large Language Model Enabled Math Search Engine
Durepos, C., McLaughlin, I., Lund, C., Sienbenmorgen, A., Largey, N., Pitcairn, A., & Mansouri, B. (2025) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl67857.2025.00063 · Cited by 2
“To extract content with mathematical definitions, MathMex-V1 used a fine-tuned SciBERT [6] model as a binary classifier at sentence level to decide if there is a definition in a sentence or not.”
CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis
Obadage, R. R., Rajtmajer, S. M., & Wu, J. (2025) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl67857.2025.00013 · Cited by 2
FakeZero: Real-Time, Privacy-Preserving Misinformation Detection for Facebook and X
Essahli, S., Sarsar, O., Fouad, I., Bentajer, A., & Motii, A. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00235 · Cited by 2
“Emoji are converted to textual 1 TruthSeeker 2023 ground-truth corpus [23].”
Closing the acronym gap in educational retrieval with ontology-guided augmentation
Alier, M., García‐Peñalvo, F. J., Casañ, M. J., Pereira, J., & Granados, A. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123181 · Cited by 2
Ruas, P., Conceição, S. I. R., & Couto, F. M. (2026) · ACM Computing Surveys · doi:10.1145/3796222 · Cited by 2
NeuroConText: Contrastive learning for neuroscience meta-analysis with rich text representation
Ghayem, F., Meudec, R., Dockès, J., Thirion, B., & Wassermann, D. (2026) · Imaging Neuroscience · doi:10.1162/imag.a.1162 · Cited by 3
Dannenfelser, R., Zhong, J., Zhang, R., & Yao, V. (2023) · neural information processing systems · doi:10.52202/075280-0481 · Cited by 4
A Modular Framework for Automated Hypothesis Validation and Refinement in Scientific Research
Chen, C., Masuda, T., Hirakawa, T., Yamashita, T., & Fujiyoshi, H. (2026) · Information · doi:10.3390/info17030244 · Cited by 2
Amamoto, Y., & Ohnishi, T. (2026) · Materials Today Communications · doi:10.1016/j.mtcomm.2026.114945 · Cited by 2
Rostam, Z. R. K., & Kertész, G. (2026) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami68106.2026.11420295 · Cited by 2
“Beltagy et al. [12] introduce SciBERT, a domain-specific PLM specifically designed for scientific text and trained on large scientific corpora.”
Li, N., Kang, B., & Bie, T. D. (2026) · Machine Learning · doi:10.1007/s10994-025-06993-w · Cited by 2
“A surprising new state of the art with a ‘mismatched’ cross-domain LLM In a surprising cross-domain application, using RoBERTa-PM (a medical domain LLM) for PLM-ICD on USPTO2M-632 (a patent dataset) improves performance (80.73/54.45 to 82.83/55.92) even more than using SciBERT (a scientific domain LLM), surpassing the…”
InfluentialFine-tuning a cancer hallmarks classification model using large language model-synthesized resources
İnan, E. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114702 · Cited by 2
Sustainable Materials Design With Multi‐Modal Artificial Intelligence
Xu, T., Wei, T., Ge, Y., Peng, B., Li, Y., Wang, M., Wen, P., Yang, C., & Wei, Y. (2026) · Advanced Science · doi:10.1002/advs.202524273 · Cited by 7
“…is large-scale knowledge extraction; domain-specific models like MatSciBERT and MatBERT significantly outperform general LLMs in mining material properties, synthesis parameters, and latent relationships from literature to construct structured, machine-readable databases [ 47, 83, 147–152 ].”
Predicting the impact of scientific articles through semantic analysis of abstracts
Süzen, N., Gorban, A. N., Levesley, J., & Mirkes, E. M. (2026) · Scientometrics · doi:10.1007/s11192-026-05647-5 · Cited by 5
Ricci, P., Luca, M. D., Ferraro, G., & Russo, A. (2025) · Pharmacophore · doi:10.51847/8pe7suiyyy · Cited by 2
Large Language Models for Pharmaceutical Knowledge Management: A Critical Review
Torres, A., & Fernandez, M. (2025) · Pharmacophore · doi:10.51847/xphozuyfsy · Cited by 2
Librarian of Alexandria: A Modular Chemical Data Extraction Pipeline to Compare LLM Performance
Grougan, M., Subasinghe, J., Hix, M. A., & Walker, A. R. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c00374 · Cited by 3
Кузнецов, И. И. (2026) · Modelirovanie, optimizaciâ i informacionnye tehnologii. · doi:10.26102/2310-6018/2026.57.6.020 · Cited by 2
NLP-PINGAN-TECH @ CL-SciSumm 2020
Chai, L., Fu, G., & Ni, Y. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.26 · Cited by 2
Injecting Event Knowledge into Pre-Trained Language Models for Event Extraction
Yang, Z., Zhan, S., Hou, M., Zeng, X., & Zhu, H. (2020) · doi:10.5121/csit.2020.101404 · Cited by 2
Domain-Transferable Method for Named Entity Recognition Task
Mikhailov, V., & Shavrina, T. (2020) · doi:10.5121/csit.2020.101407 · Cited by 2
“SciBERT [5] and BioBERT [6] achieved significant improvement over original BERT [7] on a number of downstream tasks within the domains.”
BackgroundCoordination Boundary Identification without Labeled Data for Compound Terms Disambiguation
Sawada, Y., Wada, T., Shibahara, T., Teranishi, H., Kondo, S., Shindo, H., Watanabe, T., & Matsumoto, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.271 · Cited by 2
“ELMo performs better than SciBERT, and we conjecture that this would be because ELMo employs character-level CNN to encode words and works well for aligning biological terminologies that have similar character strings.”
MethodInfluentialIwatsuki, K., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.304 · Cited by 2
Metric-Type Identification for Multi-Level Header Numerical Tables in Scientific Papers
Suadaa, L. H., Kamigaito, H., Okumura, M., & Takamura, H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.267 · Cited by 2
Kortum, H., Leimkühler, M., & Thomas, O. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86797-3_3 · Cited by 2
Kumar, V. B., Yarramsetty, M., Sun, S., & Goel, A. (2021) · Workshop on e-Commerce and NLP (ECNLP) · doi:10.18653/v1/2021.ecnlp-1.20 · Cited by 2
Hammer PDF: An Intelligent PDF Reader for Scientific Papers
Wang, S., Liu, S.-H., Che, T.-Y., Lu, Y.-F., Yang, S.-X., Huang, H., & Mao, X.-L. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557169 · Cited by 3
“Apart from basic reading functions, Hammer PDF has the following four innovative features: (1) information extraction ability, which can locate and mark spans like terms and other entities; (2) information extension ability, which can present relevant academic content of a paper, such as citations, references, codes,…”
BackgroundOpening scholarly documents through text analytics
Banerjee, B. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530948 · Cited by 2
“Language models like BERT [4] and SciBERT [1] employ transformer-based mechanisms that have been proven to perform well on a wide variety of natural language processing tasks.”
BackgroundMethodInfluentialAttention Weight is Indispensable in Joint Entity and Relation Extraction
Ouyang, J., Zhang, J., & Liu, T. (2022) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2022.028352 · Cited by 2
Science gateway adoption using plug‐in middleware for evidence‐based healthcare data management
Oruche, R., Milman, E. D., Alarcon, M. L., Cheng, X., Pandey, A., Wang, S., Calyam, P. P., & Kee, K. F. (2022) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.7195 · Cited by 4
Faramarzi, N. S., Dara, A., & Banerjee, R. (2022) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi54592.2022.00023 · Cited by 2
“With the exception of SciBERT, which is trained on research articles from multiple scientific disciplines, the other models we use are specific to the medical domain.”
MethodInfluentialRoberta Goes for IPO: Prospectus Analysis with Language Models for Indian Initial Public Offerings
Mishra, A., & Sisodia, Y. (2022) · doi:10.5121/csit.2022.121905 · Cited by 2
“SCIBERT was tested on a wide range of scientific domain-specific tasks and datasets.”
BackgroundInfluentialRamachandran, R., Ramasubramanian, M., Koirala, P., Gurung, I., & Maskey, M. (2022) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss46834.2022.9883682 · Cited by 2
Ten Years of Living Machines Conferences: Transformers-Based Automated Topic Grouping
Carniel, T., Cazenille, L., Dalle, J., & Halloy, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20470-8_2 · Cited by 2
Extracting Information about Research Resources from Scholarly Papers
Saji, A., & Matsubara, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21756-2_35 · Cited by 2
Kong, H., Yoon, H.-M., Seol, J.-W., Hyun, M.-H., Lee, H., Kim, S., & Choi, W. (2022) · IEEE Access · doi:10.1109/access.2022.3233228 · Cited by 2
Graph Embedding for Mapping Interdisciplinary Research Networks
Cunningham, E., & Greene, D. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587570 · Cited by 5
“In the case of each paper, we encode the concatenated title and abstract using the pre-trained SciBERT model and use the final representation of the [CLS] token as an embedding to represent each document (similar to the approach in [4]).”
MethodInfluentialSahijwani, H., Dhole, K., Purwar, A., Vasudevan, V., & Agichtein, E. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587657 · Cited by 2
“There has been extensive prior work on closely related problems like intent prediction and slot-flling for conversational systems [15– 17], dialog representation [12, 14], knowledge grounded language models [20], and domain-specifc language models [2].”
BackgroundArgument Mining on Clinical Trial Abstracts on Lung Cancer Patients
Khondoker, M. Y. A., & Yousuf, M. A. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-19-9483-8_5 · Cited by 2
AI for Cybersecurity: Robust models for Authentication, Threat and Anomaly Detection
[Olga Tushkanova, E. A. ]., Bergadano, F., & Giacinto, G. (2023) · MDPI eBooks · doi:10.3390/books978-3-0365-8265-8 · Cited by 4
PGLR: Pseudo Graph and Label Reuse for Entity Relation Extraction
Zhang, H., Li, L., & Shen, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191693 · Cited by 3
“SciBERT [37] is utilized as the base encoder for SciERC.”
MethodBioNART: A Biomedical Non-AutoRegressive Transformer for Natural Language Generation
Asada, M., & Miwa, M. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.34 · Cited by 2
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.