SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 801–900 of 3,596
Zhou, M., Tan, J., Song, Y., Wang, H., Wang, L., & Xiao, Z. (2023) · IEEE Access · doi:10.1109/access.2023.3299824 · Cited by 19
Value creation in emerging technologies through text mining: the case of blockchain
Chiarello, F., Belingheri, P., Bonaccorsi, A. J., Fantoni, G., & Martini, A. (2021) · Technology Analysis and Strategic Management · doi:10.1080/09537325.2021.1876221 · Cited by 19
“In the past years, transformer architectures and embedding language models such as BERT (Devlin et al. 2018) and SciBERT (Beltagy et al., 2019) have revolutionised NLP, evolving from standard word-embedding approaches (Pennington, Socher, and Manning 2014; Mikolov et al. 2013).”
MethodRybinski, M., Dai, X., Singh, S., Karimi, S., & Nguyen, A. N. (2021) · JMIR Medical Informatics · doi:10.2196/24020 · Cited by 25
Naderi, N., Knafou, J., Copara, J., Ruch, P., & Teodoro, D. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.689803 · Cited by 27
“Deep learning approaches trained on large unstructured corpora have shown considerable success in NLP problems, including NER (Lample et al., 2016; Beltagy et al., 2019; Devlin et al., 2019; Jin et al., 2019; Liu et al., 2019; Andrioli de Souza et al., 2020).”
BackgroundMethodSPR-SMN: scientific paper recommendation employing SPECTER with memory network
Ali, Z., Qi, G., Kefalas, P., Khusro, S., Khan, I. U., & Muhammad, K. (2022) · Scientometrics · doi:10.1007/s11192-022-04425-3 · Cited by 33
“Yet, content-embedding approaches such as SBERT and BERT are trained on general English corpus, which in contrast to domain-specific embeddings models like SciBERT (Beltagy et al., 2019) and SPECTER (Cohan et al., 2020), cannot learn context-preserving node embeddings (Ali et al., 2021b).”
MethodInfluentialD’Souza, J., Hoppe, A., Brack, A., Jaradeh, M. Y., Auer, S., & Ewerth, R. (2020) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/16291 · Cited by 36
Zhang, R., Zhang, J., Chen, Q., Wang, B., Liu, Y., Qian, Q., Pan, D., Xia, J., Wang, Y., & Han, Y. (2023) · Computational Materials Science · doi:10.1016/j.commatsci.2023.112441 · Cited by 24
Healthcare: A Growing Role for Large Language Models and Generative AI
Pahune, S. A., & Rewatkar, N. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.55573 · Cited by 24
Rogers, B., Noman, N., Chalup, S. K., & Moscato, P. (2023) · Evolutionary Intelligence · doi:10.1007/s12065-023-00874-8 · Cited by 20
Raval, S., Sedghamiz, H., Santus, E., Alhanai, T., Ghassemi, M. M., & Chersoni, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.300 · Cited by 25
An Extensive Study on Pretrained Models for Natural Language Processing Based on Transformers
Ramprasath, M., Dhanasekaran, K., Karthick, T., Velumani, R., & Sudhakaran, P. (2022) · International Conference on Electronics and Renewable Systems (ICEARS) · doi:10.1109/icears53579.2022.9752241 · Cited by 20
“[32] to be a viable candidate for the study.”
BackgroundTen questions concerning Large Language Models (LLMs) for building applications
Ma, N., Labib, R., Amor, R., Chong, A., Fan, C., Forth, K., Fu, X., Fuchs, S., Hong, T., Klimenkova, N., et al. (2026) · Building and Environment · doi:10.1016/j.buildenv.2026.114260 · Cited by 31
Argumentation Mining in Scientific Literature for Sustainable Development
Fergadis, A., Pappas, D., Karamolegkou, A., & Papageorgiou, H. (2021) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/2021.argmining-1.10 · Cited by 19
SpaceTransformers: Language Modeling for Space Systems
Berquand, A., Darm, P., & Riccardi, A. (2021) · IEEE Access · doi:10.1109/access.2021.3115659 · Cited by 28
Pawar, D., Phansalkar, S., Sharma, A., Sahu, G. K., Ang, C. K., & Lim, W. H. (2023) · Sustainability · doi:10.3390/su15054216 · Cited by 25
Lo, K., Shen, Z., Newman, B., Chang, J., Authur, R., Bransom, E., Candra, S., Chandrasekhar, Y., Huff, R., Kuehl, B., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.45 · Cited by 36
Self-citationBACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation
Ge, Y., Dinh, L., Liu, X., Su, J., Lu, Z., Wang, A., & Diesner, J. (2021) · doi:10.18653/v1/2021.acl-long.116 · Cited by 53
Otto, W., Zloch, M., Gan, L., Karmakar, S., & Dietze, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.548 · Cited by 35
Mercier, D., Rizvi, S., Rajashekar, V., Dengel, A., & Ahmed, S. (2021) · Proceedings of the 13th International Conference on Agents and Artificial Intelligence · doi:10.5220/0010235201590168 · Cited by 19
“(Beltagy et al., 2019) proposed SciBERT, which is a variation of BERT optimized for scientific publications and trained on 1.”
Building an end-to-end battery recipe knowledge base via transformer-based text mining
Lee, D., Mizuseki, H., Choi, J., & Lee, B. (2025) · Communications Materials · doi:10.1038/s43246-025-00825-z · Cited by 20
“To investigate this effect, we evaluated fourpre-trainedlanguagemodels — BERT 48 ,SciBERT 50 ,BatteryBERT 15 ,and MatBERT 51 — by comparing theirNER performanceinterms of F1 score,as summarizedin Supplementary Tables5 and 6.”
Query Expansion in the Age of Pre-trained and Large Language Models: A Comprehensive Survey
Li, M., Lv, X., Zou, J., Chen, T., Zhang, C., An, S., Nie, E., & Zhou, G. (2026) · ACM Transactions on Information Systems · doi:10.1145/3816248 · Cited by 24
“Hosted LLM services accessed via APIs (e.g., ChatGPT-or Gemini-style [107] systems) make prompt-only QE easy to prototype and deploy, but raise practical concerns such as latency/cost budgets, limited transparency, and version drift.”
TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration
Jansen, P. A., Thayaparan, M., Valentino, M., & Ustalov, D. (2021) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/2021.textgraphs-1.17 · Cited by 27
Moradi, M., & Samwald, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.12678 · Cited by 22
Automated Identification of Social Media Bots Using Deepfake Text Detection
Saravani, S. M., Ray, I., & Ray, I. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-92571-0_7 · Cited by 18
Lipford, M. C., Ip, W., Awasthi, S., Moore, J. L., Tippmann‐Peikert, M., Asfahan, S., Kumar, P. M., Singh, G. P., & Gudeman, J. (2024) · SLEEP Advances · doi:10.1093/sleepadvances/zpae067 · Cited by 19
“These sentences were then labeled and used to fine-tune the pretrained SciBERT [8] sentence classification model to output one of the following labels: yes (confirmed diagnosis), no (ruled-out diagnosis), maybe (under consideration for differential diagnosis), and other (family history or…”
MethodWhitton, J., & Hunter, A. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102661 · Cited by 20
Adversarial Modality Alignment Network for Cross-Modal Molecule Retrieval
Zhao, W., Zhou, D. M., Cao, B., Zhang, K., & Chen, J. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3254518 · Cited by 26
“Our method utilizes a SciBERT [5] as the text encoder and a graph transformer network (GTN) [6] as the molecule encoder to generate text and molecule representations.”
MethodBioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity RecognitionPreprint
Naseem, U., Khushi, M., Reddy, V. L., Rajendran, S., Razzak, I., & Kim, J. (2020) · doi:10.21203/rs.3.rs-90025/v1 · Cited by 18
CrossNER: Evaluating Cross-Domain Named Entity RecognitionPreprint
Liu, Z., Xu, Y., Yu, T., Dai, W., Ji, Z., Cahyawijaya, S., Madotto, A., & Fung, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04373 · Cited by 18
“In addition, we collect the corresponding five unlabeled domain-related corpora for the domain-adaptive pre-training, given its effectiveness for domain adaptation (Beltagy, Lo, and Cohan 2019; Donahue et al. 2019; Lee et al. 2020; Gururangan et al. 2020).”
MethodBidirectional Language Modeling: A Systematic Literature Review
Jahan, M. S., Khan, H. U., Akbar, S., Farooq, M. U., Gul, S., & Amjad, A. (2021) · Scientific Programming · doi:10.1155/2021/6641832 · Cited by 24
“, [22] business [23], medical [24, 25], and science [26].”
BackgroundGraphCite: Citation Intent Classification in Scientific Publications via Graph Embeddings
Berrebbi, D., Huynh, N., & Balalau, O. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524657 · Cited by 18
“SciBert is generally finetuned on the downstream task to achieve optimal performance.”
BackgroundMethodInfluentialFlamholz, Z., Crane‐Droesch, A., Ungar, L., & Weissman, G. E. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103971 · Cited by 23
Frei, J., & Krämer, F. (2022) · JMIR Formative Research · doi:10.2196/39077 · Cited by 22
Identifying and Visualizing Trends in Science, Technology, and Innovation Using SciBERT
Лобанова, П., Денисович, Б. П., & Sergienko, Y. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2023.3306569 · Cited by 18
Enhancing local citation recommendation with recurrent highway networks and SciBERT-based embedding
Dinh, T. N., Pham, P., Giang, N. L., & Vo, B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122911 · Cited by 19
Wang, Z., Zhang, H., Chen, H., Feng, Y., & Ding, J. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102119 · Cited by 21
“SPECTER (Cohan et al., 2020) is based on the SciBERT model and further trained on a corpus that includes citation relationships.”
InfluentialClassifying Math Knowledge Components via Task-Adaptive Pre-Trained BERT
Shen, J., Yamashita, M., Prihar, E., Heffernan, N. T., Wu, X., McGrew, S., & Lee, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78292-4_33 · Cited by 29
Clinical Flair: A Pre-Trained Language Model for Spanish Clinical Natural Language Processing
Rojas, M. S., Dunstan, J., & Villena, F. (2022) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2022.clinicalnlp-1.9 · Cited by 18
Zhang, C., Xiang, Y., Hao, W., Li, Z., Qian, Y., & Wang, Y. Z. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2022.101373 · Cited by 20
Foundation models of protein sequences: A brief overview.
Bjerregaard, A., Groth, P. M., Hauberg, S., Krogh, A., & Boomsma, W. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103004 · Cited by 18
A Survey of Large Language Model for Drug Research and Development
Guo, H., Xing, X., Zhou, Y., Jiang, W., Chen, X., Wang, T., Jiang, Z., Wang, Y., Hou, J., Jiang, Y., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3552256 · Cited by 24
Domain Identification of Scientific Articles Using Transfer Learning and Ensembles
Hande, A., Puranik, K., Priyadharshini, R., & Chakravarthi, B. R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-75015-2_9 · Cited by 22
Improving the robustness and accuracy of biomedical language models through adversarial training
Moradi, M., & Samwald, M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104114 · Cited by 18
Ameri, K., Hempel, M., Sharif, H., Lopez, J., & Perumalla, K. S. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3546580 · Cited by 21
“For pre-training language models in specific domain, researchers focused on domain-specific corpus to build a specific language model such as BioBERT [48] and SciBERT [9], for biomedical language representation and scientific text, respectively.”
MethodDeep learning for journal recommendation system of research papers
Gündoğan, E., Kaya, M., & Daud, A. Y. (2022) · Scientometrics · doi:10.1007/s11192-022-04535-y · Cited by 23
“However, there are also domain-based SPECTER (Cohan et al., 2020) and SciBERT (Beltagy et al., 2019) models trained with scientific documents.”
BackgroundChambon, P., Cook, T. S., & Langlotz, C. P. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00714-8 · Cited by 27
“Originally designed to help fine-tune LSTM models, we can expect these methods among others [7, 19–21] to show similar improvements on the fine-tuning of BERT-like models.”
MethodEXSCLAIM!: Harnessing materials science literature for self-labeled microscopy datasets
Schwenker, E., Jiang, W., Spreadbury, T., Ferrier, N., Cossairt, O., & Chan, M. K. Y. (2023) · Patterns · doi:10.1016/j.patter.2023.100843 · Cited by 19
Topic-Centric Unsupervised Multi-Document Summarization of Scientific and News Articles
Alambo, A., Lohstroh, C., Madaus, E., Padhee, S., Foster, B., Banerjee, T., Thirunarayan, K., & Raymer, M. L. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378403 · Cited by 24
Chang, D., Lin, E. C., Brandt, C., & Taylor, R. A. (2021) · JMIR Medical Informatics · doi:10.2196/23101 · Cited by 21
Wang, P., Hu, Q., Mei, Q., Yang, Y., Yang, Y., Guo, D., Liu, X., Hu, W., & Chen, J. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103633 · Cited by 19
Weinzierl, M., & Harabagiu, S. M. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512039 · Cited by 25
“process of further pre-training has been shown to improve performance on downstream tasks in various scientific [4], biomedical [14], and social media [23] domains.”
BackgroundKia, M. A., Garifullina, A., Kern, M., Chamberlain, J., & Jameel, S. (2022) · IEEE Access · doi:10.1109/access.2022.3170466 · Cited by 18
A hybrid human–AI tool for scientometric analysis
Correia, A., Grover, A., Jameel, S., Schneider, D., Antunes, P. A., & Fonseca, B. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10548-7 · Cited by 18
“For example, transformer-based large language models (LLMs) such as SciBERT (Beltagy et al. 2019) and GPT-3 (Floridi and Chiriatti 2020) have gained a lot of traction in the last few years due to their modelling and predictive capabilities.”
Lefebvre, G., Elghazel, H., Guillet, T., Aussem, A., & Sonnati, M. O. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102281 · Cited by 18
Deep Fuzzy Multiteacher Distillation Network for Medical Visual Question Answering
Liu, Y., Chen, B., Wang, S., Lu, G., & Zhang, Z. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3402086 · Cited by 19
Nayak, A., Timmapathini, H. P., Murali, V., Ponnalagu, K., Venkoparao, V. G., & Post, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98464-9_8 · Cited by 18
Coreference Resolution for the Biomedical Domain: A Survey
Lu, P., & Poesio, M. (2021) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/2021.crac-1.2 · Cited by 22
Hanss, K. E., Sarma, K. V., Glowinski, A. L., Krystal, A. D., Saunders, R., Halls, A. J. M., Gorrell, S., & Reilly, E. E. (2025) · Journal of Medical Internet Research · doi:10.2196/69910 · Cited by 25
“Other studies have consistently shown that LLMs may misinterpret specialized terminology (eg, “egosyntonic”) within domain-specific text [16,17].”
Recent Advances in CO (2) Electroreduction Driven by Artificial Intelligence and Machine Learning
Xue, Y., Zhang, L., & Zheng, G. (2025) · Advanced Energy Materials · doi:10.1002/aenm.202503560 · Cited by 18
Learning Representations on Logs for AIOps
Gupta, P., Kumar, H., Kar, D., Bhukar, K., Aggarwal, P., & Mohapatra, P. (2023) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud60044.2023.00026 · Cited by 35
“Inspired by the LLMs for different domains, such as Tweet-BERT [15], SciBERT [16], and BioBERT [11], we introduce BERTOps , an LLM for AI in operations (AIOps) domain, pre-trained over large-scale public and proprietary log data.”
MethodThe overview of the BioRED (Biomedical Relation Extraction Dataset) track at BioCreative VIII
Islamaj, R., Lai, P.-T., Wei, C.-H., Luo, L., de Almeida, T. D., Jonker, R. A. A., Conceição, S. I. R., Sousa, D., Phan, C.-P., Chiang, J.-H., et al. (2024) · Database · doi:10.1093/database/baae069 · Cited by 26
“We used the pretrained allenai/scibert_ scivocab_uncased SciBERT [38] model and independently fine-tuned it for the association and novelty labels.”
MethodA Hybrid System Based on Bayesian Networks and Deep Learning for Explainable Mental Health Diagnosis
Pavez, J., & Allende, H. (2024) · Applied Sciences · doi:10.3390/app14188283 · Cited by 21
Li, C., Zheng, Z., Du, X., Ma, X., Wang, Z., & Li, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112588 · Cited by 19
Triple Classification for Scholarly Knowledge Graph Completion
Jaradeh, M. Y., Singh, K., Stocker, M., & Auer, S. (2021) · International Conference on Knowledge Capture · doi:10.1145/3460210.3493582 · Cited by 24
“Our rationale for the choice is as follows: Language models such as SciBERT [6] are already trained in an unsupervised manner on large corpora of scholarly literature.”
MethodInfluentialCross-Domain Limitations of Neural Models on Biomedical Relation Classification
Alimova, I., Tutubalina, E. V., & Nikolenko, S. (2021) · IEEE Access · doi:10.1109/access.2021.3135381 · Cited by 18
Charting mobility patterns in the scientific knowledge landscape
Singh, C. K., Tupikina, L., Lécuyer, F., Starnini, M., & Santolini, M. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00451-8 · Cited by 21
“Future work should explore the reproducibility of our findings across larger and diverse datasets, leveraging other field identification methods, such as the ones using Natural Language Processing [63].”
BackgroundLopez, P., Du, C., Cohoon, J., Ram, K., & Howison, J. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481936 · Cited by 19
“07 points in F1-score depending on the task [3].”
BackgroundMethodVillegas-Ch, W. E., Gutierrez, R., García-Ortiz, J., & Guevara, V. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00438-y · Cited by 22
arXivEdits: Understanding the Human Revision Process in Scientific Writing
Jiang, C., Xu, W., & Stevens, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.641 · Cited by 18
CitationIE: Leveraging the Citation Graph for Scientific Information Extraction
Viswanathan, V., Neubig, G., & Liu, P. (2021) · doi:10.18653/v1/2021.acl-long.59 · Cited by 37
“First, contextualized word embeddings are obtained for each section by running SciBERT (Beltagy et al., 2019) on that section of text (up to 512 tokens).”
MethodPrincipe, R. A. A., Chiarini, N., & Viviani, M. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70019 · Cited by 26
“In Hu et al. (2021), section-wise chunking and Sci BERT (Beltagy et al. 2019) are utilized for obtaining chunk embed-dings, then n layers of custom Transformers are employed for sections interaction.”
Large Language Models for Market Research: A Data-Augmentation Approach
Wang, M., Zhang, D., & Zhang, H. (2026) · Marketing Science · doi:10.1287/mksc.2025.0009 · Cited by 37
DDI-MuG: Multi-aspect graphs for drug-drug interaction extraction
Yang, J., Ding, Y., Long, S., Poon, J., & Han, S. C. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1154133 · Cited by 18
“As shown in Table 4, the four bio-specific models, i.e., BioBERT, SciBERT, ouBioBERT(Wada et al., 2020), and PubMedBERT, leading to improvement over standard BERT.”
BackgroundText-guided small molecule generation via diffusion model
Luo, Y., Fang, J., Li, S., Liu, Z., Wu, J., Zhang, A., Du, W., & Wang, X. (2024) · iScience · doi:10.1016/j.isci.2024.110992 · Cited by 32
AdditiveLLM: Large language models predict defects in metals additive manufacturing
Pak, P., & Farimani, A. B. (2025) · Additive Manufacturing Letters · doi:10.1016/j.addlet.2025.100292 · Cited by 18
“33 With a dataset of over 1.14 million papers, the authors are able to pretrain the BERT implementation on a corpus of text with the use of domain specific vocabulary allowing better representation of scientific terms and symbols through their SciVocab vocabulary library.”
InfluentialAdapting a Language Model While Preserving its General Knowledge
Ke, Z., Shao, Y., Lin, H., Hu, X., Shu, L., & Liu, B. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.693 · Cited by 34
Huang, C.-Y., Hsu, T.-Y., Rossi, R. A., Nenkova, A., Kim, S., Chan, G. Y.-Y., Koh, E., Giles, C. L., & Huang, T.-H. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.6 · Cited by 28
Dipnall, J. F., Lu, J., Gabbe, B. J., Cosic, F., Edwards, E. R., Page, R. S., & Du, L. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110366 · Cited by 17
Aldahdooh, J., Tanoli, Z., & Tang, J. P. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae106 · Cited by 19
“The preprocessed data is then fed into an ensemble of pre-trained transformer models, including RoBERTa, BioBERT, SciBERT, and BioLinkBERT, to predict drug–target interactions.”
MethodInfluentialJoint Summarization-Entailment Optimization for Consumer Health Question Understanding
Mrini, K., Dernoncourt, F., Chang, W. H., Farcas, E., & Nakashole, N. (2021) · doi:10.18653/v1/2021.nlpmc-1.8 · Cited by 25
“, 2019) has been fine-tuned using biomedical text from PubMed (Beltagy et al., 2019), PMC (Lee et al.”
MethodRietberg, M. T., Nguyen, V. B., Geerdink, J., Vijlbrief, O., & Seifert, C. (2023) · Diagnostics · doi:10.3390/diagnostics13071251 · Cited by 17
“Domain-specific BERT models can have significant benefits [23,24], but training from scratch requires a large amount of data and computational resources.”
BackgroundVNLawBERT: A Vietnamese Legal Answer Selection Approach Using BERT Language Model
Chau, C.-N., Nguyen, T.-S., & Nguyen, L.-M. (2020) · NAFOSTED Conference on Information and Computer Science (NICS) · doi:10.1109/nics51282.2020.9335906 · Cited by 17
“SciBERT[9], BioBERT[10], ClinicalBERT[11], FinBERT[12] are examples, they all performed better than the original BERT in a domain-specific task.”
MethodRapid Adaptation of Chemical Named Entity Recognition Using Few-Shot Learning and LLM Distillation
Zhang, Y., Vlachos, D. G., Liu, D., & Fang, H. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00248 · Cited by 19
Graph-based technology recommendation system using GAT-NGCF
Kim, M.-S., Jang, Y., & Sung, T.-E. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128240 · Cited by 18
Knafou, J., Naderi, N., Copara, J., Teodoro, D., & Ruch, P. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.40 · Cited by 19
Unraveling Scientific Evolutionary Paths: An Embedding-Based Topic Analysis
Jin, Q., Chen, H., Zhang, Y., Wang, X., & Zhu, D. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2023.3312923 · Cited by 17
Panoutsopoulos, H., Espejo-García, B., Raaijmakers, S., Wang, X., Fountas, S., & Brewster, C. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109268 · Cited by 19
Classifying Scientific Publications with BERT - Is Self-attention a Feature Selection Method?
García-Silva, A., & Gómez-Pérez, J. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_11 · Cited by 19
CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models
Su, Y., Han, X., Lin, Y., Zhang, Z., Liu, Z., Li, P., Zhou, J., & Sun, M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3105013 · Cited by 19
“For relation extraction, we select SciERC [49] and ChemProt [50] for our experiments.”
MethodIdentifying Topics of Scientific Articles with BERT-Based Approaches and Topic Modeling
Glazkova, A. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-75015-2_10 · Cited by 17
AnomalyAdapters: Parameter-Efficient Multi-Anomaly Task Detection
Ünal, U., & Dağ, H. (2022) · IEEE Access · doi:10.1109/access.2022.3141161 · Cited by 21
CS-KG 2.0: A Large-scale Knowledge Graph of Computer Science
Dessí, D., Osborne, F., Buscaldi, D., Reforgiato Recupero, D., & Motta, E. (2025) · Scientific Data · doi:10.1038/s41597-025-05200-8 · Cited by 17
Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training
Xu, R., Yu, Y., Ho, J. C., & Yang, C. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592085 · Cited by 26
“In contrast, X-Class and LOTClass, which achieve strong results on general-domain tasks, 20 40 60 80 M ac ro F 1 MeSH 20 30 40 50 60 arXiv-CS SciBERT BM25 Contriever WANDER (a) Study of Retrievers 25 50 100 150 200 60 65 70 75 80 85 M ac ro F 1 Best baseline (MeSH) 50 55 60 65 70 75 Best baseline (arXiv-CS) MeSH arXiv…”
BackgroundMethodInfluentialComputer Science Named Entity Recognition in the Open Research Knowledge Graph
D’Souza, J., & Auer, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21756-2_3 · Cited by 27
“However, present advanced embedding models based on transformers such as BERT [20] and its variants [6] also allow finetuning the pretrained embeddings on respective experimental datasets.”
Task-Oriented ML/DL Library Recommendation Based on a Knowledge Graph
Liu, M., Zhao, C., Peng, X., Yu, S. S., Wang, H., & Sha, C. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3285280 · Cited by 24
“We implement the model based on PyTorch 1.8.1 and use SciBERT [30] for the BERT layer, which is pretrained on scientific text from papers and achieves the SOTA performance on a wide range of NLP tasks in the scientific domain.”
MethodInfluentialShen, K., & Kejriwal, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0295925 · Cited by 17
Collaborative Knowledge Infusion for Low-Resource Stance Detection
Yan, M., Joey, T. Z., & Ivor, W. T. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020021 · Cited by 23
“Another knowledge infusion paradigm is that finetune PLMs on the specific domain corpus to embed the domain-specific knowledge, as demonstrated by Sci-BERT [17], Bio-BERT [18], BERTweet [19].”
BackgroundUnsupervised Paraphrasing Consistency Training for Low Resource Named Entity Recognition
Wang, R., & Henao, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.430 · Cited by 23
Unsupervised Event Graph Representation and Similarity Learning on Biomedical Literature
Frisoni, G., Moro, G., Carlassare, G., & Carbonaro, A. (2021) · Sensors · doi:10.3390/s22010003 · Cited by 21
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.