SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,101–2,200 of 3,596
Stars, Stripes, and Silicon: Unravelling the ChatGPT’s All-American, Monochrome, Cis-centric Bias
Torrielli, F. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-74630-7_19 · Cited by 4
“As of 2023, cognitive science has identified innumerable cognitive biases [7,8,32,47,55] which can lead to flawed reasoning, irrationality, and potentially harmful consequences [49].”
Multimodal Paper Summarization with Hierarchical Fusion
Tan, Z., Zhong, X., & Chiu, B. (2024) · International Conference on Engineering and Emerging Technologies (ICEET) · doi:10.1109/iceet65156.2024.10913899 · Cited by 3
“• Text : The paper’s textual features were encoded using SciBERT [29], which is optimized for scientific content.”
Li, M., Zhuang, Y., Chen, K. G., Han, L., Li, X., wei, Y., Zhu, X., Yang, M., Yin, G., Lin, J., et al. (2025) · Materials Genome Engineering Advances · doi:10.1002/mgea.70001 · Cited by 3
Lipford, M. C., Asfahan, S., Singh, G. P., Moore, J. L., Tippmann‐Peikert, M., Kumar, P. M., Ip, W., Awasthi, S., Gudeman, J., & Krahn, L. E. (2025) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2025.111185 · Cited by 3
“These sentences were then labeled and used to fine tune the pretrained SciBERT (ie, BERT model pretrained on scientific text) [19] sentence classification model to output 1 of the following 4 labels: ‘Yes ’ (confirmed diagnosis), ‘No ’ (ruled out diagnosis), ‘Maybe ’ (under consideration for…”
Habti, W. E., & Azmani, A. (2025) · Applied Computer Science · doi:10.35784/acs_6670 · Cited by 3
“SciBERT, a variant of BERT (Bidirectional Encoder Representations from Transformers) specifically trained on scientific text, was chosen for its potential to effectively perform sentiment analysis within the biomedical domain, surpassing the capabilities of BioBert (Beltagy et al., 2019).”
Ningrum, P. K., Mayr, P., Smirnova, N., & Atanassova, I. (2025) · Journal of Informetrics · doi:10.1016/j.joi.2025.101661 · Cited by 4
Richins, G. (2025) · Accounting and Finance · doi:10.1111/acfi.70036 · Cited by 4
Verkijk, S., & Vossen, P. T. J. M. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103148 · Cited by 7
CodeComClassify: Automating Code Comments Classification using BERT-Based Language Models
Alam, K. A., Ali, W., Aziz, S., Haroon, M., Khan, M. H., Ahmad, Z., & Abbas, N. (2025) · IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) · doi:10.1109/nlbse66842.2025.00017 · Cited by 3
“Based on a thorough analysis of the existing body of knowledge, we considered to experiment with DistilBert [9], RoBERTa-Base [10], TinyBert [11], CodeBert [12], and SciBert [13].”
Large Language Models Evaluation for PubMed Extractive Summarisation
Xia, T., Bertini, F., & Montesi, D. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3766905 · Cited by 3
“Pre-training corpora BERT [13] BookCorpus and English Wikipedia RoBERTa [35] BookCorpus, English Wikipedia, CC-news, OpenWebText, and Stories DistilBERT [55] BookCorpus and English Wikipedia Longformer [7] BookCorpus, English Wikipedia, Realnews, and Stories BART [31] BookCorpus, English Wikipedia, CC-news, OpenWebTex…”
InfluentialSentiment Analysis of Online Customer Reviews Using Modern Natural Language Processing Approaches
Koruyan, K. (2025) · Erzurum Teknik Universitesi Sosyal Bilimler Enstitusu Dergisi · doi:10.29157/etusbed.1711500 · Cited by 3
Barchane, M., & Zahour, O. (2025) · Mathematical Modeling and Computing · doi:10.23939/mmc2025.04.1109 · Cited by 3
A Corpus of Very Short Scientific Summaries
Chen, Y., Polajnar, T., Batchelor, C. R., & Teufel, S. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.12 · Cited by 4
Decentralized Context-Based On-Board Planning for Earth Observation Missions
Martin, A. V., Cheng, K., Fang, A., Zheng, Z., Kress‐Gazit, H., Mehta, A. M., Selva, D., & Sun, Y. (2021) · AIAA SciTech Forum · doi:10.2514/6.2021-1469 · Cited by 5
Misogynous Text Classification Using SVM and LSTM
Devi, M. D., & Saharia, N. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-0401-0_26 · Cited by 4
Kalyan, V., Witteveen, S., & Andrews, M. (2021) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/2021.textgraphs-1.20 · Cited by 3
Predicting the Age of Scientific Papers
Savov, P., Jatowt, A., & Nielek, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77961-0_58 · Cited by 3
ACM-CR: A Manually Annotated Test Collection for Citation Recommendation
Boudin, F. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00035 · Cited by 3
An Exploratory Study on Utilising the Web of Linked Data for Product Data Mining
Zhang, Z., & Song, X. (2022) · SN Computer Science · doi:10.1007/s42979-022-01415-3 · Cited by 3
“While our work is also broadly related to the use of neural networks in creating domainspecific language models, such as the BioBERT [6], Clinical BERT [7], SciBERT[8], E-Bert[9], and SMedBERT[10], these studies did not use semantic markup data and therefore, we do not expand our literature review to this broad area a…”
ResultInfluentialLG4AV: Combining Language Models and Graph Neural Networks for Author Verification
Stubbemann, M., & Stumme, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-01333-1_25 · Cited by 4
Paharia, N., Pozi, M. S. M., & Jatowt, A. (2021) · IEEE Access · doi:10.1109/access.2021.3135051 · Cited by 3
Layout Aware Semantic Element Extraction for Sustainable Science & Technology Decision Support
Kim, H., Choi, J., Park, S., & Jung, Y. (2022) · Sustainability · doi:10.3390/su14052802 · Cited by 4
Contextual embedding and model weighting by fusing domain knowledge on biomedical question answering
Lu, Y., Yan, J., Qi, Z., Ge, Z., & Du, Y. (2022) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3535508.3545508 · Cited by 3
“We use the SciBERT [1], which has been pretrained on the Semantic Scholar corpus.”
BackgroundMethodAssessing Scientific Research Papers with Knowledge Graphs
Sun, K., Qiu, Z., Salinas, A., Huang, Y., Lee, D.-H., Benjamin, D., Morstatter, F., Ren, X., Lerman, K., & Pujara, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531879 · Cited by 7
“We apply SciBERT [7], a BERT-based [12] model pre-trained on scientific text, to encode the descriptions such that”
BackgroundMethodInfluentialSilva, D. F., da Silva, A. C. M., Ortolan, L. F., & Marcacini, R. M. (2021) · Simpósio Brasileiro de Computação Musical (SBCM) · doi:10.5753/sbcm.2021.19427 · Cited by 3
“Some examples of knowledge domains used to train domain-specific language models are biomedicine [44], finance [45], and scientific writing [46].”
BackgroundHuman-in-the-Loop Data Collection and Evaluation for Improving Mathematical Conversations
Datta, D., Phillips, M., Bywater, J. P., Lilly, S., Chiu, J. L., Watson, G. S., & Brown, D. E. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11647-6_113 · Cited by 3
Towards An Automated Physical Model Builder: CSTR Case Study
Kato, S., & Kano, M. (2022) · Computer-aided chemical engineering/Computer aided chemical engineering · doi:10.1016/b978-0-323-85159-6.50278-5 · Cited by 3
Detecting Regulation Violations for an Indian Regulatory Body through Multi Label Classification
Narayan, U., Parikh, P., Karlapalem, K., & Raman, N. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524640 · Cited by 3
“A possible solution in these closed domains is to fine-tune the model on domain specific data, and approaches such as Sci-BERT [1], BioBERT[8] have shown that fine-tuning can greatly improve performance in these closed domains.”
BackgroundCLGR-Net: a collaborative local-global reasoning network for document-level relation extraction
Ding, X., Zhou, G., Lu, J., & Zhu, T. (2022) · The Journal of Supercomputing · doi:10.1007/s11227-022-04875-9 · Cited by 4
“Besides that, we also include the SciBERT [27] baseline models, SciBERT is also a pre-trained model based on BERT, which is pre-trained on scientific text.”
MethodA Message Passing Approach to Biomedical Relation Classification for Drug–Drug Interactions
Zaikis, D., Karalka, C., & Vlahavas, I. P. (2022) · Applied Sciences · doi:10.3390/app122110987 · Cited by 3
Generalizable Natural Language Processing Framework for Migraine Reporting from Social MediaPreprint
Guo, Y., Rajwal, S., Lakamana, S., Chiang, C., Menell, P. C., Shahid, A. H., Chen, Y., Chhabra, N., Chao, W.-J., Chao, C., et al. (2022) · PubMed Central · doi:10.48550/arxiv.2212.12454 · Cited by 11
Clustering Semantic Predicates in the Open Research Knowledge Graph
Oghli, O. A., D’Souza, J., & Auer, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21756-2_39 · Cited by 4
A Relation-Aware Span-Level Transformer Network for Joint Entity and Relation Extraction
Li, Z., & Fu, L. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892140 · Cited by 3
“In the tables, w/ BERT, w/ RoBERTa and w/ SciBERT denote the model using BERT [18], RoBERTa [23] and SciBERT [24] as the pre-trained model separately.”
MethodParameter-efficient feature-based transfer for paraphrase identification
Liu, X., Rzepka, R., & Araki, K. (2022) · Natural Language Engineering · doi:10.1017/s135132492200050x · Cited by 3
“2019) and SciBERT (74.1/72.3) (Beltagy, Lo, and Cohan 2019) in terms of accuracy.”
BackgroundInfluentialWang, M., Zhang, X., Zhong, H., & Wang, D. (2023) · Journal of Information Science · doi:10.1177/01655515231151406 · Cited by 3
“%) and Computer Science (12%), which can be more suitable for natural language processing tasks related to scientific papers [28].”
Generating Topic Pages for Scientific Concepts Using Scientific Publications
Azarbonyad, H., Afzal, Z., & Tsatsaronis, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_23 · Cited by 3
“SciBERT model We use the SciBERT model [1] which is trained on scientific articles.”
MethodShenoy, N., Nayak, P., Jain, S., Kamath, S. S., & Sugumaran, V. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35320-8_29 · Cited by 3
Named Entity Recognition (NER) and Relation Extraction in Scientific Publications
Singh, A., & Garg, A. (2023) · International Journal of Recent Technology and Engineering (IJRTE) · doi:10.35940/ijrte.b7846.0712223 · Cited by 3
Topical Grouping of Thousands of Biomimetics Articles According to Their Goals, Results and Methods
Carniel, T., Cazenille, L., Dalle, J., & Halloy, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39504-8_18 · Cited by 3
Corpus Complexity Matters in Pretraining Language Models
Agrawal, A., & Singh, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.20 · Cited by 3
Detecting Contradictory COVID-19 Drug Efficacy Claims from Biomedical Literature
Sosa, D. N., Suresh, M., Potts, C., & Altman, R. B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.61 · Cited by 8
Henning, S., Macher, N., Grünewald, S., & Friedrich, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.94 · Cited by 4
Leveraging Knowledge Graph Embeddings to Enhance Contextual Representations for Relation Extraction
Laleye, F. A. A., Rakotoson, L., & Massip, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41501-2_2 · Cited by 3
“The results show a very significant performance of the PubMedBERT model on all 3 datasets compared to SciBERT, BioBERT and SpanBERT.”
BackgroundMethodInfluentialValidating Synthetic Usage Data in Living Lab Environments
Breuer, T., Fuhr, N., & Schaer, P. (2023) · Journal of Data and Information Quality · doi:10.1145/3623640 · Cited by 6
“In our analysis, often the winning queries, i.e., those queries for which the experimental system wins, directly turn into losing queries as soon as the bad ranking criterion is assigned a higher weight than that of the baseline system.”
Background2SCE-4SL: a 2-stage causality extraction framework for scientific literature
Zhang, Y. S., Bai, R., Kong, L., & Wang, X. (2023) · Scientometrics · doi:10.1007/s11192-023-04817-z · Cited by 3
“To compare the effect, SciBert[44], a pre training model based on scientific papers, will be compared with 2SCE-4SL as baseline method for CE.”
InfluentialSMAuC - The Scientific Multi-Authorship Corpus
Bevendorff, J., Sauer, P., Gienapp, L., Kircheis, W., Körner, E., Stein, B., & Potthast, M. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00013 · Cited by 3
“A corpus of 1.14 million paper full texts was used by both Ammar et al. [1] and Beltagy et al. [2].”
MethodInfluentialStepanov, I., Ivasiuk, A., Yavorskyi, O., & Frolova, A. (2023) · Frontiers in Genetics · doi:10.3389/fgene.2023.1238140 · Cited by 3
“In this section, we will describe the performance of our PMI/TF-IDF, Naïve Bayes classi fi ers and transformers such as SciBERT, Longformer, BioGPT and T5 on Validation datasets.”
MethodPark, M., Jeong, C. U., Baik, Y. S., Lee, D. G., Park, J. U., Koo, H. J., & Kim, T. Y. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294713 · Cited by 4
Anand, A., Jairath, A., Lal, N., Bangar, S., Sikka, J., Verma, A., Shah, R. R., & Satoh, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49601-1_7 · Cited by 3
Park, S. Y., & Caragea, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.805 · Cited by 8
ScdNER: Span-Based Consistency-Aware Document-Level Named Entity Recognition
Wei, Y., & Li, Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.970 · Cited by 7
Biomedical event extraction using pre-trained SciBERT
Mulya, D., & Khodra, M. L. (2023) · Journal of Intelligent Systems · doi:10.1515/jisys-2023-0021 · Cited by 3
Biomedical Event Detection Based on Domain Knowledge Injection and Model Dual Channel Fine-tuning
He, X., Han, X., Li, S., & Ren, Y. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385688 · Cited by 3
Die kommende Entwicklung großer Sprachmodelle in der Medizin
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J., Laleh, N. G., Loeffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2024) · Kompass Onkologie · doi:10.1159/000536600 · Cited by 3
Sharma, T., Farea, A., Perera, N., & Emmert‐Streib, F. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26973 · Cited by 4
Inference-Based No-Learning Approach on Pre-Trained BERT Model Retrieval
Pham, H.-L., Mibayashi, R., Yamamoto, T., Kato, M. P., Yamamoto, Y., Shoji, Y., & Ohshima, H. (2024) · International Conference on Big Data and Smart Computing · doi:10.1109/bigcomp60711.2024.00044 · Cited by 1
“For instance, SciBERT [2], which was pre-trained using a large multi-domain corpus of scientific publications, showed improved performance on downstream scientific NLP tasks.”
MethodJiang, L.-Z., & Zhang, S. (2024) · Symmetry · doi:10.3390/sym16050587 · Cited by 3
A Decoupling and Aggregating Framework for Joint Extraction of Entities and Relations
Wang, Y., Liu, X., Kong, W.-K., Yu, H.-T., Racharak, T., Kim, K., & Nguyen, L.-M. (2024) · IEEE Access · doi:10.1109/access.2024.3420877 · Cited by 3
“Therefore, important information may be lost when using sigmoid to compress high-dimensional features.”
BackgroundTeacher-informed Expansion of an Idea Detection Model for a Knowledge Integration Assessment
Li, W., Liao, Y., Steimel, K., Bradford, A., Gerard, L. F., & Linn, M. C. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664687 · Cited by 3
“The NLP model architecture included a pretrained transformer backbone SciBERT [2], which leverages 1.1 million scientific papers for pretraining and vocabulary creation, a bidirectional GRU-based RNN, and a final linear projection, trained using Binary Cross Entropy loss.”
InfluentialThe Effect of Knowledge Graph Schema on Classifying Future Research Suggestions
Alivanistos, D., Bijl, S. V. D., Cochez, M., & van Harmelen, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_10 · Cited by 3
Unveiling the Potential of Large Language Models: Redefining Learning in the Age of Generative AI
Varghese, N., & Ramasamy, G. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-5288-5.ch015 · Cited by 4
Panagides, R., Fu, S. H., Jung, S. H., Singh, A. K., Muttikkal, R. E., Broad, R. M., Meakem, T. D., & Hamilton, R. A. (2024) · AI · doi:10.3390/ai5030081 · Cited by 3
“The BERT models we trained included TinyBERT [24], SciBERT [25], Dis-tilBERT [26], and Bio-ClinicalBERT [27] for performance evaluation.”
PLRTE: Progressive learning for biomedical relation triplet extraction using large language models
Zheng, Y.-K., Zeng, B., Feng, Y., Zhou, L., & Li, Y. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104738 · Cited by 3
ExPrompt: Augmenting Prompts Using Examples as Modern Baseline for Stance Classification
Qudus, U., Röder, M., Vollmers, D., & Ngomo, A.-C. N. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679923 · Cited by 3
“Additionally, we report the results for sentence encoder-based approaches, namely SCIBERT [2], BioMe-dRoBERTa [10], RoBERTa-base [14], and RoBERTa-large [14], on the SciFact dataset reported by Wadden et al. [29].”
AutoRDF2GML: Facilitating RDF Integration in Graph Machine Learning
Färber, M., Lamprecht, D., & Susanti, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77847-6_7 · Cited by 3
Chen, Y., Li, X., Liu, Y., Bi, P., & Hu, T. (2024) · Symmetry · doi:10.3390/sym16121629 · Cited by 4
UniASA: A Unified Generative Framework for Argument Structure Analysis
Bao, J., Jing, M., Dong, K., Sun, A., Sun, Y., & Xu, R. (2025) · Computational Linguistics · doi:10.1162/coli_a_00553 · Cited by 3
Deep Learning-Based Scientific Document Summarization Considering Citation
Jyoti, D., Mahato, D. P., & Srivastava, J. (2025) · SN Computer Science · doi:10.1007/s42979-025-03821-9 · Cited by 3
“SciBERT [26], designed specifically for scientific text, provides improved performance in domain-specific applications by leveraging a vocabulary derived from scientific literature.”
Encoder models at the European Patent Office: Pre-training and use cases
Hähnke, V. D., Wéry, A., Wirth, M., & Klenner-Bajaja, A. (2025) · World Patent Information · doi:10.1016/j.wpi.2025.102360 · Cited by 3
MedCT: A Clinical Terminology Graph for Generative AI Applications in Healthcare
Chen, Y., Huang, D., Shen, Y., Xu, H., Sheng, L., Zhou, Q., & Wang, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97557-8_3 · Cited by 6
Zhou, J., Xiao, B., Liu, Q., Liu, L., & Zhang, L. (2025) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.5c08809 · Cited by 11
Reviewer Agreement and Phrase-Based Profiles
Mazumder, A., Saha, S., Malempati, S., & Kodali, R. K. (2023) · PhilPapers (PhilPapers Foundation) · doi:10.5281/zenodo.20505256 · Cited by 3
Mercadier, Y., Azé, J., & Bringay, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59137-3_9 · Cited by 3
An Instance Level Approach for Shallow Semantic Parsing in Scientific Procedural Text
Swarup, D., Bajaj, A., Mysore, S., O’Gorman, T., Das, R., & McCallum, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.270 · Cited by 3
SciBERT-based Semantification of Bioassays in the Open Research Knowledge Graph
Anteghini, M., D’Souza, J., dos Santos, V. A. P. M., & Auer, S. (2020) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.13140/rg.2.2.25898.64969 · Cited by 3
A Deep Analysis of an Explainable Retrieval Model for Precision Medicine Literature Search
Qu, J., Arguello, J., & Wang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_36 · Cited by 3
Determining the Credibility of Science Communication
Augenstein, I. (2021) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2021.sdp-1.1 · Cited by 3
Multifaceted Domain-Specific Document Embeddings
Risch, J., Hager, P., & Krestel, R. (2021) · doi:10.18653/v1/2021.naacl-demos.9 · Cited by 6
Karia, N., Kaushal, A., & Mallick, F. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.46 · Cited by 3
TADPOLE: Task ADapted Pre-Training via AnOmaLy DEtection
Madan, V., Khetan, A., & Karnin, Z. S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.463 · Cited by 3
DataQuest: An Approach to Automatically Extract Dataset Mentions from Scientific Papers
Kumar, S., Ghosal, T., & Ekbal, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91669-5_4 · Cited by 3
Bag of Tricks for Citation Intent Classification via SciBERT
Motrichenko, D., Nedumov, Y., & Skorniakov, K. (2021) · Ivannikov ISPRAS Open Conference (ISPRAS) · doi:10.1109/ispras53967.2021.00022 · Cited by 3
Muthukkumaran, A., Ravichandran, A., Shanbhag, S., Arjun, R., & Rengaswamy, R. (2022) · Computer-aided chemical engineering/Computer aided chemical engineering · doi:10.1016/b978-0-323-95879-0.50239-3 · Cited by 3
TAM GAN: Tamil Text to Naturalistic Image Synthesis Using Conventional Deep Adversarial Networks
Diviya, M., & Arockiasamy, K. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3584019 · Cited by 3
“But the researchers brought out the fact of handling scientific data by using the SCI-BERT model, which proves to be an improvement of NLP tasks in scientific areas [23].”
MethodYATO: Yet Another deep learning based Text analysis Open toolkit
Wang, Z., Wang, Y., Wu, J., Teng, Z., & Yang, J. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.11 · Cited by 5
Shi, M.-X., Chen, C.-C., Huang, H., & Chen, H. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.5 · Cited by 3
Experiments on transfer learning architectures for biomedical relation extractionPreprint
Hafiane, W., Legrand, J., Toussaint, Y., & Coulet, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.12380 · Cited by 3
“1 4 Experimental Results 4.1 Frozen Strategy Architecture F-micro σ (Beltagy et al., 2019) SciBERT+BiLSTM 75.03 – BERT + BiLSTM 64.41 2.42 + MCNN 68.26 1.54 + BiLSTM-MCNN L. 75.35 1.02 + BiLSTM-MCNN P. 62.07 1.33 BioBERT + BiLSTM 72.86 1.36 + MCNN 77.57 0.70 + BiLSTM-MCNN L. 80.08 0.80 + BiLSTM-MCNN P. 74.85 0.96 SciB…”
InfluentialRecognising Biomedical Names: Challenges and SolutionsPreprint
Dai, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12230 · Cited by 6
Abògúnr̀in, Ṣ., Queiros, L., Bednarski, M., Sumner, M., Baehrens, D., & Witzmann, A. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.30.462652 · Cited by 3
“Similarly, SciBERT is a pretrained 95 model based on BERT and trained using scientific terminology [16].”
MethodMDAPT: Multilingual Domain Adaptive Pretraining in a Single ModelPreprint
Jørgensen, R. K., Hartmann, M., Dai, X., & Elliott, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06605 · Cited by 3
Graphine: A Dataset for Graph-aware Terminology Definition GenerationPreprint
Liu, Z., Wang, S., Gu, Y., Zhang, R., Zhang, M., & Wang, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04018 · Cited by 3
“Within the three domain-specific pretrained language models, BioBERT and SciBERT obtained the most prominent performance.”
BackgroundMethodResultInfluentialLegal HeBERT: A BERT-based NLP Model for Hebrew Legal, Judicial and Legislative TextsPreprint
Chriqui, A., Yahav, I., & Bar‐Siman‐Tov, I. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4147127 · Cited by 3
Biomedical Knowledge Discovery From Unstructured Text Corpora Using Contextual Word EmbeddingsPreprint
Panesar, S. S. (2022) · Research Square · doi:10.21203/rs.3.rs-2286334/v1 · Cited by 3
Adaptive Fine-tuning for Multiclass Classification over Software Requirement DataPreprint
Çevik, M., Yıldırım, S., Parikh, D., & Basar, A. (2023) · Research Square · doi:10.21203/rs.3.rs-2434133/v1 · Cited by 3
Measuring the Commercial Potential of SciencePreprint
Masclans, R., Hasan, S., & Cohen, W. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4481928 · Cited by 3
Natural Language Processing for Financial RegulationPreprint
Achitouv, I., Gorduza, D., & Jacquier, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4638347 · Cited by 3
Hirano, M., & Imajo, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4796245 · Cited by 3
Nechakhin, V., D’Souza, J., & Eger, S. (2024) · Preprints.org · doi:10.20944/preprints202405.0124.v1 · Cited by 3
Windisch, P. Y., Dennstädt, F., Koechli, C., Schroeder, C., Aebersold, D. M., Foerster, R., & Zwahlen, D. R. (2024) · medRxiv · doi:10.1101/2024.10.23.24316005 · Cited by 3
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.