SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,601–2,700 of 3,596
Non-Autoregressive Speech Recognition with Error Correction Module
Qian, Y., Zhuang, X., Zhang, Z., Zhou, L., Lin, X., & Wang, M. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980031 · Cited by 1
“In recent years, pre-trained models have achieved good results on different natural language processing tasks [12], [13], [14].”
BackgroundLi, L., Jiang, S., & Zhang, B. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995158 · Cited by 1
“Given the input sentence, we first use SciBERT [4] to acquire the original context embedding X = {x1, · · · , xm}, where m denotes the sequence length.”
MethodBioELM: Integrating Biomedical Knowledge into Language Model with Entity-Linking
Li, Q., Wu, G., & You, T. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995583 · Cited by 2
“We compared BioELM with other similarly sized biomedical PLMs on the aforementioned datasets, including BioBERT [6],SciBERT [3], ClinicalBERT [5], BlueBERT [13] and PubMedBERT [4].”
BackgroundMethodMcPeak, G. (2022) · IEEE International Conference on Application of Information and Communication Technologies (AICT) · doi:10.1109/aict55583.2022.10013532 · Cited by 1
Min, B. L. N., & Quan, T. (2022) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf55975.2022.10013885 · Cited by 1
“We use SciBERT [18] for the embedding layer We train both SpERT.”
MethodTourismNLG: A Multi-lingual Generative Benchmark for the Tourism Domain
Bhatt, S. M., Agarwal, S., Gurjar, O., Gupta, M., & Shrivastava, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_10 · Cited by 1
Categorizing Citation Relations in Scientific Papers Based on the Contributions of Cited Papers
Chen, P.-C., Huang, H., & Chen, H. (2022) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat55865.2022.00063 · Cited by 1
“This may be because SciBERT has seen a large number of scientific papers in the pre-training stage and has already learned relevant background knowledge.”
MethodInfluentialEffective Language Representations for Danmaku Comment Classification in Nicovideo
Nagao, H., Tamura, K., & Katsurai, M. (2023) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2022dap0010 · Cited by 1
Permutation Invariant Training for Paraphrase Identification
Bai, J., Yin, C., Hong, H., Zhang, J., Li, C., Wang, Y., & Rong, W. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094896 · Cited by 1
Classifiers of Medical Eponymy in Scientific Texts
Toddenroth, D. (2023) · Studies in health technology and informatics · doi:10.3233/shti230271 · Cited by 1
“Subsequent batches were pre-annotated by fitting a logistic regression model to contextualized vectors from the last hidden layer of the SciBERT language model, using the scivocab_uncased configuration; SciBERT has been pre-trained in a domain-specific corpus, and uses vectors with 768 dimensions [6].”
BackgroundMethodFoundation Models for Information Extraction
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_5 · Cited by 1
Improving the Identification of Abusive Language Through Careful Design of Pre-training Tasks
Jarquín-Vásquez, H., Escalante, H. J., & Montes-y-Gómez, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33783-3_27 · Cited by 1
A Comparative Study of Cross-Sentence Features for Named Entity Recognition
Wang, S., Huang, J., Zhang, B., & Li, J. (2023) · International Conference on Innovations and Development of Information Technologies and Robotics (IDITR) · doi:10.1109/iditr57726.2023.10145820 · Cited by 1
“Figure 2 and Figure 3 show the performance and computing time of the models with Fill, Full and Roll strategies on SciERC dataset for different context window sizes, where PLM is SciBERT and the stitching strategy is Both.”
MethodInfluentialBuilding Knowledge Graphs in Heliophysics and Astrophysics
Khoo, F. S., Mark, M. D., Timmer, R. C., Martins, M. S. R., Foshee, E., Bugbee, K., Renard, G., & Berea, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35320-8_15 · Cited by 1
Eroom’s Law and the decline in the productivity of biopharmaceutical R&D
Scannell, J. W. (2023) · OECD eBooks · doi:10.1787/6f256d17-en · Cited by 5
The end of Moore’s Law? Innovation in computer systems continues at a high pace
Kressel, H. (2023) · OECD eBooks · doi:10.1787/63e48242-en · Cited by 3
Smădu, R.-A., Zaharia, G.-E., Avram, A.-M., Cercel, D.-C., Dascălu, M., & Pop, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.27027 · Cited by 1
Similarity-Based Memory Enhanced Joint Entity and Relation Extraction
Kościukiewicz, W., Wójcik, M., Kajdanowicz, T. J., & Gonczarek, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36021-3_29 · Cited by 1
“For the domain-specific BioCreative V CDR dataset we used SciBERT BASE [1] which was trained on scientific papers from Semantic Scholar.”
She is an Expert in this Research Field: The Signal of Recent Publications' Relevance
Zeevi, G., & Mokryn, O. (2023) · Northeast Journal of Complex Systems · doi:10.22191/nejcs/vol5/iss1/7/ · Cited by 1
MIL-BERT: Military Domain Specialized Korean Pre-trained Language Model
Heo, H.-S., Yoon, C.-M., Ryu, Y.-H., Yong, S.-H., & Kim, D. (2023) · Journal of the KNST · doi:10.31818/jknst.2023.06.6.2.201 · Cited by 1
TaxoSBERT: Unsupervised Taxonomy Expansion Through Expressive Semantic Similarity
Margiotta, D., Croce, D., & Basili, R. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-39059-3_20 · Cited by 1
Annotating Research Infrastructure in Scientific Papers: An NLP-driven Approach
Tabatabaei, S. A., Cheirmpos, G., Doornenbal, M., Zigoni, A., Moore, V., & Tsatsaronis, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.44 · Cited by 1
WeLT: Improving Biomedical Fine-tuned Pre-trained Language Models with Cost-sensitive Learning
Mobasher, G., Müller, W., Krebs, O., & Gertz, M. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.40 · Cited by 1
How Much do Knowledge Graphs Impact Transformer Models for Extracting Biomedical Events?
Zanella-Calzada, L. A., & Toussaint, Y. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.12 · Cited by 1
Neves, M., Schadock, I., Eusemann, B., Schnfelder, G., Bert, B., & Butzke, D. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.11 · Cited by 2
Promoting Fairness in Classification of Quality of Medical Evidence
Šuster, S., Baldwin, T., & Verspoor, K. M. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.39 · Cited by 2
DDI-MuG: Multi-aspect Graphs for Drug-Drug Interaction Extraction
Yang, J., Ding, Y., Long, S., Poon, J., & Han, S. C. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.15 · Cited by 1
Analyzing the Impact of Tokenization on Multilingual Epidemic Surveillance in Low-Resource Languages
Mutuvi, S., Boroş, E., Doucet, A., Lejeune, G., Jatowt, A., & Odeo, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41682-8_2 · Cited by 3
LocBERT: Improving Social Media User Location Prediction Using Fine-Tuned BERT
Khan, A., Zhang, H., Boudjellal, N., Ahmad, A., & Khan, M. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-39689-2_3 · Cited by 1
Named Entity Recognition for peer-review disambiguation in academic publishing
Cuculovic, M., Fondement, F., Devanne, M., Weber, J., & Hassenforder, M. (2023) · International Conference on Information and Computer Technologies (ICICT) · doi:10.1109/icict58900.2023.00025 · Cited by 1
“Recent deep learning models found in the literature include GPT [21], XLM [22], XLNet [23], BERT [24], and its derivative models such as RoBERTa[25], SciBERT[26], DistilBERT [27], etc.”
MethodDrBERT: Un modèle robuste pré-entraîné en français pour les domaines biomédical et clinique
Labrak, Y., Bazoge, A., Dufour, R., Rouvier, M., Morin, E., Daille, B., & Gourraud, P. (2023) · HAL · Cited by 4
Generating survey draft based on closeness of position distributions of key words
Sun, X., & Zhuge, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121422 · Cited by 2
Simple and Effective Multi-Token Completion from Masked Language Models
Kalinsky, O., Kushilevitz, G., Libov, A., & Goldberg, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.179 · Cited by 1
Lee, S., & Bloem, J. (2023) · Workshop on Slavic Natural Language Processing (SlavicNLP) · doi:10.18653/v1/2023.bsnlp-1.7 · Cited by 1
Fan, L., Lafia, S., Wofford, M. F., Thomer, A., Yakel, E., & Hemphill, L. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00039 · Cited by 2
“Going forward, machine learning models like SciBERT [5] and CiteBERT [76] that use transformer-enabled bi-directional deep learning can improve our current methods for making syntax-aware decisions and providing deeper insights into data reference functions.”
MethodOverview of NLPCC Shared Task 2: Multi-perspective Scientific Machine Reading Comprehension
Zhang, X., Zheng, H., Nie, Y., & Mao, X.-L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44699-3_10 · Cited by 1
Learning entity-oriented representation for biomedical relation extraction
Hu, Y., Chen, Y., Qin, Y., & Huang, R. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104527 · Cited by 1
An Article Similarity-Based Approach for Planning Conference Sessions
Gündoğan, E., & Kaya, M. (2023) · International Conference on Advanced Computer Information Technologies (ACIT) · doi:10.1109/acit58437.2023.10275336 · Cited by 1
“SPECTER [18] and SciBERT [19] are pre-trained models.”
MethodAutoTGRL: an automatic text-graph representation learning framework
Al-Sabri, R., Gao, J., Chen, J., Olouladé, B. M., & Lyu, T. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-09226-0 · Cited by 3
“First, we use SciBERT [47] to create contex-tualized token representations H ¼ h 1 ; h 2 ; :::; h n f g for a text with n tokens.”
Lee, Y., Lee, D., & Jung, K. (2023) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/2023.argmining-1.24 · Cited by 3
Pretraining Language- and Domain-Specific BERT on Automatically Translated Text
Ishigaki, T., Uehara, Y., Topić, G., & Takamura, H. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_060 · Cited by 4
Deep Learning Methods for Identification of Multiword Flower and Plant Names
Premasiri, D., Haddad, A. H., Ranasinghe, T., & Mitkov, R. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_095 · Cited by 1
Citation Recommendation Employing Proximity-Based Heterogeneous Network Embeddings
Ali, Z., Ullah, I., Kefalas, P., Thierry, N., Haq, K. U., & Sarkar, A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47721-8_32 · Cited by 3
Low-Resource Efficient Multi-Stage Tuning Strategy for Biomedical Question Answering Task
Wang, B., Du, Y., Jin, X., Yan, R., & Zhang, Q. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385678 · Cited by 2
“Compared to the BioBERT [9] model, our method only uses a smaller dataset in the multi-stage training and achieves better performance.”
MethodInfluentialEntity Relation Aware Graph Neural Ranking for Biomedical Information Retrieval
He, Y., Liu, X., Hu, J., & Dong, S. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385584 · Cited by 1
“The multi-task learning framework employs the pre-trained model SciBERT [16], trained on a large-scale biomedical literature, as the shared text encoder for both the document ranking module and the document entity relation aware module.”
MethodKeyword Extraction from Scientific Publications Using Local Features and Embedding Model
Kurniawan, G. S. K., & Lhaksmana, K. M. (2023) · International Conference on Signal Processing and Intelligent Systems (ICSPIS) · doi:10.1109/icspis59665.2023.10402723 · Cited by 1
“…of keyword extraction from scientific publication by combining embedding model and local statistical features using supervised method, and (ii) compare the state-of-the-art model including SciBERT [8], JinaBERT [9], and MPNet [10] for candidate keyphrase embedding and document similarity measures.”
MethodZurlo, G., & Ronchieri, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-51023-6_18 · Cited by 2
A Simple but Useful Multi-corpus Transferring Method for Biomedical Named Entity Recognition
Li, J., Yuan, C., Li, Z., Wang, H., & Tao, F. (2024) · Communications in computer and information science · doi:10.1007/978-981-99-9864-7_5 · Cited by 1
Advancing document-level relation extraction with a syntax-enhanced multi-hop reasoning network
Zhong, Y., Shen, B., & Wang, T. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-237167 · Cited by 1
Varma, A., & Bhat, C. (2024) · Artificial Intelligence and Big Data · doi:10.5121/csit.2024.140419 · Cited by 1
“Work in SCIBERT [15] and PubMedBERT [16] and other similar work in the past attempt to train transformer architectures from scratch.”
BackgroundNanoNER: Named Entity Recognition for Nanobiology Using Experts’ Knowledge and Distant Supervision
Cheng, R., Lentschat, M., & Labbé, C. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.2 · Cited by 1
Improving Knowledge Graph-based BERT Pre-Training Through Entity Hot Partition and Adapter Fusion
Zhan, D., Wu, X., Chang, Y., Tian, C., Wang, T., Chen, C., Xie, L., Huang, W., Tong, E., & Niu, W. (2023) · China Automation Congress (CAC) · doi:10.1109/cac59555.2023.10451936 · Cited by 1
MathMex: Search Engine for Math Definitions
Durgin, S., Gore, J., & Mansouri, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_17 · Cited by 1
Expert Evaluation of Export Control-Related Question Answering Capabilities of LLMs
Rzepka, R., Muraji, S., & Obayashi, A. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487735 · Cited by 1
“Fine-tuning BERT with specialized data or training from scratch including a domain topic as science [16], biology [17], finance [18] or law [19].”
BackgroundMicro Citation Importance Identification and Its Application to Literature Evaluation
Nie, W., & Ou, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57860-1_24 · Cited by 1
Indian Annual Report Assessment Using Large Language Models
Sisodia, Y. (2024) · Algorithms for intelligent systems · doi:10.1007/978-981-97-0327-2_2 · Cited by 1
Santhiya, S., Sharmila, C., Jayadharshini, P., Dharshini, M. N., Kumar, B., & Sandeep, K. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-58495-4_10 · Cited by 1
CMed-GPT: Prompt Tuning for Entity-Aware Chinese Medical Dialogue Generation
Qu, Z., Li, J., Ma, Z., & Li, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2253-2_7 · Cited by 3
Liu, X., Rzepka, R., & Araki, K. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111883 · Cited by 1
Panagides, R., Jung, S. H., Fu, S. H., Singh, A. K., & Muttikkal, R. E. (2024) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds61124.2024.10534719 · Cited by 1
“Total Parameters Dataset TinyBERT [13] 14,350,874 Original BERT corpus SciBERT [14] 109,920,002 Semantic Scholar corpus DistilBERT [15] 66,955,010 Original BERT corpus Bio-ClinicalBERT [16] 108,311,810 MIMIC-III corpus”
BackgroundMethodWebMap - Large Language Model-assisted Semantic Link Induction in the Web
Pokharel, S., Roßrucker, G. P., & Kubek, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-60433-1_8 · Cited by 1
“On the contrary, depending on the individual web servers’ content, domain-specific models such as SciBERT [6], and FinBERT [7] might be better suited to capture domain-specific characteristics and nuances.”
MaugVLink: Augmenting Mathematical Formulas with Visual Links
Lee, M.-J., & Chen, B. (2024) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis60374.2024.00048 · Cited by 1
“Lee et al. [29] propose a two-stage pipeline model based on SciBERT [12] and achieved first on the leaderboard for all the subtasks in Symlink.”
Reference Classification Using BERT Models to Support Scientific-Document Writing
Hosokawa, R., Yamato, J., Higashinaka, R., Kikui, G., & Sugiyama, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60511-6_11 · Cited by 1
dos Reis, G. R., Mos, A., Cornax, M. C., & Labbé, C. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644978 · Cited by 1
Building a Large Corpus and Pre-trained Language Models from National and Local Assembly Minutes
Nagafuchi, K., Kimura, Y., Kadowaki, K., & Araki, K. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.707 · Cited by 2
Classifying Positive Results in Clinical Psychology Using Natural Language Processing
Schiekiera, L., Diederichs, J., & Niemeyer, H. (2024) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000563 · Cited by 3
Wu, K., Chen, Z., Wu, X., Li, G., Li, J., Wang, S., Wang, H., & Feng, H. (2024) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi13070260 · Cited by 1
A Multi-Task Learning Framework for Reading Comprehension of Scientific Tabular Data
Xu, Y., Zhang, M., Fan, J., Luo, Z., & Yang, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00285 · Cited by 2
“We develop a NER model using SciBERT [59].”
MethodDevelopment of Language Model on Biomedical Domain to Pretrain Natural Language Processing
Gunturu, V., Priya, Y. D., Bachhav, G. V., Praveena, K., Anandhi, R. J., & Dhaliwal, N. (2024) · International Conference on Communication, Computer Sciences and Engineering (IC3SE) · doi:10.1109/ic3se62002.2024.10593288 · Cited by 1
“According to previous studies Over General domain language models in domain text Gains more advantage comparatively Specialised domains such as Biomedicine [3].”
Lee, L.-H., Chiou, C.-Y., & Lin, T.-M. (2024) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2024.semeval-1.209 · Cited by 1
Question Answering in Medical Domain Using Natural Language Processing: A Review
Singh, G. B., Kumar, R., Ghosh, R. C., Bhakhuni, P., & Sharma, N. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-3245-6_26 · Cited by 3
CRFLOE: Context Region Filter and Relation Word Aware for Document-Level Relation Extraction
Yang, D., Li, X., Wu, H., Zhou, A., & Liu, P. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5669-8_9 · Cited by 1
Single-Label Multi-modal Field of Research Classification
Ruosch, F., Vasu, R., Wang, R., Rossetto, L., & Bernstein, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_15 · Cited by 1
Otto, W., Upadhyaya, S., & Dietze, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_21 · Cited by 1
Identifying The Established Semantic Kinds In Text Using Named Entity Recognition
Seershika, M., Aishwarya, V. S., Vani, V., & Karthik, N. (2024) · International Conference on Signal Processing, Computation, Electronics, Power and Telecommunication (IConSCEPT) · doi:10.1109/iconscept61884.2024.10627815 · Cited by 1
Matching Problem Statements to Editorials in Competitive Programming
Dinu, I. G., Mihăescu, M. C., & Rebedea, T. E. (2024) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt61570.2024.00056 · Cited by 1
“To evaluate its performance, we conducted a comparative analysis against several state-of-the-art models, including BERT [1], SciBERT [2], RoBERTa [3], MPNet [4] and several sentence-transformer-based [5] models utilizing them as our baseline reference.”
MethodDhavale, C., Bharambe, U., Shinde, S., & Murkute, A. M. (2024) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss53475.2024.10640504 · Cited by 1
Refining Entity Descriptions with Relation Embeddings for Scientific Relation Classification
Li, C. H., Liu, X., Li, J. L., Wang, J. T.-L., Feng, Z., & Li, Q. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650696 · Cited by 1
“Sentence-level relation extraction is conducted on the SciERC dataset, the processed dataset from [15].”
BackgroundMethodAdaptation of Large Language Models for the Public Sector: A Clustering Use Case
Caudron, E., Ghesquière, N., Travers, W., & Balahur, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_31 · Cited by 1
Estimating Metadata of Research Artifacts to Enhance their Findability
Ito, K., & Matsubara, S. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678684 · Cited by 2
“We used SciBERT [7] 3 as the estimation model.”
MethodPartial Annotation Learning for Biomedical Entity Recognition
Ding, L., Colavizza, G., & Zhang, Z. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3466294 · Cited by 1
“PubMedBERTwas pre-trainedfromscratchusingtheabstractsandfull-textarticles fromPubMedCentral,whichhasbeenshowntooutperform BERT[23],RoBERTa[30],BioBERT[31],SciBERT[32]etc. onBioNERtask.”
Chen, Q., Yao, H. L., & Zhou, D. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103906 · Cited by 1
Dual-stream dynamic graph structure network for document-level relation extraction
Yu, Z. W., & Shen, B. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102202 · Cited by 1
“SciBERT is trained on scientific texts, taking into account the special structure and context of scientific texts, which enables it to better understand and represent entities in the scientific domain.”
CAM-BERT: Chinese Aerospace Manufacturing Pre-trained Language Model
Dai, J., Wang, S., Wang, P., Li, R., Chen, J. C., & Li, X. (2024) · ICON · doi:10.1109/icnlp60986.2024.10692464 · Cited by 1
“Beltagy et al. introduce the SCIBERT [11], demonstrating that employing a domain-specific vocabulary can significantly enhance performance.”
BackgroundTowards Understanding Evolution of Science through Language Model Series
Dong, J., Lyu, Z., & Ke, Q. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3821210 · Cited by 1
“…fashion to aid in downstream scientific NLP tasks where datasets are scarce, leading to the introduction of numerous scientific language models (SciLMs) for a number of fields, including biomedicine [4], [6], [8], [14]–[16], chemistry [17], material science [9], and social science [18].”
InfluentialInoshita, K., Zhou, X., & Shimizu, S. (2024) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics62450.2024.00045 · Cited by 1
“Moreover, they are generally pre-trained on large-scale datasets, such as Wikipedia [6], to improve their generalizability, and specialized tasks, such as medical or scientific literature classification, benefit from task-specific pre-training or fine-tuning [7,8].”
BackgroundMulti-layer Sequence Labeling-Based Joint Biomedical Event Extraction
Chen, G., Wu, P., Gu, J., Qian, L., & Zhou, G. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_11 · Cited by 1
Liao, Y., Xiang, H., Liu, H., & Spasić, . (2024) · IEEE Access · doi:10.1109/access.2024.3504378 · Cited by 1
Baeza-Yates, R. A., Murgai, L., & Dudy, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674138 · Cited by 1
Summarization of Scientific Articles using Hybrid SciBERT and Graph-Based Algorithms
Nugroho, G. S., amal, E. C. D., & Ilyas, R. (2024) · Proceeding of the Electrical Engineering Computer Science and Informatics · doi:10.1109/eecsi63442.2024.10776308 · Cited by 2
Enhancing Oncology Care With Federated Learning and Foundation Models
Gagan, N., & Sanand, S. (2024) · ITU Kaleidoscope Academic Conference (ITU K) · doi:10.23919/ituk62727.2024.10772940 · Cited by 1
“Among other domain adapted models are PubMedBERT [8], fine-tuned on PubMed abstracts and full-text articles, and SciBERT [9], which is fine-tuned on a large corpus of scientific literature.”
Text-To-AMP: Antimicrobial Peptide Design Guided by Natural Text
Huang, Y., Xiao, Y., Teng, A., Ye, F., Yu, Y., Lei, C., Zhang, R., Gao, Z., Tan, C., & Liu, X. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822829 · Cited by 1
What’s the data say? An LLM-based system for interrogating experimental data
Craig, D. B., & Drăghici, S. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821725 · Cited by 1
Examining the Representation of Youth in the US Policy Documents through the Lens of Research
Mokarrama, M. J., Shaikh, A. R., & Alhoori, H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825996 · Cited by 1
“In our previous study [16], we found that Bio ClinicalBERT provides a better embedding for our dataset and research context compared to other pretrained models like Scibert [23].”
Transformer-Based Text Highlighting for Medical Terms
Ozyegen, L., Çevik, M., & Basar, A. (2024) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon62161.2024.10837964 · Cited by 1
Ma, X., Kuang, Y., Yuan, L., Cheng, T., Zeng, Y., & Liu, S. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10832064 · Cited by 1
Medical Multi-Choice Question Answering with Retrieval Augmented Language Models
Liu, Y. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10832057 · Cited by 1
Language Model Embeddings to Improve Performance in Downstream Tasks
Pawar, A., Patil, P., Hiwanj, R., Kshatriya, A., Chikmurge, D. V., & Barve, S. (2024) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn63059.2024.10847347 · Cited by 1
Xu, Y., & Chen, Y. (2025) · Neural Processing Letters · doi:10.1007/s11063-025-11729-x · Cited by 1
A Framework Based on Natural Language Processing for Risk Management in Engineering
Yang, M. C. K., Petronijević, J., Etienne, A., & Siadat, A. (2024) · IEEE International Conference on Industrial Engineering and Engineering Management (IEEM) · doi:10.1109/ieem62345.2024.10856953 · Cited by 1
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.