SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 601–700 of 3,596
Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification
Zhang, Y., Shen, Z., Wu, C.-H., Xie, B.-Y., Hao, J., Wang, Y.-X., Wang, K., & Han, J. (2022) · The Web Conference · doi:10.1145/3485447.3512174 · Cited by 34
“In our experiments, since documents from both Microsoft Academic and PubMed are scientific papers, we adopt SciBERT [2] as our building block.”
BackgroundMethodHero-Gang Neural Model For Named Entity Recognition
Hu, J., Shen, Y., Liu, Y., Wan, X., & Chang, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.140 · Cited by 30
Explaining Relationships Between Scientific Documents
Luu, K., Wu, X., Koncel-Kedziorski, R., Lo, K., Cachola, I. A., & Smith, N. A. (2021) · doi:10.18653/v1/2021.acl-long.166 · Cited by 57
Self-citationZhang, Z., Parulian, N. N., Ji, H., Elsayed, A., Myers, S., & Palmer, M. S. (2021) · doi:10.18653/v1/2021.acl-long.489 · Cited by 47
Zhu, H., Peng, H. M., Lyu, Z., Hou, L., Li, J., & Xiao, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119369 · Cited by 41
“…” have corresponding semi-structured subsection titles ((sub)headings), paragraphs and well-structured knowledge triples ( Xiangshan Park, suitable (Beltagy, Lo and Cohan (2019)) use scientific publications to further pre-train the SCIBERT to improve performance on downstream scientific NLP tasks.”
BackgroundMethodInfluentialExtensive evaluation of transformer-based architectures for adverse drug events extraction
Scaboro, S., Portelli, B., Chersoni, E., Santus, E., & Serra, G. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110675 · Cited by 30
AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain
Hong, J., Kim, T., Lim, H., & Choo, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.385 · Cited by 27
“However, fine-tuning the LM is challenging when the downstream domain is significantly different from the pretrained domain, requiring domain adaptation to improve the downstream performance [Gururangan et al., 2020, Lee et al., 2020, Beltagy et al., 2019].”
InfluentialCan NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?
Xu, J., Ma, M. D., & Chen, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.138 · Cited by 32
SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug DesignPreprint
Edwards, C., Naik, A., Khot, T., Burke, M. D., Ji, H., & Hope, T. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.06.547759 · Cited by 35
“We train four BERT-based [13] language models [2, 92] and find that they outperform GraphSynergy in both name and random token settings.”
BackgroundMethodInfluentialLu, Y., Yuan, M., Liu, J., & Chen, M. (2023) · Scientometrics · doi:10.1007/s11192-022-04566-5 · Cited by 28
“14 million scientific papers in the fields of biomedicine (82%) and computer science (18%), therefore it is more suitable for natural language processing tasks of scientific papers than BERT (Beltagy et al., 2019).”
BackgroundMethodInfluentialFrom statistical methods to deep learning, automatic keyphrase prediction: A survey
Xie, B., Song, J., Shao, L., Wu, S., Wei, X., Yang, B., Lin, H., Xie, J., & Su, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103382 · Cited by 39
ExpFinder: A hybrid model for expert finding from text-based expertise data
Kang, Y., Du, H., Forkan, A. R. M., Jayaraman, P. P., Aryani, A., & Sellis, T. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118691 · Cited by 27
“To find t given t g , we use the scientific pre-trained model SciB-ERT 5 [39] that is a scientific language model trained on the fulltext of 1.14M papers and 3.1B words, where the papers were collected from ‘semanticscholar.org’.”
MethodBassignana, E., & Plank, B. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.7 · Cited by 46
Specialized document embeddings for aspect-based similarity of research papers
Ostendorff, M., Blume, T., Ruas, T., Gipp, B., & Rehm, G. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530912 · Cited by 38
“As base models, we rely on averaged FastText word vectors as document embeddings [6], documentembeddingsfromSciBERT[5] ,andSPECTER[11].”
MethodInfluentialContinual Domain-Tuning for Pretrained Language ModelsPreprint
Rongali, S., Jagannatha, A., Rawat, B. P. S., & Yu, H. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02288 · Cited by 27
Promotional language and the adoption of innovative ideas in science
Peng, H., Qiu, H. S., Fosse, H. B., & Uzzi, B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2320066121 · Cited by 26
“…methods offer a different innovation and will be a game-changer in the…” We used SciBert to estimate a grant’s positive sentiment, which is a contextual word embedding model pre-trained on scientific publications and can be used to evaluate the sentence-level sentiment of scientific text (47).”
MethodMulti-View Graph Neural Architecture Search for Biomedical Entity and Relation Extraction
Al-Sabri, R., Gao, J., Chen, J., Olouladé, B. M., & Lyu, T. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3205113 · Cited by 27
“MVGNAS uses SciBERT as the Transformer encoder [32].”
MethodEvidence Extraction to Validate Medical Claims in Fake News Detection
Deka, P., Jurek, A., & Deepak, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20627-6_1 · Cited by 59
Deep purified feature mining model for joint named entity recognition and relation extraction
Wang, Y., Wang, Y., Sun, Z., Li, Y., Hu, S., & Ye, Y. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103511 · Cited by 32
Shi, Y., He, M., Chen, J., Han, F., & Cai, Y. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011989 · Cited by 29
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
Meng, Z., Liu, F., Clark, T. H., Shareghi, E., & Collier, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.383 · Cited by 38
“, 2020), SciBERT (Beltagy et al., 2019) and PubMedBERT (Gu et al.”
BackgroundMethodInfluentialDou, M., Ding, J., Chen, G., Duan, J., Guo, F., & Tang, J. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad099 · Cited by 26
Aspect Sentiment Triplet Extraction: A Seq2Seq Approach With Span Copy Enhanced Dual Decoder
Zhang, Z., Zuo, Y., & Wu, J. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3198802 · Cited by 27
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge SearchPreprint
MacAvaney, S., Cohan, A., & Goharian, N. (2020) · arXiv (Cornell University) · Cited by 26
Self-citationA Multi-task Approach to Neural Multi-label Hierarchical Patent Classification Using Transformers
Pujari, S. C., Friedrich, A., & Strötgen, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_34 · Cited by 46
Dai, T., Zhao, J., Li, D., Tian, S., Zhao, X., & Pan, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118841 · Cited by 29
Mondal, I. (2020) · doi:10.18653/v1/2020.knlp-1.4 · Cited by 26
Meng, Z., Liu, F., Shareghi, E., Su, Y., Collins, C., & Collier, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.329 · Cited by 39
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient FrameworkPreprint
Yao, X., Zheng, Y., Yang, X., & Yang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04130 · Cited by 43
Wada, S., Takeda, T., Okada, K., Manabe, S., Konishi, S., Kamohara, J., & Matsumura, Y. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102889 · Cited by 29
Corporate Bankruptcy Prediction with Domain-Adapted BERT
Kim, A. G., & Yoon, S. (2021) · Workshop on Economics and Natural Language Processing (ECONLP) · doi:10.18653/v1/2021.econlp-1.4 · Cited by 30
“Domain shift harms BERT model performance substantially (Lee et al., 2020; Beltagy et al., 2019).”
BackgroundSLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search
MacAvaney, S., Cohan, A., & Goharian, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.341 · Cited by 40
Self-citationA Pretrained Language Model for Cyber Threat Intelligence
Park, Y. H., & You, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.12 · Cited by 45
Frei, J., Frei‐Stuber, L., & Krämer, F. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104513 · Cited by 27
OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining
Zhang, F., Shi, S., Zhu, Y., Chen, B., Cen, Y., Yu, J., Chen, Y., Wang, L., Zhao, Q., Cheng, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672354 · Cited by 38
“We use SciBERT [3] to encode concepts for a fair comparison with the next method.”
DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions
Viswanathan, V., Gao, L., Wu, T., Liu, P., & Neubig, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.573 · Cited by 44
ExplainaBoard: An Explainable Leaderboard for NLPPreprint
Liu, P., Fu, J., Yang, X., Yuan, W., Chang, S., Dai, J., Liu, Y., Ye, Z., Dou, Z.-Y., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06387 · Cited by 30
“This is true both for classical tasks such as machine translation (Sutskever et al., 2014; Wu et al., 2016), as well as for new tasks (Lu et al., 2016; Rajpurkar et al., 2016), domains (Beltagy et al., 2019), and languages (Conneau and Lample, 2019).”
BackgroundSciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning
Yang, Z., Dabre, R., Tanaka, H., & Okazaki, N. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1140 · Cited by 42
Prompting Strategies for Citation Classification
Kunnath, S. N., Pride, D., & Knoth, P. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615018 · Cited by 42
“We use the uncased scivocab version of SciBERT [2] as the scientific LM for all the experiments that updates parameters12.”
BackgroundMethodInfluentialAceso: PICO-Guided Evidence Summarization on Medical Literature.
Zhang, X., Geng, P., Zhang, T., Lu, Q., Gao, P., & Mei, J. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2984704 · Cited by 25
Unleashing the Power of Knowledge Extraction from Scientific Literature in Catalysis
Zhang, Y., Wang, C., Soukaseum, M., Vlachos, D. G., & Fang, H. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00359 · Cited by 26
Shah, S. M. A., & Ou, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104821 · Cited by 27
Zaeem, M. A., Thomas, S., Kavousi, S., Zhang, N., Mukhopadhyay, T., & Mahata, A. (2024) · 2D Materials · doi:10.1088/2053-1583/ad63b6 · Cited by 34
Self-Attention-Based Models for the Extraction of Molecular Interactions from Biological Texts
Srivastava, P. K., Bej, S., Yordanova, K., & Wolkenhauer, O. (2021) · Biomolecules · doi:10.3390/biom11111591 · Cited by 25
jurBERT: A Romanian BERT Model for Legal Judgement Prediction
Masala, M., Iacob, R. C. A., Uban, A. S., Cidota, M. A., Velicu, H., Rebedea, T. E., & Popescu, M. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.8 · Cited by 42
Rehana, H., Çam, N. B., Basmaci, M., Zheng, J., Jemiyo, C., He, Y., Özgür, A., & Hur, J. (2023) · arXiv.org · doi:10.48550/arxiv.2303.17728 · Cited by 47
“Even though there are many different BERT-based models that are either trained or fine-tuned on biomedical corpora, we have only selected the BioBERT [15], SciBERT [39], and PubMedBERT [13] ● SciBERT [39]: a BERT model pre-trained on random Semantic Scholar articles [40].”
MethodIntegrating domain knowledge for biomedical text analysis into deep learning: A survey
Cai, L., Li, J., Lv, H., Liu, W., Niu, H., & Wang, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104418 · Cited by 27
RevGNN: Negative Sampling Enhanced Contrastive Graph Learning for Academic Reviewer Recommendation
Liao, W., Zhu, Y., Li, Y., Zhang, Q., Ou, Z., & Li, X. (2024) · ACM Transactions on Information Systems · doi:10.1145/3679200 · Cited by 31
“We attribute this to its limited approach of extracting keywords from the vocabulary and constructing embeddings, coupled with inadequate similarity calculation methods.”
Wankhade, M., & Rao, A. C. S. (2022) · Scientific Reports · doi:10.1038/s41598-022-21604-7 · Cited by 28
“BERT comes in various flavors, including BioBERT 52 , sciBERT 53 , and others trained on domain-specific corpora.”
BackgroundPark, S., & Caragea, C. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.472 · Cited by 25
“Moreover, the domain-specific BERT models such as SciBERT (Beltagy et al., 2019), produce better results compared to a general domain-based BERT with respect to scientific knowledge required tasks such as scientific term classification e.g., 64.57% F1 on the SciIE dataset (Luan et al., 2018).”
BackgroundMethodInfluentialH2CGL: Modeling dynamics of citation network for impact prediction
He, G., Xue, Z., Jiang, Z., Kang, Y., Zhao, S. X., & Lu, W. P. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103512 · Cited by 30
The Automatic Detection of Dataset Names in Scientific Articles
Heddes, J., Meerdink, P., Pieters, M., & Marx, M. (2021) · Data · doi:10.3390/data6080084 · Cited by 33
“The model configuration and architecture are the same as those in the SciBERT paper [15].”
BackgroundMethodInfluentialKnowledge-Enhanced Graph Topic Transformer for Explainable Biomedical Text Summarization
Xie, Q., Tiwari, P., & Ananiadou, S. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3308064 · Cited by 33
“Cai et al. [26] incorporated word co-occurrence information of input documents to improve the SciBERT [27] encoder, for abstractive summarization of COVID-19 literature.”
MethodDeep learning and embeddings-based approaches for keyphrase extraction: a literature review
Giarelis, N., & Karacapilidis, N. I. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02164-w · Cited by 29
RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
Jiang, W., Subramanian, S., Graves, C., Alonso, G., Yazdanbakhsh, A., & Dadu, V. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731093 · Cited by 56
“The rapid adoption of Large Language Models (LLMs) across diverse applications—spanning question answering [22, 24, 26], code generation [65, 68, 78], and scientific discoveries [19, 59, 91]— showcases their profound impact on automating knowledge-based tasks.”
Predictive Chemistry Augmented with Text Retrieval
Qian, Y., Li, Z., Tu, Z., Coley, C. W., & Barzilay, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.784 · Cited by 35
“In addition, TextReact benefits from the SciBERT checkpoint, which has been pretrained on scientific text.”
MethodInfluentialLongitudinal Citation Prediction using Temporal Graph Neural Networks
Holm, A. N., Plank, B., Wright, D., & Augenstein, I. (2020) · VBN Forskningsportal (Aalborg Universitet) · doi:10.48550/arxiv.2012.05742 · Cited by 25
Generating Realistic Tabular Data with Large Language Models
Nguyen, D., Gupta, S., Do, K., Nguyen, T., & Venkatesh, S. (2024) · Proceedings · doi:10.1109/icdm59182.2024.00040 · Cited by 36
“Some methods are based on other generative models e.g. TVAE [44] uses Variational Autoencoder (VAE) [22], Great [7] and TapTap [48] use Large Language Models (LLM) [5].”
Advances in Multi-turn Dialogue Comprehension: A SurveyPreprint
Zhang, Z., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04984 · Cited by 29
Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical StudyPreprint
Nadkarni, R., Wadden, D., Beltagy, I., Smith, N. A., Hajishirzi, H., & Hope, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09700 · Cited by 31
Self-citationFayyazi, R., Taghdimi, R., & Yang, S. J. (2024) · arXiv · doi:10.48550/arxiv.2401.00280 · Cited by 25
CitePrompt: Using Prompts to Identify Citation Intent in Scientific Papers
Lahiri, A., Sanyal, D. K., & Mukherjee, I. (2023) · 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl57899.2023.00017 · Cited by 24
“Beltagy et al. [2] use SciBERT, a BERT-type model [6] trained on scientific data and fine-tuned for several downstream scientific tasks including citation intent classification. al.”
InfluentialPapadopoulos, D., Papadakis, N. K., & Λίτκε, Α. (2020) · Applied Sciences · doi:10.3390/app10165630 · Cited by 25
Wu, Z., Liang, J., Zhang, Z., & Lei, J. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103683 · Cited by 26
Kumar, A., Ganesh, S., Gupta, D., & Kodamana, H. (2022) · Process Safety and Environmental Protection · doi:10.1016/j.cherd.2022.05.018 · Cited by 24
Kang, D., Head, A., Sidhu, R., Lo, K., Weld, D. S., & Hearst, M. A. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.22 · Cited by 24
Self-citationClinical trial search: Using biomedical language understanding models for re-ranking
Rybinski, M., Xu, J., & Karimi, S. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103530 · Cited by 32
Hu, G., Flexner, N., Tiscornia, M. V., & L’Abbé, M. R. (2023) · Nutrients · doi:10.3390/nu15194167 · Cited by 29
“…models in food-and nutrition-related tasks is relatively limited, these models have consistently demonstrated an expectational performance in tasks such as text classification, document summarization, question and answering, and generating interpretable explanations in a variety of domains [21–23].”
BackgroundLarge Language Models Trained on Equipment Maintenance Text
Abijith, P., Patidar, P., Nair, G., & Pandya, R. (2023) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/216336-ms · Cited by 25
Language Identification of Hindi-English tweets using code-mixed BERT
Ansari, M. Z., Beg, M. M. S., Ahmad, T., Khan, M. J., & Wasim, G. (2021) · IEEE International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC) · doi:10.1109/iccicc53683.2021.9811292 · Cited by 31
“In earlier works BERT has been adapted to a variety of domains, particularly those with their own complex vocabulary, such as the biomedical domain, legal domain, and financial document processing and scientific texts or publications [17-20].”
BackgroundGlobal Span Selection for Named Entity Recognition
Zaratiana, U., Khbir, N. E., Holat, P., Tomeh, N., & Charnois, T. (2022) · doi:10.18653/v1/2022.umios-1.2 · Cited by 27
A systematic review of multi-modal large language models on domain-specific applications
Li, S., Wong, K. W., Wang, G., & Duong, T.-T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11398-1 · Cited by 27
“Firstly, multi-modal LLMs fail to generalise to unseen environments due to the lack of high-quality annotated data (Nie et al. 2023).”
Investigating the Performance of Fine-tuned Text Classification Models Based-on Bert
Mohammadi, S., & Chapon, M. (2020) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-smartcity-dss50907.2020.00162 · Cited by 30
“SciBert [12] is introduced as a pre-trained language model based on Bert getting trained on a large scientific dataset to leverage performance on downstream scientific NLP tasks.”
BackgroundMethodGeoscience language models and their intrinsic evaluation
Lawley, C. J. M., Raimondo, S., Chen, T., Brin, L., Zakharov, A., Kur, D., Hui, J., Newton, G., Burgoyne, S. L., & Marquis, G. (2022) · Applied Computing and Geosciences · doi:10.1016/j.acags.2022.100084 · Cited by 24
Gong, Z., Zhou, K., Zhao, X., Sha, J. J., Wang, S., & Wen, J.-R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.408 · Cited by 34
Aftab, W., Apostolou, Z., Bouazoune, K., & Straub, T. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05902-7 · Cited by 28
Proximity dimensions and the emergence of collaboration: a HypTrails study on German AI research
Koopmann, T., Stubbemann, M., Kapa, M., París, M., Buenstorf, G., Hanika, T., Andreas Hotho, A., Jäschke, R., & Stumme, G. (2021) · Scientometrics · doi:10.1007/s11192-021-03922-1 · Cited by 24
Coreference Resolution in Research Papers from Multiple Domains
Brack, A., Müller, D. U., Hoppe, A., & Ewerth, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_6 · Cited by 25
Application of artificial intelligence and machine learning in drug repurposing
Ghandikota, S. K., & Jegga, A. G. (2024) · Progress in molecular biology and translational science · doi:10.1016/bs.pmbts.2024.03.030 · Cited by 28
Windisch, P. Y., Dennstädt, F., Koechli, C., Schröder, C., Aebersold, D. M., Forster, R. E., & Zwahlen, D. R. (2024) · Cureus · doi:10.7759/cureus.75748 · Cited by 37
“Using natural language processing (NLP) to extract data from biomedical research publications has garnered increasing interest, particularly with the advent of more powerful architectures, most notably large language models (LLMs) [1-3].”
Enhancing Legal Argument Mining with Domain Pre-training and Neural Networks
Zhang, G., Nulty, P., & Lillis, D. (2022) · Journal of Data Mining & Digital Humanities · doi:10.46298/jdmdh.9147 · Cited by 28
Multilingual Entity and Relation Extraction Dataset and Model
Seganti, A., Firląg, K., Skowrońska, H., Satława, M., & Andruszkiewicz, P. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.166 · Cited by 52
PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge
He, Y., Wang, Z., Zhang, Y., Huang, R., & Caverlee, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.611 · Cited by 30
Using large language models to extract plant functional traits from unstructured text
Domazetoski, V., Kreft, H., Bestová, H., Wieder, P., Koynov, R., Zarei, A., & Weigelt, P. (2025) · Applications in Plant Sciences · doi:10.1002/aps3.70011 · Cited by 26
“SciBERT : SciBERT (Beltagy et al., 2019) uses the same architecture as BERT but is trained on papers from the corpus of Semantic Scholar (https://www.semanticscholar. org/), consisting of 1.14 million papers and 3.1 billion tokens.”
InfluentialZhan, Z., Wang, J., Zhou, S., Deng, J., & Zhang, R. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf128 · Cited by 33
General-Purpose Models for the Chemical Sciences: LLMs and Beyond
Alampara, N., Aneesh, A., Ríos-García, M., Mirza, A., Schilling-Wilhelmi, M., Aghajani, A. A., Sun, M., Prastalo, G., & Jablonka, K. M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00583 · Cited by 37
Panayi, A., Ward, K., Benhadji-Schaff, A., Ibanez-Lopez, A. S., Xia, A., & Barzilay, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02351-w · Cited by 23
Document-level Relation Extraction via Separate Relation Representation and Logical Reasoning
Huang, H., Yuan, C., Liu, Q., & Cao, Y. (2023) · ACM Transactions on Information Systems · doi:10.1145/3597610 · Cited by 27
“[1] Washington Place was built byWilliam and Annie Washington in north Romney between 1863 and 1874 on land given to Annie by her former owner , Susan Blue Parsons of Wappocomo plantation.”
BackgroundMethodMartenot, V., Masdeu, V., Cupe, J., Gehin, F., Blanchon, M., Dauriat, J., Horst, A., Renaudin, M., Girard, P., & Zucker, J. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-02085-0 · Cited by 25
Attention-based Multi-level Feature Fusion for Named Entity Recognition
Yang, Z., Chen, H., Zhang, J., Ma, J., & Chang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/497 · Cited by 26
Proceedings of The 5th Workshop on BioNLP Open Shared Tasks
Kim, J.-D., Nédellec, C., Bossy, R., & Deléger, L. (2019) · doi:10.18653/v1/d19-57 · Cited by 29
Can language representation models think in bets?
Tang, Z., & Kejriwal, M. (2023) · Royal Society Open Science · doi:10.1098/rsos.221585 · Cited by 23
“Many technical and domain-specific variants of these models have also been developed and deployed, including SciBERT [20], BioBERT [17] and AlBERTa [86].”
MethodWang, P., Jiang, Y., Liao, W., Wang, R., Lai, M., Kou, H., Liang, X., Li, J., Lookman, T., & Yuan, R. (2025) · Acta Materialia · doi:10.1016/j.actamat.2025.121275 · Cited by 23
Sequence Generation with Label Augmentation for Relation Extraction
Li, B., Yu, D., Ye, W., Zhang, J., & Zhang, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26532 · Cited by 30
Hierarchical Multi-Label Classification of Scientific Documents
Sadat, M., & Caragea, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.610 · Cited by 38
Will AI solve the patent classification problem?
Kamateri, E., Salampasis, M., & Perez‐Molina, E. (2024) · World Patent Information · doi:10.1016/j.wpi.2024.102294 · Cited by 28
Invernici, F., Curati, F., Jakimov, J., Samavi, A., & Bernasconi, A. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01189-4 · Cited by 28
“In the figure, the five largest topics for each area are connected to their corresponding word-clouds.”
Accurate Medical Named Entity Recognition Through Specialized NLP Models
Hu, J., Bao, R., Lin, Y., Zhang, H., & Xiang, Y. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10912885 · Cited by 23
“Precision% F1-score Bert [23] 82.5 81.0 ClinicalBERT [24] 85.2 83.5 SciBert [25] 84.1 82.8 BlueBert [26] 87.3 85.0 BioBert [27] 89.8 87.6 Through the experimental results, we can see that BioBERT performs best in the task of medical named entity recognition, surpassing other models in precision and…”
InfluentialPasini, M. L., Choi, J. Y., Mehta, K. R., Zhang, P., Rogers, D., Bae, J., Ibrahim, K. Z., Aji, A. M., Schulz, K. W., Polo, J., et al. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07029-9 · Cited by 25
UDAPTER - Efficient Domain Adaptation Using Adapters
Malik, B., Kashyap, A. R., Kan, M., & Poria, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.165 · Cited by 23
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.