SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,001–2,100 of 3,596
Pretrained Models with Adversarial Training for Named Entity Recognition in Scientific Text
Ma, H., Zhang, Y., & Wang, J. (2022) · International Conference on Asian Language Processing · doi:10.1109/ialp57159.2022.9961309 · Cited by 4
“The experimental results are shown in Table IV, SciBERT baseline model achieved the highest F1-score of 84.39%.”
MethodInfluentialMORTY: Structured Summarization for Targeted Information Extraction from Scholarly Articles
Jaradeh, M. Y., Stocker, M., & Auer, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21756-2_23 · Cited by 4
“BERT [7] (scholarly counterpart SciBERT [1]) are some of the most commonly used transformer models capable to automatically summarize text.”
BackgroundFine-Tuning BERT for Question and Answering Using PubMed Abstract Dataset
Cheon, S., & Ahn, I. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference · doi:10.23919/apsipaasc55919.2022.9980097 · Cited by 4
“In addition, BERT models specialized for each domain were developed, for example, BioBERT [9], SciBERT [10], ClinicalBERT [11], LEGAL-BERT [12], FinBERT [13].”
MethodTokala, Y. S. S. S., Aluru, S. S., Vallabhajosyula, A., Sanyal, D. K., & Das, P. P. (2023) · Expert Systems · doi:10.1111/exsy.13238 · Cited by 7
Natural Language Explanatory Arguments for Correct and Incorrect Diagnoses of Clinical Cases
Marro, S., Molinet, B., Cabrio, E., & Villata, S. (2023) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0011927000003393 · Cited by 7
Role of Punctuation in Semantic Mapping Between Brain and Transformer Models
Lamprou, Z., Pollick, F. E., & Moshfeghi, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25891-6_35 · Cited by 8
The Use of Pretrained Model for Matching App Reviews and Bug Reports
Wang, X., Zhang, W., Lai, S., Ye, C., & Zhou, H. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00034 · Cited by 5
“For example, SciBERT [4], BioBERT [15], Clinical-BERT [1], PatentBERT [16], FinBERT [2], He et al. injected disease-related knowledge into BERT and achieved better results in three tasks: consumer health quiz, medical language inference, and disease name recognition [12].”
MethodA Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks
Ding, R., Han, X., & Wang, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.24 · Cited by 7
“Besides task-dependent methods, there are also some unified training strategies to incorporate knowledge (domain-specific corpora) into PLMs, leading to domain-specific PLMs such as BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019), ClinicalBERT (Alsentzer et al., 2019), and UmlsBERT (Michalopoulos et al., 20…”
BackgroundMethodInfluentialQaDialMoE: Question-answering Dialogue based Fact Verification with Mixture of Experts
Wang, L., Zhang, P., Lu, X., Zhang, L., Yan, C., & Zhang, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.229 · Cited by 6
Using citation networks to evaluate the impact of text length on keyword extraction
Tohalino, J. A. V., Silva, T. C., & Amancio, D. R. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294500 · Cited by 5
Zhang, T., Xu, R., Wang, C., Duan, Z., Chen, C., Qiu, M., Cheng, D., He, X., & Qian, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.969 · Cited by 7
Wang, M., He, X., Liu, L., Fang, Q.-M., Zhang, M., Chen, H., & Liu, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3368288 · Cited by 5
“The SciBERT [58] leverages unsupervised pretraining on a huge corpus of scientific papers across multiple domains to increase performance on subsequent scientific NLP tasks.”
BackgroundMethodNeural Transformers for Bias Detection: Assessing Pakistani News
Ali, O., Zaland, Z., Bazai, S. U., Ghafoor, M. I., Hussain, L., & Haider, A. (2024) · International Conference on Advancements in Computational Sciences (ICACS) · doi:10.1109/icacs60934.2024.10473263 · Cited by 4
Mo, Y., & Li, Z. (2024) · Electronics · doi:10.3390/electronics13071407 · Cited by 5
“We processed it following the works [60–62].”
MethodFinDeBERTaV2: Word-Segmentation-Free Pre-trained Language Model for Finance
Suzuki, M., Sakaji, H., Hirano, M., & Izumi, K. (2024) · Transactions of the Japanese Society for Artificial Intelligence · doi:10.1527/tjsai.39-4_fin23-g · Cited by 4
Low-resource multi-granularity academic function recognition based on multiple prompt knowledge
Liu, J., Xiong, Z., Jiang, Y., Ma, Y., Lu, W., Huang, Y., & Cheng, Q. (2024) · The Electronic Library · doi:10.1108/el-01-2024-0022 · Cited by 4
“Previous methods on academic function recognition tasks can mainly be summarized in three categories: (1) a machine learning method based on hand-crafted features [29, 41, 45]; (2) a deep learning model trained from scratch [6, 7, 49, 63]; (3) a PLM fine-tuning method [2, 22, 42, 72].”
BackgroundMethodInfluentialExploring Optimal Transport-Based Multi-Grained Alignments for Text-Molecule Retrieval
Min, Z., Liu, B., Zhang, L., Song, J. T., Su, J., He, S., & Bo, X. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822800 · Cited by 4
A New Neural Search and Insights Platform for Navigating and Organizing AI Research
Fadaee, M., Gureenkova, O., Barrera, F. R., Schnober, C., Weerkamp, W., & Zavrel, J. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.23 · Cited by 8
Do sentence embeddings capture discourse properties of sentences from Scientific Abstracts ?
Huber, L., Memmadi, C., Dargnat, M., & Toussaint, Y. (2020) · Workshop on Computational Approaches to Discourse (CODI) · doi:10.18653/v1/2020.codi-1.9 · Cited by 4
SDCF: semi-automatically structured dataset of citation functions
Basuki, S., & Tsuchiya, M. (2022) · Scientometrics · doi:10.1007/s11192-022-04471-x · Cited by 6
“Using BERT and SciBERT, AL is used in the filtering and fine-grained stages.”
MethodInfluentialApplying Lexical Link Analysis to Discover Insights from Public Information on COVID-19Preprint
Zhao, Y., & Zhou, C. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.05.06.079798 · Cited by 4
Information retrieval in an infodemic: the case of COVID-19 publicationsPreprint
Teodoro, D., Ferdowsi, S., Borissov, N., Kashani, E., Alvarez, D. V., Copara, J., Gouareb, R., Naderi, N., & Amini, P. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.01.29.428847 · Cited by 4
Hematialam, H., & Zadrozny, W. (2021) · Research Square · doi:10.21203/rs.3.rs-500521/v1 · Cited by 5
“1 [31], SciBERT scivocab-uncased [46], as well as BlueBERT base-PubMed and base-PubMed+MIMIC-III [47].”
BackgroundEncoding Explanatory Knowledge for Zero-shot Science Question AnsweringPreprint
Zhou, Z., Valentino, M., Landers, D., & Freitas, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05737 · Cited by 4
“BERT-based models have demonstrated remarkable results in Question Answering when directly fine-tuned on the answer prediction task or additionally pre-trained using domain specific knowledge (Clark et al., 2020; Beltagy et al., 2019).”
MethodOn the Universality of Deep Contextual Language ModelsPreprint
Bhatt, S., Goyal, P., Dandapat, S., Choudhury, M., & Sitaram, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07140 · Cited by 5
Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource SettingPreprint
Lange, L., Adel, H., & Strötgen, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08597 · Cited by 4
“, pretraining with masked language modeling (MLM) objectives on documents from the target domain [1, 7].”
A BERT-Based Hybrid System for Chemical Identification and Indexing in Full-Text ArticlesPreprint
Erdengasileng, A., Li, K., Han, Q., Tian, S., Wang, J., Hu, T. W., & Zhang, J. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.10.27.466183 · Cited by 5
“In BioCreative VII challenge, we first experimented with several BERT-based models, including BERT large uncased (5), BioBERT (7), BlueBERT (8), SciBERT (6), ClinicalBERT (9), and PubMedBERT (10).”
MethodWiki to Automotive: Understanding the Distribution Shift and its impact on Named Entity RecognitionPreprint
Nayak, A., & Timmapathini, H. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.00283 · Cited by 4
“, 2019) and SciBERTBase-Scivocab-Uncased (Beltagy et al., 2019) have lead to interesting findings that showed: 1) The performance of SciBERT is better than BERT when used for automotive domain, 2) Fine-tuning the language models with automotive domain text did not make significant improvements to the NER performance,…”
MethodInfluentialZhang, J., Sharma, A., Bolanos, L. D., Li, T., Tanwar, A., Gupta, V., & Guo, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08891 · Cited by 4
Understanding Telecom Language Through Large Language ModelsPreprint
Bariah, L., Zou, H., Zhao, Q., Mouhouche, B., Bader, F., & Debbah, M. A. (2023) · doi:10.36227/techrxiv.23501271 · Cited by 4
Xu, T., Smirnova, N., Upadhyaya, S., Yu, R., Culbert, J. H., Sun, C., Otto, W., & Mayr, P. (2023) · doi:10.2196/preprints.54346 · Cited by 4
BioEmoDetector: A flexible platform for detecting emotions from health narratives
Alshouha, B., Serrano‐Guerrero, J., Chiclana, F., Romero, F. P., & Olivas, J. Á. (2024) · SoftwareX · doi:10.1016/j.softx.2024.101670 · Cited by 5
“This innovative tool employs a hybrid model that integrates several PLMs such as CODER [28], Clinical-Longformer [29], BlueBERT [30], SciBERT [31], BioMed-RoBERTa [32], Bio-ClinicalBERT [33], and BioBERT [34] to enhance the prediction accuracy.”
Thillainathan, S., Ranathunga, S., Yuan, S., Jayasena, S., & Lee, E.-S. A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5061381 · Cited by 4
Annotated corpus for traditional formula-disease relationships in biomedical articles
Yea, S.-J., Jang, H., Kim, S. Y., Lee, S., & Kim, J. U. (2025) · Scientific Data · doi:10.1038/s41597-025-04377-2 · Cited by 4
Lamprou, Z., Pollick, F. E., & Moshfeghi, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.06278 · Cited by 5
Drechsel, J., Reusch, A., & Herbold, S. (2025) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2502.20855 · Cited by 2
“…models (Vaswani et al., 2017), such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), led to the development of domain-specific models, including SciBERT for scientific texts (Beltagy et al., 2019) and CodeBERT for programming (Feng et al., 2020).”
SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature RetrievalPreprint
Shao, C., Xu, F., & Li, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.03260 · Cited by 4
Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific IdeasPreprint
Liu, M., Shen, Y., Zhou, D., & Huang, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.08901 · Cited by 5
Optimized deep learning-based multimodal method for irregular medical timestamped data
Rabhi, S. (2022) · theses.fr (ABES) · doi:10.70675/154ab19dz373dz4f8az9c16zaba9b75e66ed · Cited by 5
Enriching large language models with semantic lexicons and analogies
Zervakis, G. (2023) · theses.fr (ABES) · doi:10.70675/4f4cd0e5z5987z422az813bzc6e431d1a918 · Cited by 7
Dhrangadhariya, A., & Müller, H. (2022) · Proceedings of the 21st Workshop on Biomedical Language Processing · doi:10.18653/v1/2022.bionlp-1.34 · Cited by 3
“…towards pharma intervention classes overshadowing non-pharma ones leading to a substandard performance on it in the previous SOTA fully-supervised PICO entity/span recognition models (Beltagy et al., 2019; Brockmeier et al., 2019; Zhang et al., 2020) and weakly supervised model (Liu et al., 2021).”
MethodSemantically Orthogonal Framework for Citation Classification: Disentangling Intent and Content
Duan, C., & Tan, Z. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-05409-8_12 · Cited by 3
“We employed two open-source, state-of-the-art CIC models with distinct architectures: – CitePrompt [32]: A prompt-based fine-tuning framework built on SciB-ERT [10], implemented via the OpenPrompt toolkit [19].”
Multiple Evidence Combination for Fact-Checking of Health-Related Information
Deka, P., Jurek, A., & Deepak, P. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.20 · Cited by 16
An End-to-End Pipeline for Bibliography Extraction from Scientific Articles
Joshi, B., Symeonidou, A., Danish, S. M., & Hermsen, F. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.12 · Cited by 9
Custom Natural Language Understanding for Healthcare Chatbots and A Case Study
Inupakutika, D., Akopian, D., Gunnam, G. R., Chalela, P., Kaghyan, S., & Mundlamuri, R. (2024) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh62654.2024.00028 · Cited by 4
“Of the existing domain-specific BERT models available publicly online, in the current work, we utilize SciBERT [48], a variant of BERT pre-trained on specialized corpora consisting of a random sample of 1.14M Semantic Scholar full-text papers [25].”
MethodA Deep Learning Framework for Classifying and Mitigating Bias in News Reporting
Mitra, A., Das, R., Paul, J., Majumder, R., Saha, A., Sengupta, N., & Sil, J. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-73125-9_27 · Cited by 3
Biomedical document-level relation extraction with thematic capture and localized entity pooling
Li, Y., & Shao, X.-H. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104756 · Cited by 4
A Framework for Training and Deploying Foundational Language Models for Embedded Sensing
Kandala, S. V., & Varshney, A. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3695901 · Cited by 3
“When a tunnel diode-based sensor is kept close to a subject at a high level, the person’s breathing causes movement of the chest and other body parts.”
Yamauchi, K., & Katsurai, M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0868-3_5 · Cited by 3
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 6
Exploring Social Media Research Trends in Malaysia using Bibliometric Analysis and Topic Modelling
Tariq, R., Isawasan, P., Shamugam, L., Asmawi, M. A. H. A., Azman, N. A. N., & Zolkepli, I. A. (2025) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.7003 · Cited by 4
“The uncased version of SciBERT is chosen because it simplifies the model and increases its effectiveness by reducing the number of unique words it needs to handle [18].”
Re-SciBERT: An Entity-Enriched Language Model to Enhance Biomedical Relation Extraction
Aid, A., Azzoune, R., Abdellaoui, I., & Haddouche, A. (2024) · International Conference on Innovative and Intelligent Information Technologies (IC3IT) · doi:10.1109/ic3it63743.2024.10869408 · Cited by 4
“SciBERT, introduced by Beltagy et al. [3], is another domain-specific pre-trained language model designed for scientific texts, particularly in biomedical and computer science fields.”
Xi, H., Zhang, C., & Zhang, H. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702489 · Cited by 3
Peng, Y., Yu, G., Shi, R., Chen, L., Wang, X., Du, W., Huo, X., & Yang, Y. (2025) · Artificial Intelligence Chemistry · doi:10.1016/j.aichem.2025.100091 · Cited by 3
Dasgupta, D., Mondal, A., & Chakrabarti, P. P. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-8912-5_5 · Cited by 4
FoodPuzzle: Toward Developing Large Language Model Agents as Autonomous Flavor Scientists
Huang, T., Lee, D. H., Sweeney, J., Shi, J., Steliotes, E., Lange, M. C., May, J., & Chen, M. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737384 · Cited by 3
“…and biomedical domains have demonstrated significant performance improvements in tasks such as protein classification and text mining, un-derscoring the value of tailored pretraining for scientific corpora (Lee et al. 2019a; Vig et al. 2021; Beltagy et al. 2019; Gu et al. 2021; Taylor et al. 2022).”
Zhang, Y., Yu, J., Li, G., He, Z., & Yen, G. G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3596257 · Cited by 3
“By continued pretraining [3], [4] or training from scratch [5], [6] on the BERT architecture, biomedical pretrained language models (PLMs) have exhibited strong capabilities in a variety of biomedical tasks including sentence similarity, RE, and named entity recognition (NER).”
InfluentialFine-grained multimodal molecular pretraining via prompt learning
Li, Y., Wei, Z., Liu, C., & Wang, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114381 · Cited by 5
Active Knowledge Structuring for Large Language Models in Materials Science Text Mining
Zhang, X. Q., Yuan, J., Zhang, P., Liu, J., & Li, L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.36 · Cited by 4
Aksoy, M., Yanık, S., & Amasyalı, M. F. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200587 · Cited by 4
Emerging topics detection using motif-based analysis of term citation networks
Wang, W., Lu, M., & Huang, X. (2025) · Scientometrics · doi:10.1007/s11192-025-05434-8 · Cited by 3
“The ETs are encoded with SciBERT, reduced in dimensionality with Umap, and automatically clustered with Hdbscan (Beltagy et al., 2019; McInnes et al., 2018; Campello et al., 2013).”
Khadka, A., & Sthapit, S. (2025) · Informatics · doi:10.3390/informatics12040108 · Cited by 3
“SciBERT is a domain-specific BERT model trained on 1.14M full-text papers from Semantic Scholar [216].”
Albarrak, M., Alqudhaibi, A., & Jagtap, S. (2025) · International Journal of Information Security · doi:10.1007/s10207-025-01146-5 · Cited by 3
“The proposed model is compared against a range of established baseline models, including domain-specific language models, which are: SecBERT [59], SecureBERT [29], TTPHunter [69], and TTPXHunter [55], as well as general-purpose models like DistilBERT [71] and SciBERT [44].”
FLeW: Facet-Level and Adaptive Weighted Representation Learning of Scientific Documents
Dou, Z., Wang, D., Zhuang, F., Ren, J., & Hu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-3827-0_22 · Cited by 3
Athena: A Plug-and-Play Advisor for Retrieval-Augmented Generation using VectorDB
Liang, N., Wenz, F., Giceva, J., & Wills, L. W. (2025) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc66894.2025.00013 · Cited by 4
“Large Language Models (LLMs) have demonstrated remarkable success in a wide range of natural language processing tasks, including question answering, conversational agents, machine translation, summarization, and code generation [4], [7], [9], [12], [30], [33], [36], [50], [58].”
Yaacoub, J.-P. A., Noura, H., Salman, O., & Pujolle, G. (2025) · Annals of Telecommunications · doi:10.1007/s12243-025-01134-9 · Cited by 4
“In scientific research, LLMs accelerate literature reviews, summarise key findings, and identify thematic connections across large corpora [21].”
Large Language Models and Machine Learning Framework for Predicting Dental Ceramics Performance
Zhou, H., Bai, Y., Chen, Y., Fan, D., Wang, P., Ji, P., & Chen, T. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.109358 · Cited by 5
AI Agent for Scientific Paper Analysis
Aware, V. S., Pimparkar, A., Unavane, C., Wagh, A., Pandit, P., & Yenkikar, A. (2025) · IEEE International Conference on Intelligent Signal Processing and Effective Communication Technologies (INSPECT) · doi:10.1109/inspect67393.2025.11350774 · Cited by 3
“T5, on the other hand, shows competing performance for summary metrics through remapping NLP tasks into text-to-text format [6].”
Yutong, G. U. O., & Yang, C. (2026) · Metals · doi:10.3390/met16020162 · Cited by 5
“Evaluation here centers on the model’s ability to translate high-level natural language descriptions into syntactically correct and functionally appropriate code or configuration files [138].”
Goswami, P., Raman, R. K. S., & Alankar, A. (2026) · Small Structures · doi:10.1002/sstr.202500720 · Cited by 3
“Domain-specific language models such as SciBERT [316] further enhance the accuracy of this process in the context of technical literature.”
Wang, Z., Li, X., Rui, X., Ding, L., & Li, J. (2026) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi15030124 · Cited by 3
“Fine‑tuning refers to the process of adapting a LLM to acquire additional knowl‑ edge not covered during its initial training, thereby improving its performance in new domains [45].”
Mousavi, R., Kitchens, B., Oliver, A. G., & Abbasi, A. (2026) · Information Systems Research · doi:10.1287/isre.2024.1143 · Cited by 12
Fairness, equity, and equality in humanitarian logistics: a systematic review analysis
Dik, N. Y., Lo, C. Y., & Lai, K.-H. (2026) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2026.104927 · Cited by 3
Hao, J., & Zhang, C. (2026) · Scientometrics · doi:10.1007/s11192-026-05652-8 · Cited by 3
“Each input text chunk is then embedded using SciBERT (Beltagy et al., 2019) to represent it as a 768 dimensional vector.”
UNSEEN BIAS: HOW AI SHAPES RELIGIOUS NARRATIVES IN DIGITAL SPACES
Nurjaman, H., Siregar, A. F. Y., & Yasinta, J. R. E. (2025) · Jurnal Middle East and Islamic Studies (MEIS) · doi:10.7454/meis.v12i2.200 · Cited by 4
Large language models unlock large text corpora in the search for data on medicinal plants and fungi
Richard‐Bollans, A., Civita, F., Cossu, T. A., Saifi, I., Patmore, K., Wilkinson, C., & Allkinꝉ, R. (2026) · Plants People Planet · doi:10.1002/ppp3.70250 · Cited by 3
Mol-CADiff: text-conditional molecule generation via causality-aware autoregressive diffusion
Ahamed, M. A., Ye, Q., & Cheng, Q. S. (2026) · Nature Communications · doi:10.1038/s41467-026-75702-5 · Cited by 3
Antibody Watch: Text mining antibody specificity from the literature
Hsu, C., Chang, C., Poopradubsil, T., Lo, A., William, K. A., Lin, K., Bandrowski, A. E., Özyurt, İ. B., Grethe, J. S., & Martone, M. E. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008967 · Cited by 3
FrBMedQA: the first French biomedical question answering dataset
Kaddari, Z., & Bouchentouf, T. (2022) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v11.i4.pp1588-1595 · Cited by 6
Race, Gender, and Age Biases in Biomedical Masked Language Models
Kim, M. J., Kim, J., & Johnson, K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.749 · Cited by 3
Miah, M. S. U., Sulaiman, J. B., Sarwar, T. B., Ferdous, I. U., Islam, S. S., & Haque, M. S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39821-6_16 · Cited by 3
Integrating Linguistic and Citation Information with Transformer for Predicting Top-Cited Papers
Ochi, M., Shiro, M., Mori, J., & Sakata, I. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-43088-6_7 · Cited by 3
Applying unsupervised keyphrase methods on concepts extracted from discharge sheets
Memarzadeh, H., Ghadiri, N., Samwald, M., & Shahreza, M. L. (2023) · Pattern Analysis and Applications · doi:10.1007/s10044-023-01198-0 · Cited by 4
“The KeyBERT [20] algorithm, proposed by Grootendorst, utilizes pre-trained BERT-based word embedding models to enhance the quality of key-phrase extraction.”
BackgroundMethodEconBERTa: Towards Robust Extraction of Named Entities in Economics
Lasri, K., de Castro, P. V. Q., Schirmer, M., Martin, L. E. S., Wang, L., Dulka, T., Naushan, H., Pougué-Biyong, J., Legovini, A., & Fraiberger, S. P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.774 · Cited by 6
RADM-DRE:Retrieval Augmentation for Document-Level Relation Extraction with Diffusion Model
Zhang, Q., Yuan, Q., Duan, J., Jiang, Y., Wang, H., Gao, Z., He, L., & Liu, J. (2023) · International Conference on Asian Language Processing · doi:10.1109/ialp61005.2023.10337090 · Cited by 3
“We use cased BERT base [14] as an encoder on DocRED and SciBERT [15] on CDR and GDA.”
MethodYuan, Z. P., Cai, R., Maimaiti, M., & Abiderexiti, K. (2023) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml59573.2023.10348182 · Cited by 3
Xie, Z. Y., Lv, Y., Song, Y., & Wang, Q. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101498 · Cited by 3
Klemm, C. P., & Frömling, T. (2024) · Journal of Computational Chemistry · doi:10.1002/jcc.27334 · Cited by 3
Intermediate-Task Transfer Learning for Peer Review Score Prediction
Muangkammuen, P., Fukumoto, F., Li, J., & Suzuki, Y. (2023) · doi:10.18653/v1/2023.ijcnlp-srw.6 · Cited by 7
Examining between-sectors knowledge transfer in the pharmacology field
Syafiandini, A. F., Yoon, J., Lee, S., Song, C., Yan, E., & Song, M. (2024) · Scientometrics · doi:10.1007/s11192-024-05040-0 · Cited by 3
“Recent studies fine-tuned domain-specific language models such as BioBERT (Lee et al., 2019), SciBERT (Beltagy et al., 2019), and BioMedNLP-PubMedBERT (Gu et al., 2021) by training them on customized datasets to perform NER tasks.”
Dielectric Ceramics Database Automatically Constructed by Data Mining in the Literature
Wang, X., Zhang, W., & Zhang, W. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00282 · Cited by 3
Deep Sketched Output Kernel Regression for Structured Prediction
Ahmad, T. E., Yang, J., Laforgue, P., & d’Alché–Buc, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70352-2_6 · Cited by 5
Song, C., Zeng, Z., Tian, C., Li, K., Yao, Y., Zheng, S., Liu, Z., & Sun, M. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.002 · Cited by 3
BATRACIO: BAsic, TRAnslational, Clinical, Research Phase Identification in BiOmedical Publications
Durán-Silva, N., Carrillo‐de‐Albornoz, J., Plaza, L., Ricardo, S., Massucci, F. A., Veiga, S. R., & Ramos-Prats, A. (2024) · IEEE Access · doi:10.1109/access.2024.3463717 · Cited by 5
Gray, M., Milanova, M. G., & Wu, L. (2024) · JMIR Medical Informatics · doi:10.2196/60272 · Cited by 3
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 5
DPN: Dynamics Priori Networks for Radiology Report Generation
Yang, B., Lei, H., Huang, H., Han, X., & Cai, Y. (2024) · Tsinghua Science & Technology · doi:10.26599/tst.2023.9010134 · Cited by 3
GeomCLIP: Contrastive Geometry-Text Pre-training for Molecules
Xiao, T., Cui, C., Zhu, H., & Honavar, V. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822346 · Cited by 3
“The predictor customizes molecular representation for distinct pretraining tasks while sharing the previous backbone encoder, and acts”
Document-level Biomedical Relation Extraction Based on Relation-guided Entity-level Graphs
Gao, L., Yang, Z., Tan, H., Wang, L., Liu, W., Zhang, Y., Luo, L., Xu, B., Wang, J., Yang, Y., et al. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822828 · Cited by 3
“Using the SciBERT [9] encoder enhances semantic representation with deep Transformer layers.”
A homogeneous multimodality sentence representation for relation extraction
Wang, K., Chen, Y., Yang, W., & Qin, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102968 · Cited by 5
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.