SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,901–3,000 of 3,596
NeurSEG: A Segment Driven Deep Neural Model for Nested Named Entity Recognition
Wang, Z., Li, F., & Long, C. (2023) · doi:10.1145/3607827.3616837
Authorship attribution of small messages
Júnior, A. C. T. C. (2022) · doi:10.47749/t/unicamp.2022.1254885
ReviVal: Towards Automatically Evaluating the Informativeness of Peer Reviews
Verma, R., Ghosal, T., Bhattacharjee, S., Ekbal, A., & Bhattacharyya, P. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625341 · Cited by 1
“We also normalize the Section Exhaustiveness , Aspect Exhaustiveness , and Review Strength scores to lie in the range [1,10].”
MethodInfluentialZhang, Z., Yang, J., & Liu, H. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8184-7_23
Bora, K., Baishya, N. J., Talukdar, C., Jain, D., & Borah, M. D. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-3481-2_50
Identifying Influential References in Scholarly Papers Using Citation Contexts
Ikoma, T., & Matsubara, S. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8088-8_13
An efficient word-embedding training procedure using smart batching for biomedical applications
Mohammed, A. H., & Ali, A. H. (2023) · AIP conference proceedings · doi:10.1063/5.0165234
DREAM: Deployment of Recombination and Ensembles in Argument Mining
Ruosch, F., Sarasua, C., & Bernstein, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.320 · Cited by 3
Rafif, S., & Perdana, R. S. (2023) · International Conference on Sustainable Information Engineering and Technology (SIET) · doi:10.1145/3626641.3626927
“In the preprocessing stage, tokenizing, stemming, stopword removal, case folding are applied to the source and target domain using the SciBERT Pre-Trained Model [3].”
Enhancing Relation Extraction for Long Documents with Prompt Decomposition and Multi-task Learning
Jo, S., & Oh, H. (2023) · Journal of Digital Contents Society · doi:10.9728/dcs.2023.24.12.3131
NLP Meets Agronomy: Document Classification for Plant Health Surveillance
Reina, L. A. V., & Bossy, R. (2023) · HAL
Innovation points mining from scientific papers with deep learning models
Yu, X., Xie, B., Lyu, X., & Liu, X. (2023) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir59294.2023.10391256
“Specifically, the proposed classification model in this section is based on the pre-trained language model Sci-BERT [23], which performs remarkably on various text classification tasks.”
MethodEntity Fusion Contrastive Inference Network for Biomedical Document Relation Extraction
Cai, H., Yuan, J., Sang, G., Liu, Z., Lin, H., & Zhang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-99-9864-7_10
Uncovering Research Potentials: Research Areas Evolution Analysis in Scientific Articles
Figueiredo, L., & da Silva, L. A. (2023) · Congresso Brasileiro de Inteligência Computacional (CBIC) · doi:10.21528/cbic2023-127
Improving Peer-Review Score Prediction via Pretrained Model with Intermediate-Task Training
Muangkammuen, P., Fukumoto, F., Li, J., & Suzuki, Y. (2024) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom60618.2024.10418356
“S CI B ERT is a pretrained language model based on the Transformer architecture [15].”
MethodLiyanage, V. P., & Buscaldi, D. (2023) · HAL
CovPTM : COVID-19 PreTrained Model Generic Model vs Biomedical Model Adaptation
Mercy Faustina, J., Raghuvir, A., & Sivaselvan, B. (2023) · IEEE India Conference · doi:10.1109/indicon59947.2023.10440931
“Many bio-medical domain specific models have already been proposed which are built on top of successful pretrained models like BERT [5], RoBERTa [10], ELECTRA [4] etc. Models such as SciBERT [2], BioBERT [9] are improvised versions of BERT that are adapted to scientific domain.”
MethodSome Useful Things to Know When Combining IR and NLP: The Easy, the Hard and the Ugly
Alonso, O., & Church, K. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3636452 · Cited by 1
Le traitement automatique des langues face à l'évolution des usages de la langue
Grouin, C. (2023) · HAL
SPECTER-Based Transformer Model For Multi-Label Research Paper Classification
Lahoti, K., Ahuja, V., & Patil, A. (2023) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon58714.2023.10450144
“To mitigate this problem, the research focused on pre-trained models like SCIBERT [21] for word-level embeddings and SPECTER [17] for document-level embeddings, as they are specifically trained on scientific datasets and understand scientific terms.”
MethodRelation Classification in Scientific Article Abstracts using SciBERT with Entity Marker
Inayah, N., Liebenlito, M., Fitriyati, N., Manaqib, M., & Aryanti, N. (2023) · International Conference on Cyber and IT Service Management (CITSM) · doi:10.1109/citsm60085.2023.10455639
“Furthermore, this study surpassed the performance of prior research [8].”
MethodResultInfluentialExtracting Definienda in Mathematical Scholarly Articles with Transformers
Jiang, S., & Senellart, P. P. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.4
Graph-Enriched Biomedical Language Models: A Research Proposal
Sakhovskiy, A., Panchenko, A., & Tutubalina, E. V. (2023) · doi:10.18653/v1/2023.ijcnlp-srw.11
Reddy, T. B. P., & Vydeki (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-9037-5_22
Chen, B., Zhu, J., Liu, Y., Zeng, B., Pan, J., & Ding, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447344
“In the CLS task, for Text-Only RoBERTa [10] 75.40 SciBERT [22] 77.70 Multi-Modal SAN [23] 72.30 NLF [13] 76.60 M3AE* [3] 77.80 Ours 79.94 Med-ITC, we compare with various methods, among which the multimodal methods SAN [23], NLF [13] M3AE [3], our method achieves better results on previously unseen…”
MethodInfluentialBCC: Bidirectional Consistency Constraint Method for Hierarchical Text Classification
Shen, Y., Yan, Y., Yin, D., & Shen, H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447200
Cross-lingual Information Extraction for the Assessment and Prevention of Adverse Drug Reactions
Raithel, L. (2024) · theses.fr (ABES) · doi:10.70675/5478ceccz18cdz4fcbzb106z2fb1ec257de2 · Cited by 3
Measuring the Commercial Potential of SciencePreprint
Masclans, R., Hasan, S., & Cohen, W. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4771238 · Cited by 1
Exploiting Graph Embeddings from Knowledge Bases for Neural Biomedical Relation Extraction
Tang, A., Deléger, L., Bossy, R., Zweigenbaum, P., & Nédellec, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_28
HTGAT: A Heterogeneous Temporal Graph Attention Network for Scholars’ Impact Prediction
Chen, Z., Jiang, T., & Niu, Y. (2023) · International Conference on Computer Science and Management Technology (ICCSMT) · doi:10.1145/3644523.3644594
Generating Fluent Fact Checking Explanations with Unsupervised Post-Editing
Atanasova, P. (2022) · Springer eBooks · doi:10.1007/978-3-031-51518-7_5
Using large language models for analyzing drug label data
ValizadehAslani, T., Liang, H., & ROSEN, G. L. (2024) · iDEA - Drexel University Institutional Repository · doi:10.17918/00001912
Instance Discrimination for Improving Chinese Text Classification
Hu, H., & Chen, Y. (2023) · International Conference on Machine Learning and Computer Application (ICMLCA) · doi:10.1145/3650215.3650220
“Beltagy et al. [20] introduced a BERT-based pretrained language model that leverages unsupervised pretraining on a large-scale, multi-domain scientific publication corpus, thereby enhancing the performance of downstream scientific NLP tasks.”
BackgroundICD-10 WHO Search With RAKE Algorithm
Wijayakusuma, I. G. N. L., & Gautama, I. P. W. (2023) · Jurnal Matematika · doi:10.24843/jmat.2023.v13.i02.p163
“Natural language processing (NLP) is ability of a computer program to understand human language as it is spoken (Beltagy et al., 2019).”
BackgroundJiang, Q., Sun, Y., Li, J., Qin, Y., Wu, Z., Huang, J., Wang, Z., & Liu, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4823073
A Large Financial Chat ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-lsc94
Unsupervised learning from textual data with neural text representations
Ait-Saada, M. (2023) · theses.fr (ABES) · doi:10.70675/aeb40eb3z15b8z496cz91aazcef8fd2c59a7 · Cited by 1
SD-WEAT: Towards Robustly Measuring Bias in Input Embeddings
Gray, M., & Wu, L. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-62110-9_5
Sebe, C., Cohen‐Boulakia, S., Ferret, O., & Névéol, A. (2024) · HAL
MathBERTRes: Mathematical BERT Model for Sequence Classification of Mathematical Research Articles
Srivani, M., Murugappan, A., & Mala, T. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-5015-7_16
Automated Scheduling for Thematic Coherence in Conferences
Emu, M., Ahmed, T., & Choudhury, S. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3665085
A survey on short text similarity measurement methods
Zadeh, A. R., & Amirkhani, H. (2023) · Archives de l'Institut d'Hessarek. · doi:10.61186/jsdp.20.3.103
Ensuring Transparency in Takeovers: Evaluating Information Retrieval in the Indian Financial Market
Sisodia, Y. (2024) · Algorithms for intelligent systems · doi:10.1007/978-981-97-3526-6_5
Named Entity Recognition From Biomedical Data
Refaat, M., Rafea, A., & GabAllah, N. A. (2023) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci62032.2023.00141
Public health fact verification based on semantic hierarchy graph
Sun, Y., & Zhang, G. (2024) · International Conference on Digital Signal and Computer Communications (DSCC) · doi:10.1117/12.3033405
“Iz Beltagy et al. [16] introduced SCIBERT, a pre-trained language model built upon BERT [17] with the aim of addressing the shortage of high-quality, large-scale labeled scientific data.”
Stochastic Featurization for Active Learning
Le, L., Nguyen, M.-T., Tran, K. P., Zhao, G., Zhang, X., Zuccon, G., & Demartini, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-67751-9_5
TVD-BERT: A Domain-Adaptation Pre-trained Model for Textural Vulnerability Descriptions
Wang, Z., Liang, X., Du, R., Tian, J., & Zhang, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5663-6_17
Alkan, A. K., Grouin, C., & Zweigenbaum, P. (2023) · HAL
FoRC@NSLP2024: Overview and Insights from the Field of Research Classification Shared Task
Ahmad, R. A., Borisova, E., & Rehm, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_12 · Cited by 6
Classifiers of Data Sharing Statements in Clinical Trial Records
Mamaghani, S. J., Strantz, C., & Toddenroth, D. (2024) · Studies in health technology and informatics · doi:10.3233/shti240541
“SciBERT has been trained on a compilation of scientific papers [7].”
Sun, Z. J., Xing, L., Zhang, L., Cai, H., & Guo, M. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.053588 · Cited by 1
Hillebrand, L., Pradhan, P., Bauckhage, C., & Sifa, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70359-1_23
Graph Neural Network for Molecular Structure: Application in HIV Inhibitor Molecule PredictionPreprint
Nguyen, Q. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/c3mqy
Information Analysis Using Biomedical Text Mining
Ahmad, O., & Sharan, A. (2024) · Transactions on computer systems and networks · doi:10.1007/978-981-97-3962-2_3
Kantor, K., & Morzy, M. (2024) · Proceedings of the International Conference on Information Systems Development · doi:10.62036/isd.2024.53 · Cited by 2
Deep Learning-Driven Innovative Model for Generating Functional Knowledge Units
Pan, Q., Hu, Y., You-bai, X., Xianghui, M., & Yilun, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70546-5_14
Kang, W., Zuo, J., Dai, Q., Hu, Y., & Wang, M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_37
CouBRE: Counterfactual NLI For Low-Resource Biomedical Relation Extraction
Zhong, J., Liu, Y., Ouyang, C., Ren, L., Zhu, C., & He, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_9
Zhao, H., Liu, H., Zhou, H., Yang, M., & Zhang, X. (2024) · Research Square · doi:10.21203/rs.3.rs-5338009/v1
Building an extensible, AI-augmented ecological momentary assessment platform
Zhao, F. (2024) · MOspace Institutional Repository (University of Missouri) · doi:10.32469/10355/105952 · Cited by 2
Complex networks in entrepreneurial ecosystems : clustering methodologies and topological structure
Carniel, T. (2024) · theses.fr (ABES) · doi:10.70675/38813fa7z841dz4b31z9c1cz496277b05abd
Weakly-Supervised Learning Strategy for Domain-Specific Entity Recognition
Weber, L. J., Ramalingam, K. J., Liu, C., Beyer, M., & Zimmermann, A. P. (2024) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds62089.2024.10756418
“Especially in domains like chemistry [8], science [9], engineering [10], and automotive [11], word distributions and semantics of model’s pretraining corpora and domain-specific corpora are different [12].”
Enhanced Adverse Drug Event Extraction Using Prefix-Based Multi-Prompt Tuning in Transformer Models
Modi, S., Kasmiran, K. A., Sharef, N. M., & Sharum, M. Y. (2024) · JOIV International Journal on Informatics Visualization · doi:10.62527/joiv.8.3-2.3454
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674128
Searching Personal Collections
Bendersky, M., Metzler, D., Najork, M., & Wang, X. (2024) · ACM eBooks · doi:10.1145/3674127.3674142
Query-driven Search Functionality
Li, L. (2024) · ACM eBooks · doi:10.1145/3674127.3674133
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702962
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674129
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702961
Hersh, W., & Roberts, K. E. (2024) · ACM eBooks · doi:10.1145/3674127.3702960
Bast, H., Kalmbach, J., Klumpp, T., & Korzen, C. (2024) · ACM eBooks · doi:10.1145/3674127.3674134 · Cited by 1
Ishihara, S., Takahashi, H., & Shirai, H. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1563
Identification and Extraction of Biomarker Information
Kavumkandathil, A. S., Dakal, T. C., Berua, A., & Kumar, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95502-7.00215-3
-, D. B., -, K. S., -, A. B., & -, S. M. A. H. (2024) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2024.v06i06.32271
Luo, M., Ai, S., & Zhang, X. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822483
“For each u ∈ U T , its textual description is encoded as a vector H [ u ] using SciBERT [14].”
Materials-aware Entity Augmentation for Continued Pre-training in Materials Science
Kim, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825208
“We conduct experiments using both BERT and SciBERT as backbone models [11] [13].”
Document-Level Biomedical Entity Relation Extraction Based on Semantic Segmentation
Wei, Z., & Li, G. (2024) · International Conference on Electronics and Devices, Computational Science (ICEDCS) · doi:10.1109/icedcs64328.2024.00015
Ruosch, F., Vasu, R., Wang, R., Rossetto, L., & Bernstein, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78955-7_16
A Research on Citation Sentiment Classification Based on Large Language Model
孔, 明. (2025) · Computer Science and Application · doi:10.12677/csa.2025.151021
Token Embedding Graph Neural Network for Paper Classification
Cai, S., & Gan, X. (2024) · International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice63571.2024.10864147
“A series of works leverage the transformer language model such as SciBert[3] to build a text encoder module in parallel with the GNN module.”
A Two-Phased AI-Enabled Framework for Innovation with User-Generated Data from Consumer Review Sites
Mirkovski, K., Kang, J., Liu, L., Indulska, M., & Liu, H. (2025) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2025.081
Unveiling Global Discourse Structures: Theoretical Analysis and NLP Applications in Argument MiningPreprint
Le, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.08371
Tracing architecture of machine learning models through their mentions in scholarly articles
Karmakar, S., & Kamps, J. (2024) · International Conference on Data Science and Information Technology (DSIT) · doi:10.1109/dsit61374.2024.10881057
“The first model is chosen as a benchmark, which holds a strong foundation within the field of scientific domain: SciBERT [20], which has persistently demonstrated good performance in various scholarly NER tasks, placing itself as an obvious choice for pre-training in the scientific domain.”
Exploiting Dual Contrastive Learning Algorithm for Scientific Texts Classification
Ma, J., & Zhao, R. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-2794-3_47
Which Types of Word Embedding Models are More Effective for Novelty Measurement?
Wu, W., Zhang, C., & Zhao, Y. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702491
“Firstly, we used word2vec [8], BERT [9], SciBERT [10], RoBerta [11], AlBERT [12], and XLNet [13] as word embedding models to vectorize the references’ title.”
MPR: A Dataset for Extracting Relations Between Method Entities and Scientific Papers
He, Y., Gu, G., Chen, C., & Yan, H. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702579
“Notably, the advent of pre-training language models, such as BERT [14] and SciBERT [15], has markedly boosted the accuracy of entity recognition.”
Chronological Evaluation of Novel Methodology Extraction from AI Literature
Ghosh, M., Ganguly, D., Basuchowdhuri, P., & Naskar, S. K. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702589
“For our experiments, we fine-tune the standard SciBERT model on our labeled data (Section 5.1.3) with the sequence labeling loss (Equation 1).”
InfluentialST-KAN: Document Classification Using Sentence Transformer and the Kolmogorov-Arnold Network
Göz, F. (2025) · doi:10.1109/it64745.2025.10930251
“Recently, deep learning models, particularly transformer-based architectures, have significantly improved text classification tasks such as scientific document classification [9]–[13].”
Conceptual transformation in distance education: Changes in the period 2006-2023 and future trends
Özçınar, H. (2025) · ODÜ Sosyal Bilimler Araştırmaları Dergisi (ODÜSOBİAD) · doi:10.48146/odusobiad.1613383
Bertweetro: Pre-Trained Language Models for Romanian Social Media Content
Neagu, D. C. (2025) · Studia Universitatis Babeş-Bolyai. Oeconomica · doi:10.2478/subboec-2025-0005
A Review Article for Argumentation Mining of Text through Machine Learning Techniques and Strategies
Singh, R. B. (2025) · Journal of Information Systems Engineering & Management · doi:10.52783/jisem.v10i26s.4237
“Variants like BioBERT are specialized for processing medical text, ClinicalBERT is tailored for clinical text, and SciBERT is designed for handling scientific text [22].”
Hierarchical Prefixes for Long Document Representations
Boucharenc, I. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88720-8_28
Learning the syntax of plant assemblages
(2025) · doi:10.21203/rs.3.rs-6304381/v1
Entity extraction in clinical summary of mammary malignancy data using transformer-based models
Patra, P., Marco, A. D., & Nguyen, P. T. (2026) · International Journal of Data Science and Analytics · doi:10.2139/ssrn.5199243
A Dataset of Semantically Related Multiword Terms of the Electrical Engineering Domain
Druselmann, M., & Harbusch, K. (2024) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3711542.3711550
Ochi, M., Shiro, M., Mori, J., & Sakata, I. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4613-5_9
Enhancing Query Expansion for Rare Diseases in PubMed Using Embedding-Based Semantic RepresentationsPreprint
Kelly, S. K. (2025) · doi:10.31224/4535
AI-Based Adaptive Legal Analysis Engine for Enhanced Policing and Predictive Law Enforcement
Singh, N., Tiwari, A., Tiwari, R., Tiwari, P., Pushkarna, C., & Choudhary, J. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-97-9619-9_10
Harnessing Pre-trained Language Models for Efficient Move Recognition in Biomedical Abstracts
Patil, G., & Mercer, R. E. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86623-4_17
Duan, Z., Gu, H., Wang, D., & Zhou, D. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02438-x · Cited by 2
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.