SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 3,201–3,300 of 3,596
A Modular OCR and NLP Pipeline for Digitizing Calibration Reports in Metrology
Fong, K. K., & Wong, N. H. L. (2025) · IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI) · doi:10.1109/soli66289.2025.11396102
“It combines document preprocessing, layout analysis via PaddleOCR [5], and named-entity recognition (NER) using SciBERT [6], a fine-tuned variant of bidirectional encoder representations from transformers (BERT) [7] which is pretrained This work was supported by the Singapore Institute of…”
HelBERT: A BERT-Based Pretraining Model for Public Procurement Tasks in Portuguese
Lima, W. E. M., Silva, V. R. D., da Silva, J. C., Rabêlo, R. A. L., & de Paiva, A. C. (2026) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2026.5511 · Cited by 1
“[2019] represents one of the first initiatives in the biomedical domain, followed by subsequent efforts in domains such as clinical medicine [Alsentzer et al ., 2019] and science [Beltagy et al ., 2019].”
AI-DRIVEN ACADEMIC SCREENING: PENGEMBANGAN SISTEM REVIEWER OTOMATIS BERBASIS AI AGENT
Riyanto, V., Saryoko, A., Anton, Mazia, L., Nurmalasari, & Mardiana, T. (2026) · Informasi dan Teknologi Ilmiah (INTI) · doi:10.33480/inti.v20i1.6793
“Model BERT telah menjadi standar dalam pemrosesan bahasa alami untuk tugas klasifikasi teks, Model BERT dan turunannya (misalnya, SciBERT) mencapai akurasi 91% dalam klasifikasi teks akademik (Callegari et all., 2023) .”
InfluentialFindCoResearcher: Retrieval-based Collaborative Researcher Recommendation System
Sung, S., Kim, J., Cheon, J., Son, J., & Kang, P. (2026) · Journal of Korean Institute of Industrial Engineers · doi:10.7232/jkiie.2026.52.1.055
A Comparative Study of AI-Driven Ontology Enrichment for Environmental Sustainability
Khetarpaul, S., Zghal, H. B., Gupta, S. D., Bist, V., & Bhatnagar, D. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402007
“Fine-tuning improves the model’s ability to capture nuanced semantic relationships relevant to environmental concepts [30].”
Duddukuri, Y. H. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00278
“…approaches include hypergraph-based models like HGERE (32) and various joint models designed to improve task interaction, such as CARE 16, PFN 19, and MGE.20 The development of domain-specific language models like SciBERT (3) has also been instrumental in advancing performance on this benchmark.”
SuSEF: Summarized Section Embedding with Fusion for Scientific Paper Recommendation
Westmoreland, J., & Liang, H. (2025) · International Conference on Robotics, Automation and Artificial Intelligence (RAAI) · doi:10.1109/raai67517.2025.11423046
“A SciBERT-based classifier [7] splits each abstract into background , method , and results sentences, which are grouped into three sections and embedded separately with SPECTER alongside a whole-abstract embedding.”
Massively parallel distributed fine-tuning of transformer-based language models
Farkas, A. T., & Kertész, G. (2026) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami68106.2026.11420325
“SciBERT [7] to downstream tasks while solving overfitting and instability issues.”
Autogenerating Material Databases and Language Models for Stress Engineering Materials Science
Kumar, P. (2024) · Apollo (University of Cambridge) · doi:10.17863/cam.128312
PiKGL: Leveraging Pruned Knowledge Graphs for Explainable Stance Detection
Wang, B., Lin, J., Bai, Z., Song, X., Wang, Q., Yang, M. Y., Zeng, X., Li, J., & Xu, R. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.612
Jadhav, A. V., Masrouri, M., & Qin, Z. (2026) · Advanced Intelligent Systems · doi:10.1002/aisy.202501399
Classification of health product defect reports by deep learning
Sancenón, V., Huang, Y., Zou, L., Teo, D. C. H., Dorajoo, S. R., Ang, P. S., Goh, H. L., & Ta, A. W. A. (2026) · Scientific Reports · doi:10.1038/s41598-026-43961-3
“…in the Scientific and Clinical domains, we fine-tuned SciBERT, Bio_ClinicalBERT and Bio_Discharge_Summary_BERT leveraging the same dataset and methodology used to fine-tune MedDefects-BERT: • SciBERT: Initialised from BERT base and trained on a corpus of papers taken from Semantic Scholar 69 .”
GPT-MM: Improving Multimodal In-context Learning with Task-specific Retrieval and Reasoning
Wan, Z., Cheng, F., & Kurohashi, S. (2026) · Journal of Natural Language Processing · doi:10.5715/jnlp.33.207 · Cited by 1
MaterialBERT for Natural Language Processing of Materials Science Texts
Kawano, H., Sato, F., Yoshitake, M., MOTEKI, F., & TERAOKA, H. (n.d.) · NIMS Materials Data Repository
Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learning
Wu, H., Xiang, H., Gao, J., Zhao, X., Wu, D., & Li, J. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792568 · Cited by 1
Zhou, F., Lokker, C., Parrish, R., Haynes, R. B., Iorio, A., Saha, A., & Afzal, M. (2026) · JMIR AI · doi:10.2196/77311
“…b [67], BioLinkBERT c [28], BiomedBERT d (abstracts only) [70], BiomedBERT (abstracts+full text) [70], SciBERT e -cased [27], SciBERT-uncased [27] 7 Pretrained model Yes, no 2 CW f adjustment 1 × 10 –5 , 3 × 10 –5 , 5 × 10 –5 3 where N is the total number of samples and n i is the number of…”
Deng, S., Xiang, R., Zhu, Q., & Chen, F. (2026) · Information Research · doi:10.47989/ir31iconf64192
“With the advent of pre-trained language models such as BERT and SciBERT (Beltagy et al., 2019; Shen et al., 2023), deep learning approaches have achieved notable success in this domain (Puccetti et al., 2023), such as SciBERT+CRF and SciBERT+BiLSTM+CRF proposed to further enhance performance(Zhang…”
InfluentialA SciBERT-Bipartite Graph Hybrid Model for Scientific Paper Recommendation
Smail, B., & Aliane, H. (2026) · INTERNATIONAL JOURNAL OF NEXT-GENERATION COMPUTING · doi:10.47164/ijngc.v16i3.1979
Zaidi, A., Meenakshi, K., & Kushwaha, P. (2025) · 2025 3rd International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit68829.2025.11434046
“Similarly, Beltagy et al. (2019) showed that SciBERT, pre-trained in scientific literature, achieves superior performance on scientific NLP tasks [9],[19].”
Afifi, B., Abu-Elkheir, M., & Abdennadher, S. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.01.036
Designing mRNA coding sequence via multimodal reverse translation language modeling with Pro2RNAPreprint
Bian, B., Zhang, Y., Zhang, J., Asai, K., & Saito, Y. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.18.712790 · Cited by 1
Gonc¸alves, A., Cardoso, H. L., & Reis, L. P. (2026) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2026.3676374
Koneru, S., Wu, J., & Rajtmajer, S. M. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846042
Liu, X., Lin, W., Hu, M., Xu, C., & Li, H. (2026) · Research Square · doi:10.21203/rs.3.rs-9084876/v1
A Cusp Catastrophe Model for Early Identifying Scientific BreakthroughsPreprint
Wang, Z., Zhao, X., Shen, H., Fang, J., Zhang, J., Luczak–Roesch, M., & Haihua, C. (2026) · Research Square · doi:10.21203/rs.3.rs-10941810/v1
MedFusion-GPT: A Multimodal Generator Combining GPT-2 and BioGPT for Clinical Report Synthesis
Abdaoui, H., Barbaria, S., Ridene, T., Salah, R. B., & Rahmouni, H. B. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.03.076
Tracing the Flow of Knowledge From Science to Technology Using Deep LearningPreprint
Rose, M. E., Ghosh, M., Erhardt, S., Li, C. K., Buunk, E., & Harhoff, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.24259
An SPO Feature-Enhanced Framework for Extracting Innovative Contributions from Academic Papers
Ma, B., Ren, X., Guo, C., Zhu, G., & Liu, X. (2025) · International Conference on Big Data and Data Mining (BDDM) · doi:10.1109/bddm68348.2025.11442016
“The main contributions of this study are: Evaluated multiple text classification models (e.g., SciBERT [10] , ALBERT [11] , XLNet [12] , TextCNN [13] ) and loss functions, identifying SciBERT with Focal Loss as the optimal baseline; Constructing an annotated corpus of innovation statements from…”
FePTP: A text-mined dataset of transformation pathways among iron-containing phases
Lin, L., Ren, C., Xiao, Y., Nie, J., Qi, C., Li, X., Wang, H., & Lin, Z. (2026) · Scientific Data · doi:10.1038/s41597-026-07067-9
Research Rewired AI in Computing
Azeroual, O. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0434-0.ch004
Martin, S. R. (2025) · Buleria (Universidad de León) · doi:10.18002/10612/28003
A Multilayered Approach to Constructing a Patent Field of Technology Ontology
Qiu, X., & Zhou, Y. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3803791
Matching Researchers to Funding Calls: A Reproducible Institution-Level FrameworkPreprint
Arroyo‐Machado, W., Lázaro-Soraluce, L., Ortega-Sevilla, C., Fuente-Gutiérrez, E., & Torres‐Salinas, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.06321
Jeong, S. Y., Choi, K., Choi, J., Kim, K., Lee, H. W., & Yeo, B. C. (2026) · Chemical Engineering Journal Advances · doi:10.1016/j.ceja.2026.101164
Object-Centric Extraction and Analysis of Scientific Publications
Küsters, A., Pitsch, C., Rennert, C., van Detten, J. N., Norouzifar, A., Peeva, V., Li, T., & Knopp, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-17618-9_43
Pegasus-copynet: a novel summarization generation framework for scientific and technological texts
Wang, S., Wang, H., Wang, X., Jiang, Y., Sun, S., Pan, X., & Ren, P. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02986-2
岩月, 賢. (2021) · Institutional Repositories DataBase (IRDB) · doi:10.15083/0002007204
Venetsianou, N. K., Paragkamian, S., Kalaentzis, K., Loukas, A., Damianou, C., Lagani, V., Jensen, L. J., & Pafilis, E. (2026) · Microbial Ecology · doi:10.1007/s00248-026-02730-5
“While models like SciBERT and other BERT-based models were pre-trained with masked-language mod - eling and excel at generating domain-specific embeddings ([ 43 , 44 ]), they are not inherently designed for prompt-driven reasoning tasks without extensive fine-tuning ([ 44 ]).”
事前学習済み言語モデルを用いた検索モデルに対する教師なしドメイン適応
飯田, 大., & Iida, H. (2024) · Institutional Repositories DataBase (IRDB)
Gazdasági elemzés strukturálatlan adatokon: természetesnyelv-feldolgozási esetek
Temesvári, C., Horváth, B., & Ónozó, L. R. (2026) · Hitelintézeti szemle · doi:10.25201/hsz.25.1.27
Samal, A. K., Subramaniyam, V., Mathur, T., Kiran, A., Tyagi, P., Tiwary, U. S., & Varadwaj, P. K. (2026) · Intelligent Data Analysis · doi:10.1177/1088467x261433365
Natural Language Processing-Driven Use-Cases for Economic Analysis Using Unstructured Data
Temesvári, C., Horváth, B., & Ónozó, L. R. (2026) · Financial and Economic Review · doi:10.33893/fer.25.1.27
A Framework for Identifying Problem-Solution in Research Articles: Focusing on Social Issue
Kim, M.-S., Shin, H., Chun, H.-W., & Lee, J.-M. (2026) · IEEE Access · doi:10.1109/access.2026.3678862
Classification of Engineering Documents for Review Prioritization Using Natural Language Processing
Yang, Z., Shi, W., Zhu, L., Zhu, Z., Wang, M., & Zhang, B. (2025) · International Conference on Computer Science and Management Technology (ICCSMT) · doi:10.1145/3795154.3795206
“…that combines the statistical specificity of Term Frequency-Inverse Document Frequency (TF-IDF) [1] with the semantic depth of domain-specific Transformer models (SciBERT) [2], we demonstrate the ability to accurately categorize documents into ’Critical’, ’High’, and ’Routine’ priority tiers.”
Embedding-Based Topic Modeling for Strategic Research Portfolio Analysis
Silva, M. C. G. (2026) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Zhang, H., Wang, J., Yu, J., & Liu, L. (2026) · Information · doi:10.3390/info17040345
“To address this, we employ SciBERT [17] in place of traditional BERT.”
Contextualization and Generalization in Entity and Relation Extraction
Taillé, B. (2022) · theses.fr (ABES) · doi:10.70675/72fea173z5549z4bd5z8332zf81f26d3c2b9
Learning Like a Student: From Open Book to Closed Book for Enhanced Domain-Specific QA
Truong, Q. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21625-0_4
Bilimsel makalelerin alıntı yapılarını dikkate alan gösterimlerin öğrenimi
Çelik, E. Y. (2024) · İYTE (İZMİR YÜKSEK TEKNOLOJİ ENSTİTÜSÜ)
Saisie de prescriptions médicales en langage naturel sur terminaux mobiles
Kocabiyikoglu, A. C. (2022) · theses.fr (ABES) · doi:10.70675/998d4b74z9eb1z49bcz9763zfa1e8ebcbd43
Indexation de bout-en-bout dans les bibliothèques numériques scientifiques
Gallina, Y. (2022) · theses.fr (ABES) · doi:10.70675/6c6d3371zd10ez4799z96eczc65e63363695
A Data-driven Approach to Natural Language Processing for Contemporary and Historical French
Suárez, P. O. (2022) · theses.fr (ABES) · doi:10.70675/9b4a07fdza641z489fza523zddeee03a6544 · Cited by 1
Intégration de données textuelles pour la détection des risques naturels en agriculture
Jiang, S. (2022) · theses.fr (ABES) · doi:10.70675/4f90cd87zefffz4b51z84b4zfc08012c5ea2
Towards efficient, general and robust entity disambiguation systems
Chen, L. (2023) · theses.fr (ABES) · doi:10.70675/abee8a0cz824dz4e34z917azffda9fe5cd83
Modeling and optimization of an online publishing application
Cuculovic, M. (2022) · theses.fr (ABES) · doi:10.70675/5425dc78z25adz4de4z956fzf34b008ec511
Fake identity & fake activity detection in online social networks based on transfer learning
Zarei, K. (2022) · theses.fr (ABES) · doi:10.70675/ad847383z33f5z4bb7zab0fzafb8c023b56e · Cited by 7
Machine Learning Approaches for Sub-surface Geological Heterogeneous Sources
Arman, M. (2023) · theses.fr (ABES) · doi:10.70675/42bb8288za220z4273za3f7zb042954f2cac
Learning Recruitment-Related Representations from Graphs and Sequential Data
Zhu, J. (2023) · theses.fr (ABES) · doi:10.70675/45fbe640z43b8z454bz902azb8501bcf111d · Cited by 2
Wang, Y. (2022) · theses.fr (ABES) · doi:10.70675/9bae8e4fz2724z43f3z8be4z7f0367c36fae · Cited by 1
Ordering nodes to scale to large real-world networks
Lécuyer, F. (2023) · theses.fr (ABES) · doi:10.70675/ac8302eazf54cz4ddfzaa71zd313e52dcc8f
Text mining of aviation occurrence reports
Kierszbaum, S. (2022) · theses.fr (ABES) · doi:10.70675/d744df4az0e21z46acz9736z3727d1ed22ab
How Well Do Ratings Reflect Sentiment? Evidence From a Large Italian Review Corpus
Biasetton, N., Ricciardi, R., Salmaso, L., & Zuccolotto, P. (2026) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.70090 · Cited by 1
Rafif, S., & Pradana, F. Z. (2026) · JATI (Jurnal Mahasiswa Teknik Informatika) · doi:10.36040/jati.v10i2.17805
Business relation extraction from texts
Khaldi, H. (2022) · theses.fr (ABES) · doi:10.70675/ce231ce4z5d07z4ab7z9758za927ea7b5302
Toofanee, M. S. A. (2023) · theses.fr (ABES) · doi:10.70675/600337cazbfe4z405fz9538za725b88272b7
Une Approche Collaborative pour la Classification des Sites Web
Koufi, M. Z. (2024) · theses.fr (ABES) · doi:10.70675/1c35c522z32c3z4c0aza5eaz331e61ba1cd3
TALMed : Traitement Automatique de la Langue Médicale
Bazoge, A. (2024) · theses.fr (ABES) · doi:10.70675/f970ff27z65b9z4988za234zac33e61a0071
Natural Language Processing for Analyzing Messages of Astrophysical Observations
Alkan, A. K. (2024) · theses.fr (ABES) · doi:10.70675/ab93cd1az4468z404czb2b3z8dc680a04e4d · Cited by 2
Hadi, A. (2024) · theses.fr (ABES) · doi:10.70675/2ab806f6z8e59z4fcdzbe6cz771e49e0be32 · Cited by 1
Biomedical Event Extraction Based on Transformers and Knowledge Graphs
Zanella-Calzada, L. A. (2023) · theses.fr (ABES) · doi:10.70675/982498bdz45f3z4b19z9493zcce91684c91d
Hesslow, D. (2024) · theses.fr (ABES) · doi:10.70675/f705b608z49b0z490cz88dez9f4819f93f11 · Cited by 2
Multimodal learning to predict breast cancer prognosis
Mbaye, N. M. (2024) · theses.fr (ABES) · doi:10.70675/cdccc54az52f7z4e20za5a4z3ec09cd3a564
Joint representations for heterogeneous image sets and texts
Meudec, R. (2024) · theses.fr (ABES) · doi:10.70675/355832f0z0174z46a6zbc20zcd347ea26a05
Argumentation quality : from general principles to healthcare applications
Marro, S. (2023) · theses.fr (ABES) · doi:10.70675/189775d8z55d6z4189zbf60zc08afacac314 · Cited by 1
Analyses of Green Energy Innovation Using Patent Data
Cheng, Y.-C., & Hsieh, C. (2026) · Journal of Sustainability Research · doi:10.20900/jsr20260029
Prédiction structurée basée sur des segments pour l'extraction de graphes d'information
Zaratiana, U. (2024) · theses.fr (ABES) · doi:10.70675/ae81adb9za996z4b40zb1d2z1f44e6717c93
Argument-based natural language explanation generation and assessment in healthcare
Molinet, B. (2024) · theses.fr (ABES) · doi:10.70675/dcb29a70z3429z4ab0zb096z98206dd11eab · Cited by 1
Détection des éléments générés automatiquement Contenu académique
Liyanage, V. P. (2024) · theses.fr (ABES) · doi:10.70675/80dff7a0z6e58z4151zb20cz6f0bc537ce85
Learning deep kernel networks : application to efficient and robust structured prediction
Ahmad, T. E. (2024) · theses.fr (ABES) · doi:10.70675/5335037bz0868z4fd4zaedez3d7d4d3252f7
Aouina, O. (2024) · theses.fr (ABES) · doi:10.70675/2c8a820cz57f8z4594z9cb8zeaf9ec41fe5e · Cited by 1
Prompt-based Relation Extraction for Pharmacovigilance
Yeh, H.-S. (2024) · theses.fr (ABES) · doi:10.70675/762a3598ze739z4c47za72ez62e5a986fd1f · Cited by 1
Goffredo, P. (2024) · theses.fr (ABES) · doi:10.70675/9eb833d6zed9bz48dcz8b70z8b2d360ea5cf · Cited by 1
Enrichissement et alignement sémantique d'οntοlοgies biοmédicales par mοdèles de langue
Menad, S. (2024) · theses.fr (ABES) · doi:10.70675/8ebf6270z9566z4bbez91b7z08485656b6e2
Happi, B. G. H. (2024) · theses.fr (ABES) · doi:10.70675/20022d2az1be9z4bd9za6e1zb410bde66ce1 · Cited by 1
Park, K., Kim, K., Kim, K., Kim, K., Seo, W.-Y., Shin, J.-M., Kim, J., & Kim, S. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132394
Piat, G. (2023) · theses.fr (ABES) · doi:10.70675/bb18b592z47d5z40f0z805dz97e6aa90c6e5 · Cited by 1
Neuro-Symbolic Attention Modulation in Biomedical QA via Signal Algebra-Augmented LLMs
Sohn, H. Y., Mishra, S. K., & Bijalwan, V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18162-6_12
Domain Adaptation of Named Entity Recognition for Plant Health Monitoring
Borovikova, M. (2024) · theses.fr (ABES) · doi:10.70675/aa08ea7az594ez48adz814bz17ff1279b349
Laï-king, M. (2025) · theses.fr (ABES) · doi:10.70675/e9422bf7zf52dz461bza1a5z2593be8e49c3
Ningrum, P. K. (2025) · theses.fr (ABES) · doi:10.70675/cb5199baz28b9z4b73zb5f7z36d58a485d64 · Cited by 1
TRIZ-guided discovery of interdisciplinary innovation patterns via AI-enabled literature mining
Douard, N. (2025) · theses.fr (ABES) · doi:10.70675/9d86d913z0d0bz45d2z863dz0dbb7f6a2a46
Peuplement semi-automatisé de bases de connaissances à partir de documents non structurés.
Prieur, M. (2025) · theses.fr (ABES) · doi:10.70675/6c995326zb3b9z422dz9e5azbbabf8c75862
Enhancing surrogate regression methods for structured prediction : An odyssey with loss functions
Junjie, Y. A. N. G. (2025) · theses.fr (ABES) · doi:10.70675/8a087025za007z4013zaa7dz6fde23f91948
Riabi, A. (2025) · theses.fr (ABES) · doi:10.70675/5a649ef5za082z43d8z906azd8051be91d15 · Cited by 4
Han, M., Ou, S., Li, J., Zhu, K., & 聂为民 (2026) · Scientometrics · doi:10.1007/s11192-026-05606-0
“SciBERT is developed by the Allen Institute for Artificial Intelligence.”
InfluentialAlzheimer Diagnosis and Cost Estimation Bot
Kumar, S., Kundu, N., & Rani, G. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14757-8_41
A Multi-Channel Knowledge-Enhanced Model for Biomedical Relation Extraction
Lu, H., Li, L., & Tang, J. (2026) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2026.3683397
“Experiment Settings: To capture the deep contextual sequential features of the entity, we use SciBERT [27] to learn the representations of the entity definitions.”
Benchmarking MeSH-augmented embeddings for biomedical document similarity
Ravinder, R., Geist, L., Quiñones, N., Venkatesh, S., Castro, L. J., & Rebholz‐Schuhmann, D. (2026) · Journal of Biomedical Semantics · doi:10.1186/s13326-026-00365-6
“In the biomedical domain, specialized models such as BioBERT [15] and SciBERT [16] have been trained on large-scale biomedical corpora, leading to significant improvements in tasks like document classification, information retrieval and semantic similarity analysis.”
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.