SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,801–2,900 of 3,596
GeoCQA: A Large-Scale Geography-Domain Chinese Question Answering Dataset from Examination
Cui, Z. S., Wang, B., & Ju, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_13
Gold Panning: Automatic Extraction of Scientific Information from Publications
Hong, Z., Chard, K., & Foster, I. T. (2021) · IEEE International Conference on eScience · doi:10.1109/escience51609.2021.00042
LIMEADE: A General Framework for Explanation-Based Human Tuning of Opaque Machine Learners.Preprint
Lee, B. C. G., Downey, D., Lo, K., & Weld, D. S. (2021) · arXiv (Cornell University)
Self-citationHuman Learning-Augmented Machine Learning Frameworks for Text Analytics
Xia, L. (2020) · VTechWorks (Virginia Tech)
Corporate Bankruptcy Prediction with BERT Model
Kim, A., & Yoon, S. (2021)
Learning Entity-Likeness with Multiple Approximate Matches for Biomedical NER
Le, A. N., Morita, H., & Iwakura, T. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_117
Сходство документов на основе аспекта на примере научных статей
Ostendorff, M., РЕМ, Г., РУАС, Т., БЛЮМЕ, Т., & Gipp, B. (2021) · Международный форум по информации · doi:10.36535/0203-6460-2021-01-4
Locating narrative in scientific papersPreprint
Toth, B. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/fv7ha
Zheng, X., Zhang, L., Zheng, W., & Hu, W. (2021) · IEEE International Conference on Information, Communication and Networks (ICICN) · doi:10.1109/icicn52636.2021.9673905
CORPUS PARA O DOMÍNIO ACADÊMICO: MODELOS E APLICAÇÕES
Pinto, I. D. J. P. (2021) · doi:10.17771/pucrio.acad.55901
Coordinate Structure Analysis without Labeled Data for Recognizing Compound Named Entities
Sawada, Y., Teranishi, H., Matsumoto, Y., & Watanabe, T. (2021) · Journal of Natural Language Processing · doi:10.5715/jnlp.28.1089
Self-Supervised Source Code Annotation from Related Research Papers
Haritz, P., Pfahler, L., Liebig, T., & Kotthaus, H. (2021) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw53433.2021.00140
Information Retrieval in an Infodemic: The Case of COVID-19 Publications (Preprint)Preprint
Teodoro, D., Ferdowsi, S., Borissov, N., Kashani, E., Alvarez, D. V., Copara, J., Gouareb, R., Naderi, N., & Amini, P. (2021) · doi:10.2196/preprints.30161
AIM in Genomic Basis of Medicine: Applications
Kamada, M., & Okuno, Y. (2022) · Springer eBooks · doi:10.1007/978-3-030-64573-1_264
Zeng, C., Li, S., & Chen, B. (2022) · Wireless Communications and Mobile Computing · doi:10.1155/2022/7088126
Transformer : des réseaux de neurones pour le traitement automatique des langues
Yvon, F. (2022) · Technologies logicielles Architectures des systèmes · doi:10.51257/a-v1-h3735
Graph-Enhanced Document Representation for Court Case Retrieval
Fink, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_59 · Cited by 2
Carrillo-Larco, R. M., Castillo‐Cara, M., & Lovón-Melgarejo, J. (2021) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16867.2
How to Churn Deep Contextual Models?
Hasan, M. R. (2021) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3486622.3493962
“Generally, the DS models should perform better than the DA models as they capture more informative prior about the target domain [1, 13, 16].”
BackgroundMethodCarrillo-Larco, R. M., Castillo‐Cara, M., & Lovón-Melgarejo, J. (2021) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16867.3
Perera, W. L., Messemer, H., Clados, C., & Heinz, M. (2021) · Gemeinschaften in Neuen Medien · doi:10.25368/2022.41
Kantor, K., & Morzy, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-09342-5_39
Dablain, D. A., Huang, L., & Sepulvado, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.06360
“Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), a general corpus to use as our baseline; SciBERT, which is trained on 1.14M papers from Semantic Scholar (Beltagy et al., 2019), BioBERT, which is trained on PubMed abstracts, PubMed Central full-text articles, and English Wikipedia…”
MethodThe Digitalization of Bioassays in the Open Research Knowledge Graph
D’Souza, J., Monteverdi, A., Haris, M., Anteghini, M., Farfar, K. E., Stocker, M., dos Santos, V. A. P. M., & Auer, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-12423-5_5
Crude Oil-Related Events Extraction and Processing: A Transfer Learning ApproachPreprint
Lee, M., Soon, L.-K., & Siew, E.-G. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4183577
Context Enhanced and Data Augmented $$\mathrm W^{2}NER$$ System for Named Entity Recognition
Ma, C., Xu, Z., Feng, M., Yin, J., Ruan, L., & Su, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17189-5_12
Carrillo-Larco, R. M., Castillo‐Cara, M., & Lovón-Melgarejo, J. (2022) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16867.5
Towards Language-driven Scientific AIPreprint
Gómez-Pérez, J. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.15327
“Until now, scientific language models like SciBert (Beltagy et al., 2019), BioBert (Lee et al.”
Barman, H., Venkateswaran, S., Santo, A. D., Rao, K. K., Raghunatha, B., Yoo, U., Kottschade, L. A., Block, M. S., Chandler, G. S., Wagner, T., et al. (2022) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2022-sitc2022.1274
Entity Factor: A Balanced Method for Table Filling in Joint Entity and Relation Extraction
Liu, Z., Tao, M., & Zhou, C. (2022) · Electronics · doi:10.3390/electronics12010121
Investigating Continued Pretraining for Zero-Shot Cross-Lingual Spoken Language Understanding
Louvan, S., Casola, S., & Magnini, B. (2022) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.10709
Yang, M., Zhang, Y., Banbhrani, S. K., Lin, H., & Lu, M. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-7596-7_4
Natural Language Processing for Scientific Paper Evaluation: Comparing Human and Machine Judgements
Xu, T., Hinton, N., Bennett, M. T., & Maruyama, Y. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-19682-9_90
Overview of STEM Science as Process, Method, Material, and Data Named Entities
D’Souza, J. (2022) · Knowledge · doi:10.3390/knowledge2040042 · Cited by 1
Albarqawi, A. (2023) · Research Square · doi:10.21203/rs.3.rs-2401418/v2
MathCLM: Mathematical Cognitive Learning Model Based on the Evolution of Knowledge Graph
Lin, G., Zhong, X., & Fu, H. (2022) · International Conference on Control, Automation, Robotics and Vision (ICARCV) · doi:10.1109/icarcv57592.2022.10004260
“However, the language model methods redefine this field with superior performance [11].”
BackgroundMethodArtificial intelligence and health
Oficina, C. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.57952/tnn0-z653
A Deep Learning-Based Innovative Points Extraction Method
Yu, C., Wang, R., Zhan, H., Lin, Y., & Yu, J. (2023) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-20738-9_16
The model thinks what?! Interpreting deep NLP models with rationales and influence
Jain, S. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20467212
What does the evidence say? Automatic information extraction from medical trial reports
Nye, B. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20467219
Nolano, G., L’Orientale", I., di Buono, M. P., & Monti, J. (2022) · International Conference EUROPHRAS · doi:10.26615/978-954-452-080-9_024
Transformer-based Detection of Multiword Expressions in Flower and Plant Names
Premasiri, D., Haddad, A. H., Ranasinghe, T., & Mitkov, R. (2022) · International Conference EUROPHRAS · doi:10.26615/978-954-452-080-9_021
A Biomedical Named Entity Recognition Framework with Multi-granularity Prompt Tuning
Liu, Z., Chi, T., Zhang, P., Wu, X., & Che, C. (2023) · Communications in computer and information science · doi:10.1007/978-981-19-9865-2_7
An Applied Computational Linguistics Approach to Clustering Scientific Research Papers
Vora, A., Mishra, M., & Muthukumar, S. (2023) · Advances in science and technology · doi:10.4028/p-s8m133
WORD EMBEDDING FOR UNKNOWN WORDS: ADDING NEW WORDS INTO BERT’S VOCABULARY
Maciel, D., Artese, L., & Gonçalves, A. L. (2023) · CONTECSI - International Conference on Information Systems and Technology Management · doi:10.5748/19contecsi/pse/dsc/7035
Reinforcement learning based distantly supervised biomedical named entity recognition
Bali, M., & Anandaraj, S. P. (2023) · Intelligent Decision Technologies · doi:10.3233/idt-220205
Document-Level Relation Extraction with Distance-Dependent Bias Network and Neighbors Enhanced Loss
Liang, H., & Zhou, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_38
Research on Entity recognition of Chinese place Names based on BERT
Zhang, B., Wang, W., & Deng, H. (2023) · IEEE Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) · doi:10.1109/itnec56291.2023.10082135
MCHPT: A Weakly Supervise Based Merchant Pre-trained Model
Zeng, Z., She, X., Qiu, X., Chai, H., & Yang, Y. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-1639-9_37
Seo, S., & Hwang, S. (2023) · Journal of Korean Institute of Industrial Engineers · doi:10.7232/jkiie.2023.49.2.133
A Study on Iconic Paper Discovery Based on Deep Learning Models
CHEN, Y., Guan, Z., Li, C., Wang, C., & Wang, Z. (2023) · International Conference on Science, Technology and Innovation Indicators (STI) · doi:10.55835/64382c384cc7592973d4786e
A Modular Hierarchical Model for Paper Quality Evaluation
Deng, X., Li, S., Yu, J., Ma, J., Ji, B., Lin, W., Song, S., & Yi, Z. (2023) · Artificial Intelligence Advances · doi:10.5121/csit.2023.130702
“Then we feed the words token sequence in each sentence contained in the title, abstract, and body into SciBERT, taking the embedding of the [CLS] token as the sentence representation.”
MethodInfluentialGaliani, S., Gálvez, R. H., & Nachman, I. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4460244
“Specifically, we fine-tuned a pre-trained SciBERT model (Beltagy, Lo, and Cohan, 2019) to adapt it for multiclass classification.”
Automatic Short Answer Grading Using Paragraph Vectors and Transfer Learning Models
Alreheli, A., & Alghamdi, H. S. (2022) · Journal of King Abdulaziz University-Computing and Information Technology Sciences · doi:10.4197/comp.11-2.2
“We specify two pre-trained transformer models to apply to this experiment; (Roberta-large) [21] and (Scibert) [22].”
MethodData-efficient domain adaptation for pretrained language models
Guo, X. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/167965 · Cited by 2
Slikker, W., Wu, L., Xu, J., & Shafer, T. J. (2023) · Elsevier eBooks · doi:10.1016/b978-0-12-824010-6.00048-4 · Cited by 1
Auxiliary Learning for Named Entity Recognition with Multiple Auxiliary Training Data
Watanabe, T., Ichikawa, T., Tamura, A., Iwakura, T., Ma, C., & Kato, T. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.507
A Survey of Domain-Specific Named-Entity Recognition in Japanese
Fukuda, M., & Sekine, S. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.800
Is there a slowdown in research productivity? Evidence from China and Germany
OECD (2023) · OECD eBooks · doi:10.1787/6dc66969-en
Tourassi, G. D., Shankar, M. A., & Wang, F. (2023) · OECD eBooks · doi:10.1787/0e1b4c2f-en
Szalay, K. Z. (2023) · OECD eBooks · doi:10.1787/6717b361-en
Quantifying the “cognitive extent” of science and how it has changed over time and across countries
Milojević, S. (2023) · OECD eBooks · doi:10.1787/1f717652-en
Interpretability: Should – and can – we understand the reasoning of machine-learning systems?
Cartwright, H. (2023) · OECD eBooks · doi:10.1787/2d0478b9-en
OECD (2023) · OECD eBooks · doi:10.1787/f1262928-en
Cabezudo, M. A. S. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2023.tde-19042023-103916
Knowledge Graphs for COVID-19: A Survey
Minoli, M., & Ambegoda, T. D. (2012) · Internet of things · doi:10.1007/978-3-031-28631-5_1
CiteVis : Visual Analysis of Overlapping Citation Intents as Dynamic Sets (Paper)
Agarwal, S., Ghosh, U., Beck, F., & Sreevalsan‐Nair, J. (2022) · doi:10.20378/irb-54949
Towards Querying Multimodal Annotations Using Graphs
Bruschke, J., Kröber, C., Utescher, R., & Niebling, F. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-38871-2_5
Disease Network Constructor: a Pathway Extraction and Visualization
Sohrab, M. G., Duong, K., Topić, G., Ikeda, M., Nagano, N., Natsume‐Kitatani, Y., Kuroda, M., Itoh, M. N., & Takamura, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.53
Hierarchy Builder: Organizing Textual Spans into a Hierarchy to Facilitate Navigation
Yair, I., Taub-Tabib, H., & Goldberg, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.27
“We chose SAP-BERT for its entity-linking specialization, and since it outperformed other models we tried, such as SciBert(Beltagy et al., 2019), in detecting semantic similarity for our specific case.”
MethodChandrasekhar, S., Huang, C.-Y., & Huang, T.-H. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.8 · Cited by 2
Manegold, H., & Girrbach, L. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.35
Explore Unsupervised Structures in Pretrained Models for Relation Extraction
Yang, X., Ji, T., & Wu, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.453
Yang, F., Xu, X., Wu, M., Wang, X., Shen, L., Qian, Q., & Li, J. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.08.022
Citation Recommendation Chatbot for Professional Communities
Neumann, A. T., Słupczyński, M., Yin, Y., Li, C., & Decker, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42141-9_4 · Cited by 1
Biomedical Knowledge Representation Learning
Zeng, Z., Liu, Z., Lin, Y., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_12
Multilabel Classification for Keyword Determination of Scientific Articles
Rafif, S., Perdana, R. S., & Adikara, P. P. (2023) · Journal of Information Technology and Computer Science · doi:10.25126/jitecs.202382560
Yang, Z., Li, Z., Xu, Z., Gan, Z., & Cao, W. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-6222-8_20
The Fourth Workshop on Insights from Negative Results in NLP
Tafreshi, S., Akula, A., Sedoc, J., Валентинович, Д. А., Rogers, A., & Rumshisky, A. (2023) · doi:10.18653/v1/2023.insights-1 · Cited by 1
Künstliche Intelligenz und ihr Einfluss auf das Unternehmen
Warschat, J., Wierse, A., & Riedel, O. (2023) · Springer eBooks · doi:10.1007/978-3-642-45370-0_131-1
Going beyond research datasets: Novel intent discovery in the industry setting
Chrabrowa, A., Hadeliya, T., Kajtoch, D., Mroczkowski, R., & Rybak, P. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.68
Mining the CORD-19: Review of Previous Work and Design of Topic Modeling Pipeline
Ech-chorfi, S. E., & Zemmouri, E. M. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-43520-1_35
XBNet and text mining-based genetic diseases classification
Abd, D. H., Jassim, M. A., Omri, M. N., Khan, W., & Hussain, A. J. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-11910-w
“To further validate the performance of the proposed XBNet model, we compared it against several state-of-the-art NLP models specifically designed for biomedical text, including BioBERT [ 38 ], SciBERT [ 39 ], PubMed - BERT [ 40 ], and a fine-tuned BERT-based LLM classifier.”
Can Publicly Available Models Understand NATO Language? A Named Entity Recognition Case Study
D’Ercole, R., Kok, A., Eles, P., Valiyev, G., & Street, M. (2023) · International Conference on Military Communications and Information Systems (ICMCIS) · doi:10.1109/icmcis59922.2023.10253530
“Lastly, the consistent ability of models to better recognize certain classes (e.g. person for SciBERT and location for BERT-base) underlines that distinct “out -of-the- shelf” public models leveraged different training data in the pre-training phase which can in turn deliver different capability to perform final tasks.”
MethodInfluentialEstimating the Quality of a Selection of Scientific Papers Using a Collection of Short Texts
Mikhaylov, D. V., & Emelyanov, G. M. (2023) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661823030288
Chourasia, S. S., Kose, T. D., Kharkate, S. K., & Patle, M. R. (2023) · International Journal of Chemistry and Materials Research · doi:10.18488/64.v11i1.3484 · Cited by 1
“Other models which are based on domain specific corpus and knowledge [95-97] have proven to be beneficial in this regard but they have been place above Bidirectional Encoder Representations from the Transformers (BERT).”
BackgroundEnhanced CGSN System for Machine Reading Comprehension
Zheng, L., Jia, H., Hongyan, X., Zhang, X., & Shang, Y.-M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44699-3_8
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.