SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
“We highlight specific systems developed by users with different goals following the architect, trainer, and end-user distinction of Strobelt et al. (2017): Case 1: Model Architects AllenAI, a major NLP research lab, developed a new pretrained model for improved extraction from biomedical texts called SciBERT (Beltagy…”
InfluentialOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
“However, in both application areas, the training data has been shown to have problematic characteristics [18, 38, 42, 47, 61] resulting in models that encode stereotypical and derogatory associations along gender, race, ethnicity, and disability status [11, 12, 69, 69, 132, 132, 157].”
BackgroundHan (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
“BioBERT [69] uses Transformer architecture for biomedical text mining tasks; SciBERT [7] is developed by training Transformer on 114M scientific articles covering biomedical and computer science field, aiming to execute NLP tasks related to scientific domain more precisely; Huang et al. [55] proposes ClinicalBERT util…”
MethodHuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
“, using in-domain text in pretraining should help with domain-specific applications. Indeed, prior work has shown that pretraining with PubMed text leads to better performance in biomedical NLP tasks [8, 33, 44]. The main question is whether pretraining should include text from other domains. The prevailing assumption…”
BackgroundInfluentialDon’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Self-citationThe false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“Recently, some studies have proposed PTMs trained on specialty corpora, such as BioBERT [98] for biomedical text, SciBERT [11] for scientific text, ClinicalBERT [69, 3] for clinical text.”
BackgroundLEGAL-BERT: The Muppets straight out of Law School
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., & Androutsopoulos, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.261 · Cited by 994
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
“When applying to specific domain (e.g., biomedicine), BERT models pre-trained on general domain can be further improved if pre-trained on in-domain text data [8, 18, 10].”
BackgroundMethodSPECTER: Document-level Representation Learning using Citation-informed Transformers
Cohan, A., Feldman, S., Beltagy, I., Downey, D., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.207 · Cited by 1,108
Self-citationBERTweet: A pre-trained language model for English Tweets
Nguyen, D. Q., Vu, T., & Tuan Nguyen, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.2 · Cited by 843
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., D., M., Châtel, G., Chaumond, J., Rault, T., Voss, C., Wang, F., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3385997 · Cited by 1,037
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 557
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886
Green Algorithms: Quantifying the Carbon Footprint of Computation
Lannelongue, L., Grealey, J., & Inouye, M. (2021) · Advanced Science · doi:10.1002/advs.202100707 · Cited by 543
A Frustratingly Easy Approach for Entity and Relation Extraction
Zhong, Z., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.5 · Cited by 477
“7 We also use scibert-scivocab-uncased (Beltagy et al., 2019) as the base encoder for SciERC, as this in-domain pre-trained model is shown to be more effective than BERT (Wadden et al., 2019).”
BackgroundMethodInfluentialERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
MTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548
“, 2020a) relies on the pre-trained SciBERT (Beltagy et al., 2019) variant instead and fine-tunes on citation graphs.”
MethodDeep entity matching with pre-trained language models
Li, Y., Li, J., Suhara, Y., Doan, A., & Tan, W.-C. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3421424.3421431 · Cited by 339
CORD-19: The COVID-19 Open Research Dataset
Wang, L. L., Lo, K., Chandrasekhar, Y., Reas, R., Yang, J., Burdick, D., Eide, D., Funk, K., Katsis, Y., Kinney, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.10706
Self-citationA Frustratingly Easy Approach for Entity and Relation Extraction
Zhong, Z., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2010.12812 · Cited by 531
“5 We also use scibert-scivocab-uncased (Beltagy et al., 2019) as the base encoder for SciERC, as this in-domain pre-trained model is shown to be more effective than BERT (Wadden et al., 2019).”
BackgroundMethodSelf-Alignment Pretraining for Biomedical Entity Representations
Liu, F., Shareghi, E., Meng, Z., Basaldella, M., & Collier, N. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.334 · Cited by 335
LinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.551 · Cited by 327
Span-Based Joint Entity and Relation Extraction with Transformer Pre-Training
Markus, E., & Adrian, U. (2020) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia200321 · Cited by 168
S2ORC: The Semantic Scholar Open Research Corpus
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.447 · Cited by 457
Self-citationEvolution of Semantic Similarity—A Survey
Chandrasekaran, D., & Mago, V. (2022) · ACM Computing Surveys · doi:10.1145/3440755 · Cited by 303
Multilingual is not enough: BERT for Finnish
Virtanen, A., Kanerva, J., Ilo, R., Luoma, J., Luotolahti, J., Salakoski, T., Ginter, F., & Pyysalo, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07076 · Cited by 104
MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24188 · Cited by 137
“LLM chatbots, e.g., C HAT GPT [36], Google Bard [20], and Bing Chat [22], showcase an impressive § Equal Contribution † Corresponding Author capability to assist in various tasks with their high-quality generation [13], [34], [35].”
BackgroundAlmanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424
Shifting machine learning for healthcare from development to deployment and from models to data
Zhang, A., Xing, L., Zou, J., & Wu, J. C. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00898-y · Cited by 420
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
“Notable LLM chatbots like CHATGPT [31], Google Bard [17], and Bing Chat [19] exhibit a remarkable competence in aiding with various tasks due to their exceptional text generation abilities [12], [29], [30].”
BackgroundMatSciBERT: A materials domain language model for text mining and information extraction
Gupta, T., Zaki, M., Krishnan, N. M. A., & Mausam (2022) · npj Computational Materials · doi:10.1038/s41524-022-00784-w · Cited by 342
“Because of the larger overlap with the vocabulary of SciBERT, we tokenize our corpus using the SciBERT vocabulary and initialize the MatSciBERT weights with that of SciBERT as made publicly available by Beltagy et al. (2019)21.”
MethodFact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382
Self-citationA Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Hedderich, M. A., Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.201 · Cited by 260
Llemma: An Open Language Model For Mathematics
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., & Welleck, S. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.10631 · Cited by 505
“Other work explores adapting pretrained models to novel domains by continued training (Rozière et al., 2023; Beltagy et al., 2019), parameter-efficient finetuning methods (Yong et al.”
BackgroundLarge language models in health care: Development, applications, and challenges
Yang, R., Tan, T. F., Lu, W., Thirunavukarasu, A. J., Ting, D. S. W., & Liu, N. (2023) · Health Care Science · doi:10.1002/hcs2.61 · Cited by 366
“SCIBERT was trained from scratch on the Semantic Scholar corpora (18% computer science papers and 82% biomedical papers), rather than fine ‐ tuning the generalist BERT model [15].”
MethodBERT applications in natural language processing: a review
Gardazi, N. M., Daud, A., Malik, M. K., Bukhari, A., Alsahfi, T., & Alshemaimri, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11162-5 · Cited by 251
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 430
“For example, the word actyeltransferase is split into [“ace”, “ty”, “lt”, “ran”, “sf”, “eras”, “e’] by BERT model whereas the same word is split into meaning tokens [“acetyl”, “transferase”] by SciBERT which has domain-specific vocabulary [47].”
BackgroundMethodInfluentialBloombergGPT: A Large Language Model for Finance
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., & Mann, G. (2023) · arXiv · Cited by 357
“Beltagy et al. (2019) constructed a vocabulary specific to scientific text and observed that their domain-specific trained vocabulary only had a 42% overlap with the non-domain-specific BERT vocabulary trained on general domain text.”
BackgroundMethodDeep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350
“SciBERT is an extension to BERT that has been pre-trained on Scientific Text.”
MethodDocument-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling
Zhou, W., Huang, K., Ma, T., & Huang, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17717 · Cited by 316
Zhou, H., Hu, C., Yuan, Y., Cui, Y., Jin, Y., Chen, C., Wu, H., Yuan, D., Jiang, L., Wu, D., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3465447 · Cited by 244
“LLM is equipped with rich internalized knowledge about the world in the pre-training stage [53].”
BackgroundTranslation between Molecules and Natural Language
Edwards, C., Lai, T., Ros, K., Honke, G., Cho, K., & Ji, H. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.26 · Cited by 165
Lawformer: A pre-trained language model for Chinese legal long documents
Xiao, C., Hu, X., Liu, Z., Tu, C., & Sun, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.06.003 · Cited by 278
ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model
Huang, H., Zheng, O., Wang, D., Yin, J., Wang, Z., Ding, S., Yin, H., Xu, C., Yang, R., Zheng, Q., et al. (2023) · International Journal of Oral Science · doi:10.1038/s41368-023-00239-y · Cited by 324
Neural Unsupervised Domain Adaptation in NLP—A Survey
Ramponi, A., & Plank, B. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.603 · Cited by 230
A comprehensive survey on automatic text summarization with exploration of LLM-based methods
Zhang, Y., Jin, H., Meng, D., Wang, J., & Tan, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131928 · Cited by 324
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
“Zhang et al. [127] further pretrained SciBERT [105] on MIMIC-III clinical notes and showed that the performance of the model is different for different groups.”
MethodTransformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 197
“He et al. [140] proposed a procedure for consumer health question answering and medical language inference tasks using models such as BERT[21], BioBERT[55], SciBERT[101], ClinicalBERT[56], BlueBERT[84], and ALBERT[97].”
Neural Natural Language Processing for unstructured data in electronic health records: A review
Li, I., Pan, J., Goldwasser, J., Verma, N., Wong, W. P., Nuzumlalı, M. Y., Rosand, B., Li, Y., Zhang, M., Chang, D., et al. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100511 · Cited by 254
Large Language Models on Graphs: A Comprehensive Survey
Jin, B., Liu, G., Han, C., Jiang, M., Ji, H., & Han, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3469578 · Cited by 147
“SPECTER [51] constructs the positive text/node pairs with the citation relation, explores random negatives and structure hard negatives, and fine-tunes SciBERT [25] with the triplet loss.”
BackgroundMethodDomain Adaptation: Challenges, Methods, Datasets, and Applications
Singhal, P., Walambe, R., Ramanna, S., & Kotecha, K. (2023) · IEEE Access · doi:10.1109/access.2023.3237025 · Cited by 234
A Bibliometric Review of Large Language Models Research from 2017 to 2023
Fan, L., Li, L., Ma, Z., Lee, S., Yu, H., & Hemphill, L. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3664930 · Cited by 196
“We experimented with using SciBERT (Beltagy et al., 2019), which can improve some peripheral clustering results, while the overall topical coherence is less than the default BERT model as word embeddings.”
InfluentialA Survey of Text Representation and Embedding Techniques in NLP
Patil, R., Boit, S., Gudivada, V., & Nandigam, J. (2023) · IEEE Access · doi:10.1109/access.2023.3266377 · Cited by 214
Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review
Cui, C., Yang, H., Wang, Y., Zhao, S., Asad, Z., Coburn, L. A., Wilson, K. T., Landman, B. A., & Huo, Y. (2023) · Progress in Biomedical Engineering · doi:10.1088/2516-1091/acc2fe · Cited by 218
“Meanwhile, several studies [27] [22] [20] deployed multi-task learning and showed improvements.”
TimeLMs: Diachronic Language Models from Twitter
Loureiro, D., Barbieri, F., Neves, L., Espinosa Anke, L., & Camacho-collados, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.25 · Cited by 263
Multi-modal molecule structure–text model for text-based retrieval and editing
Liu, S., Nie, W., Wang, C., Lu, J., Qiao, Z., Liu, L., Tang, J., Xiao, C., & Anandkumar, A. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00759-6 · Cited by 179
CLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 281
“The pre-trained SciBERT (Beltagy et al., 2019) is adopted as the knowledge encoder of MOTOR.”
A Survey of Knowledge Enhanced Pre-Trained Language Models
Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., & Li, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3310002 · Cited by 196
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“For this reason, it was suggested to use F2 score [141] over the F1 score.”
InfluentialGPT-RE: In-context Learning for Relation Extraction using Large Language Models
Wan, Z., Cheng, F., Mao, Z., Liu, Q., Song, H., Li, J., & Kurohashi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.214 · Cited by 162
GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks
Liu, Z., Yu, X., Fang, Y., & Zhang, X. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583386 · Cited by 195
“Inspired by the application of pre-training models in language [2, 7] and vision [1, 29] domains, graph pretraining [49] emerges as a powerful paradigm that leverages selfsupervision on label-free graphs to learn intrinsic graph properties.”
MethodA Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 169
“Except for the general domain, eforts have been proposed to extend PLMs in speciic domains such as: [71] trains CodeBERT in the programming language and [18] trains SciBERT on scientiic publications and biological sequence.”
BackgroundMethodInfluentialDocument-level Relation Extraction as Semantic Segmentation
Zhang, N., Chen, X., Xie, X., Deng, S., Tan, C., Chen, M., Huang, F., Si, L., & Chen, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/551 · Cited by 181
An Analysis of Simple Data Augmentation for Named Entity Recognition
Dai, X., & Adel, H. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.343 · Cited by 184
Explainable AI for Bioinformatics: Methods, Tools and Applications
Karim, M. R., Islam, T., Shajalal, M., Beyan, O., Lange, C., Cochez, M., Rebholz-Schuhmann, D., & Decker, S. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad236 · Cited by 196
Scientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Yu, J., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 122
“SciBERT [16] 2019.09 117M BERT Semantic Scholar ScholarBERT [85] 2023.05 340M, 770M BERT Wiki, Books, etc. DARWIN-Base [243] 2023.08 7B LLaMA SciQ, Web of Science SciGLM [264] 2024.01 6B, 32B ChatGLM3 SciInstruct Uni-SMART [29] 2024.03 7B — Patents, news, literature, etc. × SciDFM[204] 2024.09…”
InfluentialInformation Retrieval: Recent Advances and Beyond
Hambarde, K., & Proença, H. P. M. C. (2023) · IEEE Access · doi:10.1109/access.2023.3295776 · Cited by 284
“Future research should explore more efficient ways to model this knowledge for IR, as well as strive for greater interpretability in modeling knowledge for downstream tasks [270], [271], [272], [273], [274].”
MethodSecureBERT: A Domain-Specific Language Model for Cybersecurity
Aghaei, E., Niu, X., Shadid, W., & Al-Shaer, E. (2023) · Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-25538-0_3 · Cited by 210
“Beltagy et al. [7] unveiled SciBERT following the exact BERT’s architecture, a model that improves performance on downstream scientific NLP tasks by exploiting unsupervised pretraining from scratch on a 1 .”
Xu, B., Wang, Q., Lyu, Y., Zhu, Y., & Mao, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17665 · Cited by 192
End-to-End Transformer-Based Models in Textual-Based NLP
Rahali, A., & Akhloufi, M. A. (2023) · AI · doi:10.3390/ai4010004 · Cited by 164
BioMedGPT: An Open Multimodal Large Language Model for BioMedicine
Luo, Y., Zhang, J., Fan, S., Yang, K., Hong, M., Wu, Y., Qiao, M., & Nie, Z. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3505955 · Cited by 227
Dynamic Graph Enhanced Contrastive Learning for Chest X-Ray Report Generation
Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., & Chang, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00325 · Cited by 206
“Settings Gpre KI IRC IRM ViT [10] SciBert [4] CIDEr BLEU-4 ROUGE-L METEOR”
MethodInfluentialNicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021) · bioRxiv (published in Quantitative Science Studies) · doi:10.1101/2021.03.15.435418 · Cited by 202
State of the art: a review of sentiment analysis based on sequential transfer learning
Chan, J. Y.-L., Bea, K. T., Leow, S. M. H., Phoong, S. W., & Cheng, W. K. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10183-8 · Cited by 179
“For example, BioBERT (Lee et al. 2020) was pretrained based on a massive biomedical corpus, SciBERT (Beltagy et al. 2019) was pretrained based on the multidomain corpus of scientific publication, and FinBERT (Yang et al. 2020b) was pretrained based on the financial corpus of the corporate report,…”
MethodLewis, P., Ott, M., Du, J., & Stoyanov, V. (2020) · Proceedings of the 3rd Clinical Natural Language Processing Workshop · doi:10.18653/v1/2020.clinicalnlp-1.17 · Cited by 167
Explainable Automated Fact-Checking for Public Health Claims
Kotonya, N., & Toni, F. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.623 · Cited by 195
Naseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3434237 · Cited by 194
“Some of the most famous in the biomedical domain are Sci-BERT [9], BioBERT [79] and BioAL-BERT [109].”
BackgroundUnleashing the Power of Artificial Intelligence in Materials Design
Badini, S., Regondi, S., & Pugliese, R. (2023) · Materials · doi:10.3390/ma16175927 · Cited by 179
“These adaptations enable LLMs to be effectively applied in various domains, including dataset mining, molecular modeling, microstructure generation, and material structure extraction [77,78].”
BackgroundHadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 103
Cold-start active learning through self-supervised language modeling
Yuan, M., Lin, H.-T., & Boyd-Graber, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.637 · Cited by 84
pysentimiento: A Python Toolkit for Opinion Mining and Social NLP tasksPreprint
Pérez, J. M., Rajngewerc, M., Giudici, J. C., Furman, D., Luque, F. M., Alemany, L. A., & Martínez, M. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09462 · Cited by 204
“SciBERT (Beltagy et al, 2019) is a BERT model trained on scientific texts, while MediBERT (Rasmy et al, 2021) was trained on medical documents.”
MethodFields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 249
AI-based language models powering drug discovery and development
Liu, Z., Roberts, R. A., Lal-Nag, M., Chen, X., Huang, R., & Tong, W. (2021) · Drug Discovery Today · doi:10.1016/j.drudis.2021.06.009 · Cited by 167
CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., Huang, F., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.544 · Cited by 152
Opportunities and challenges of text mining in materials research
Kononova, O., He, T., Huo, H., Trewartha, A., Olivetti, E. A., & Ceder, G. (2021) · iScience · doi:10.1016/j.isci.2021.102155 · Cited by 182
Trewartha, A., Walker, N., Huo, H., Lee, S., Cruse, K., Dagdelen, J., Dunn, A., Persson, K. A., Ceder, G., & Jain, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100488 · Cited by 163
“It has been shown that pretraining on different corpora can lead to different performances.(21) This is of particular interest in technical fields where commonly used words and phrases may not be well-represented or even carry the same meaning in other contexts.”
BackgroundMethodInfluentialLearning to Faithfully Rationalize by Construction
Jain, S., Wiegreffe, S., Pinter, Y., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.409 · Cited by 136
Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again
Jimenez Gutierrez, B., McNeal, N., Washington, C., Chen, Y., Li, L., Sun, H., & Su, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.329 · Cited by 115
A survey on Named Entity Recognition — datasets, tools, and methodologies
Jehangir, B., Radhakrishnan, S., & Agarwal, R. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100017 · Cited by 239
BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
Yuan, H., Yuan, Z., Gan, R., Zhang, J., Xie, Y., & Yu, S. (2022) · Proceedings of the 21st Workshop on Biomedical Language Processing · doi:10.18653/v1/2022.bionlp-1.9 · Cited by 127
Jin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W. J., & Lu, Z. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad651 · Cited by 175
Brain-inspired learning in artificial neural networks: A review
Schmidgall, S., Ziaei, R., Achterberg, J., Kirsch, L., Hajiseyedrazi, S. P., & Eshraghian, J. K. (2024) · APL Machine Learning · doi:10.1063/5.0186054 · Cited by 219
Esteva, A., Kale, A., Paulus, R., Hashimoto, K., Yin, W., Radev, D., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00437-0 · Cited by 109
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.