SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,201–1,300 of 3,596
CoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs
Sun, Q., Liu, Y., Yang, H., Zhang, R., Dun, M., Li, M., Liu, X., Xiao, W., Li, Y., Luan, Z., et al. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00044 · Cited by 17
“Deep learning (DL) has received ubiquitous adoption in many real-world application domains, ranging from object detection [64] and image classification [26] to language processing [5] and machine translation [46].”
BackgroundNishida, N., & Matsumoto, Y. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00451 · Cited by 11
“Implementation Details As the pre-trained transformer encoders, we used SciBERT (Beltagy et al., 2019) and SpanBERT (Joshi et al.”
MethodHybrid Data Augmentation for Citation Function Classification
Zhang, Y., Wang, Y., Sheng, Q. Z., Mahmood, A., Zhang, W. E., & Zhao, R. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191695 · Cited by 16
“SciBERT [28] shares the same architecture with BERTBASE , but is pre-trained on a largescale multi-domain scientific paper corpus.”
MethodGuesmi, M., Chatti, M. A., Kadhim, L., Joarder, S., & Ain, Q. U. (2023) · Multimodal Technologies and Interaction · doi:10.3390/mti7090091 · Cited by 13
Opinion Mining by Convolutional Neural Networks for Maximizing Discoverability of Nanomaterials
Xie, T., Wan, Y., Wang, H., Østrøm, I., Wang, S., He, M., Deng, R. D., Wu, X., Grazian, C., Kit, C., et al. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00746 · Cited by 13
Artificial Intelligence–Assisted Cancer Status Detection in Radiology Reports
Arya, A., Niederhausern, A., Bahadur, N. S., Shah, N. J., Nichols, C., Chatterjee, A., & Philip, J. (2024) · Cancer Research Communications · doi:10.1158/2767-9764.crc-24-0064 · Cited by 11
“Pastfewyearshaveseendevelopmentofseveralbiomedi-calandclinicaldomainadaptedBERTmodels(9–14)andhaveshownveryhigh accuracy and F 1 score in discriminative tasks such as text classification (15–17).”
BackgroundIngram, W. A., Wu, J. F., Kahu, S. Y., Manzoor, J. A., Banerjee, B., Ahuja, A., Choudhury, M. H., Salsabil, L., Shields, W., & Fox, E. A. (2024) · International Journal on Digital Libraries · doi:10.1007/s00799-024-00395-4 · Cited by 13
Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy
Kang, S., Agarwal, S., Jin, B., Lee, D., Yu, H., & Han, J. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645512 · Cited by 20
“To validate the effectiveness of each strategy of ToTER, we provide comparisons with various baselines for each retrieval stage.”
MethodGLAMOR: Graph-based LAnguage MOdel embedding for citation Recommendation
Ali, Z., Qi, G., Ullah, I., Mohammed, A. A. Q., Kefalas, P., & Muhammad, K. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688171 · Cited by 11
“In contrast, CCA outperforms SPECTER2 by leveraging network structure and research papers’ content.”
Wang, S., Bian, J., Huang, X., Zhou, H., & Zhu, S. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3520579 · Cited by 11
“7) PubLabeler PLM : We implemented PubLabeler using different pre-trained language models (PLM), including BERT [16], SciBERT [25], and BioBERT [26], to identify the most suitable language model for the biomedical domain.”
Kong, M., Zhang, Y., Sheng, L., & Hong, K. (2025) · Scientometrics · doi:10.1007/s11192-025-05356-5 · Cited by 11
“In our experiments, we compared the performance of three embedding models (Specter (Cohan et al., 2020), BERT (Kenton & Toutanova, 2019), and SciBERT (Beltagy et al., 2019)) in calculating the Pearson correlation coefficients between citation structural diversity and citations under the CSD-SCEC.”
InfluentialStructured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search
Kim, E., Shrestha, M., Foty, R. G., DeLay, T., & Seyfert‐Margolis, V. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825160 · Cited by 15
Modern machine learning methods for protein property prediction
Dosajh, A., Agrawal, P., Chatterjee, P., & Priyakumar, U. D. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102990 · Cited by 11
Sela, A., Neter, O., Lohr, V., Cihelka, P., Wang, F., Zwilling, M., Sabou, J. P., & Ulman, M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0309688 · Cited by 12
Embedding Methods for Electronic Health Record Research
Kauffman, J., Miotto, R., Klang, E., Costa, A. B., Norgeot, B., Žitnik, M., Shameer, K., Fei, W., Nadkarni, G. N., & Glicksberg, B. S. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094729 · Cited by 16
Ontology-conformal recognition of materials entities using language models
Potu, S. T., Murthy, R. N., Thomas, A., Mishra, L., Prange, N., & Durmaz, A. R. (2025) · Scientific Reports · doi:10.1038/s41598-025-03619-y · Cited by 17
“For instance, SciBERT 13 is optimized for the biomedical sector using scientific texts, while MatSciBERT 14 focuses on materials science articles.”
A supervised machine learning approach for assessing grant peer review reports
Okasa, G., León, A. R., Strinzel, M., Jorstad, A., Milzow, K., Egger, M., & Müller, S. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.23 · Cited by 11
Nentidis, A., Katsimpras, G., Krithara, A., Krallinger, M., Ortega, M. R., Eduard, R.-L., Loukachevitch, N., Sakhovskiy, A., Tutubalina, E. V., Dimitriadis, D., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_12 · Cited by 41
“They fine-tune a separate BERT-based model [17] for 18 https://huggingface.co/cambridgeltl/SapBERT-from-PubMedBERT-fulltext 19 https://huggingface.co/andorei/BERGAMOT-multilingual-GAT English [5], Russian 20 , and multilingual [76] tracks, respectively.”
Zhang, K., Huang, T., Malin, B., Osterman, T. J., Long, Q., & Jiang, X. (2025) · Communications Medicine · doi:10.1038/s43856-025-01116-x · Cited by 12
Automated Coding of Communications in Collaborative Problem‐Solving Tasks Using ChatGPT
Hao, J., Cui, W., Kyllonen, P. C., Kerzabi, E., Liu, L., & Flor, M. (2025) · Journal of Educational Measurement · doi:10.1111/jedm.70014 · Cited by 14
“In addition, we examined whether the presence of scientific terminology influences ChatGPT coding accuracy, given that domain-specific terms have been associated with lower-quality embeddings compared to more commonly used vocabulary (Beltagy et al., 2019; OpenAI, 2023).”
Bridging data and discovery: a survey on knowledge graphs in AI for science
Ding, K., Zhu, Z., Tang, Y., Feng, K., Zhuang, X., Yang, Y., Yang, Y., Du, H., Ni, Z., Wang, S., et al. (2026) · National Science Review · doi:10.1093/nsr/nwag140 · Cited by 17
Dependency parsing of biomedical text with BERT
Kanerva, J., Ginter, F., & Pyysalo, S. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03905-8 · Cited by 14
“…how the previous generation of context-free word representations benefit from initialization on in-domain data [44, 45], deep contextual models such as BERT should generally be pre-trained using data that reflects the domains that the models will be fine-tuned for to maximize performance [46–48].”
BackgroundBiomedical Text Summarization: A Graph-Based Ranking Approach
Gupta, S., Sharaff, A., & Nagwani, N. K. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-16-2008-9_14 · Cited by 11
Modeling Mathematical Notation Semantics in Academic Papers
Jo, H., Kang, D., Head, A., & Hearst, M. A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.266 · Cited by 16
Montenegro, J. L. Z., & da Costa, C. A. (2022) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-022-00115-0 · Cited by 12
“SCI-BERT [10], which leveraged unsupervised pre-training on a large multi-domain corpus of scientific publications, was introduced in and BioBERT, which was pre-trained on biomedical domain corpora (e.”
Domain-Specific Language Model Pre-Training for Korean Tax Law Classification
Gu, Y. H., Piao, X., Yin, H., Dong, J., Zheng, R., & Yoo, S. J. (2022) · IEEE Access · doi:10.1109/access.2022.3164098 · Cited by 11
CORWA: A Citation-Oriented Related Work Annotation Dataset
Li, X., Mandal, B., & Ouyang, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.397 · Cited by 30
MORE: Toward Improving Author Name Disambiguation in Academic Knowledge Graphs
Gong, J., Fang, X., Peng, J., Zhao, Y., Zhao, J., Wang, C., Li, Y., Zhang, J., & Drew, S. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01686-5 · Cited by 11
“…papers representation learning, prior art focused on text messages that mostly used Word2Vec [37], Doc2Vec [38], ELMO [39], and etc. BERT-base [40] and BERT have now derived a series of new versions, pretraining models in specific domains such as Sentence-BERT [41], SCI-BERT [42], and OAG-BERT [7].”
SciFoodNER: Food Named Entity Recognition for Scientific Text
Cenikj, G., Petelin, G., Seljak, B. K., & Eftimov, T. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020459 · Cited by 15
“In order to generate vectorized representation of the textual data, we use four pre-trained transformed-based models: BERT [9], RoBERTa [10], BioBERT [9], and SciBERT [11] which achieve state-of-the-art results in Natural Language Processing.”
BackgroundMethodHashtag-Guided Low-Resource Tweet Classification
Diao, S., Keh, S. S., Pan, L., Tian, Z., Song, Y., & Zhang, T. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583194 · Cited by 16
“This method is called domain-adaptive continual pre-training and has been verifed in other domains, like biomedical domain [23], clinical domain [1], scientifc domain [5], social media domain [30, 39] and so on.”
MethodContextualized medication event extraction with striding NER and multi-turn QA
Tsujimura, T., Yamada, K., Ida, R., Miwa, M., & Sasaki, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104416 · Cited by 14
Large-scale neural biomedical entity linking with layer overwriting
Tsujimura, T., Miwa, M., & Sasaki, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104433 · Cited by 13
Knowledge and skills extraction from the job requirements texts
Евгеньевич, Н. И. (2023) · Ontology of Designing · doi:10.18287/2223-9537-2023-13-2-282-293 · Cited by 11
LMRank: Utilizing Pre-Trained Language Models and Dependency Parsing for Keyphrase Extraction
Giarelis, N., & Karacapilidis, N. I. (2023) · IEEE Access · doi:10.1109/access.2023.3294716 · Cited by 11
Yang, X., Ya, Z., Zuo, J. L., Zhang, X., Wilson, S., & Petzold, L. R. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.446 · Cited by 19
A Dataset of Argumentative Dialogues on Scientific Papers
Ruggeri, F., Mesgar, M., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.425 · Cited by 29
PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English
Chi, J., Ahmad, W. U., Tian, Y., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.31 · Cited by 11
SciLit: A Platform for Joint Scientific Literature Discovery, Summarization and Citation Generation
Gu, N., & Hahnloser, R. H. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.22 · Cited by 20
Scientific Paper Extractive Summarization Enhanced by Citation Graphs
Chen, X., Li, M., Gao, S., Yan, R., Gao, X., & Zhang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.270 · Cited by 17
SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research
Antypas, D., Ushio, A., Barbieri, F., Neves, L., Rezaee, K., Espinosa-Anke, L., Pei, J., & Camacho-Collados, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.838 · Cited by 30
A recollect-tuning method for entity and relation extraction
Yi-zhao, W., Chen, Y., Qin, Y., Tang, R., & Zheng, Q. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123000 · Cited by 12
HyperPIE: Hyperparameter Information Extraction from Scientific Publications
Saier, T., Ohta, M., Asakura, T., & Färber, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_17 · Cited by 15
“Proposed methods for the task utilize BiLSTMs, BERT and SciBERT [5].”
MethodBandwidth Characterization of DeepSpeed on Distributed Large Language Model Training
Hanindhito, B., Patel, B., & John, L. K. (2024) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass61541.2024.00031 · Cited by 18
NSPG: Natural language Processing-based Security Property Generator for Hardware Security Assurance
Meng, X., Srivastava, A., Arunachalam, A., Ray, A., Silva, P. H., Psiakis, R., Makris, Y., & Basu, K. (2024) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3649329.3656255 · Cited by 11
Liu, C., Zhang, X., Zhao, H., Liu, Z., Xi, X., & Yu, L. (2025) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2025.3550203 · Cited by 11
“To validate the effectiveness of the proposed method, we selected the following baseline models for comparison with ) Pretrained Language Models: We fine-tuned Bert and SciBert [44] (the latter is an unsupervised pretrained model based on a wide range of scientific publications).”
Vahidnia, S., Abbasi, A., & Abbass, H. A. (2021) · Journal of Data and Information Science · doi:10.2478/jdis-2021-0024 · Cited by 11
“However, pre-trained BERT models are very popular and are known to achieve the state of the art performance in numerous downstream tasks and datasets, including relation extraction and sentence classification tasks (Beltagy et al., 2019).”
InfluentialSuman, T. A., & Jain, A. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.118 · Cited by 11
Keyword Extraction from Biomedical Documents Using Deep Contextualized Embeddings
Çelikten, A., Uğur, A., & Bulut, H. (2021) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista52262.2021.9548470 · Cited by 11
Measuring Gender Bias in Contextualized Embeddings
Katsarou, S., Rodríguez-Gálvez, B., & Shanahan, J. (2022) · doi:10.3390/cmsf2022003003 · Cited by 19
Medical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning
Ziletti, A., Akbik, A., Berns, C., Herold, T., Legler, M., & Viell, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.21 · Cited by 15
“• sciBERT(Beltagy et al., 2019) is a BERT model which is also pretrained from scratch.”
BackgroundMethodA multi-head adjacent attention-based pyramid layered model for nested named entity recognition
Cui, S., & Joe, I. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07747-8 · Cited by 12
Çelikmasat, G., Akturk, M. E., Ertunc, Y. E., Issifu, A. M., & Ganiz, M. C. (2022) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista55318.2022.9894270 · Cited by 11
“For future work, we shall explore more medical domain embeddings based on transformers such as BioELECTRA [39] , SciBERT [40], and static embeddings such as BioWordVec [41], and cui2vec [42].”
MethodPhraseMap: Attention-Based Keyphrases Recommendation for Information Seeking
Tu, Y., Qiu, R., Wang, Y.-S., Yen, P.-Y., & Shen, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3225114 · Cited by 13
“We directly utilize the SciBERT model [66], and Bio-BERT [67] for the other two literature as the pre-trained datasets of the two models are in the same literature with our focus.”
MethodEmotion Recognition on StackOverflow Posts Using BERT
Bleyl, D., & Buxton, E. K. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020161 · Cited by 15
“domain range from pretraining BERT from scratch with a domain-specific vocabulary [27] to fine-tuning the original Fig.”
BackgroundSpan-based Named Entity Recognition by Generating and Compressing Information
Nguyen, N. T. H., Miwa, M., & Ananiadou, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.146 · Cited by 11
Sequence Labeling for Disambiguating Medical Abbreviations
Çevik, M., Jafari, S. M., Myers, M., & Yıldırım, S. (2023) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-023-00146-1 · Cited by 14
“(cid:129) SciBERT : This model has two key differences from BERT base : its pre-training corpus and vocabulary [41].”
MethodUsing word embedding to detect keywords in texts modeled as complex networks
Tohalino, J. A. V., Silva, T. C., & Amancio, D. R. (2024) · Scientometrics · doi:10.1007/s11192-024-05055-7 · Cited by 11
“Pre-trained models tend to be more robust, which enhances model generalization (Beltagy et al., 2019).”
De-identification of Privacy-related Entities in Job PostingsPreprint
Jensen, K. N., Zhang, M., & Plank, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11223 · Cited by 22
“Next to “vanilla” BERT, they experiment with fine-tuning different domain specific pre-trained language models, such as SciBERT (Beltagy et al., 2019) and BioBERT (Lee et al., 2020).”
FBERT: A Neural Transformer for Identifying Offensive ContentPreprint
Sarkar, D., Zampieri, M., Ranasinghe, T., & Ororbia, A. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05074 · Cited by 11
“, 2020), scientific texts (SciBERT) (Beltagy et al., 2019), and microblogging (BerTweet) (Nguyen et al.”
BackgroundZhang, Z., Jasaitis, T., Freeman, R., Alfrjani, R., & Funk, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.03458 · Cited by 11
“Studies (Alsentzer et al., 2019; Beltagy et al., 2019) have shown that usually, language models need to be further trained using domain specific corpora when they are to be applied to domain specific contexts.”
BackgroundLabrador: Exploring the Limits of Masked Language Modeling for Laboratory DataPreprint
Bellamy, D. R., Kumar, B., Wang, C., & Beam, A. L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.11502 · Cited by 12
“For instance, existing applications of MLMs in medicine have already yielded promising results in tasks related to medical text understanding (Lee et al., 2020; Alsentzer et al., 2019; Huang et al., 2019; Yang et al., 2019; Beltagy et al., 2019).”
Large Language Models for Market Research: A Data-augmentation ApproachPreprint
Wang, M., Zhang, D., & Zhang, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5057769 · Cited by 16
Druskat, S., Hong, N. P. C., Buzzard, S., Konovalov, O., & Kornek, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.14602 · Cited by 11
“The detection used a machine leraning model based on SciBERT(Beltagy et al., 2019), also trained on the SoftCite dataset (Du et al., 2021).”
MethodPatenTEB: A Comprehensive Benchmark and Model Family for Patent Text EmbeddingPreprint
Ayaou, I., & Cavallucci, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.22264 · Cited by 12
“Scientific domains provide closer parallels: SciBERT [3] and BioBERT [21] show that scientific vocabulary requires specialized pretraining.”
Towards Agentic Intelligence for Materials SciencePreprint
Zhang, H., Li, Y., Huang, W., Hou, Z., Song, Y., Liu, X., Effaty, F., Jiang, J., Wu, S., Ding, Q., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.00169 · Cited by 15
Al-Hussaini, I., Nakajima An, D., Lee, A. J., Bi, S., & Mitchell, C. S. (2022) · 2022 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata55660.2022.10020807 · Cited by 10
“• SciBERT [44]: Trained a BERT [28] model on scientific papers taken from 1.”
MethodMS2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · arXiv · doi:10.48550/arxiv.2104.06486 · Cited by 10
“…with work spanning the whole NLP pipeline: datasets (S2ORC (Lo et al., 2020), CORD-19 (Wang et al., 2020b)), pretrained transformer models (SciBERT (Beltagy et al., 2019), BioBERT (Lee et al., 2020), ClinicalBERT (Huang et al., 2019), SPECTER (Cohan et al., 2020)), NLP tasks like NER (Nye et al.,…”
MethodInfluentialSelf-citationTri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER
Zeng, Q., Yu, W., Yu, M., Jiang, T., Weninger, T., & Jiang, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.429 · Cited by 10
“Therefore, SciBERT (Beltagy et al., 2019) was developed on paper corpus (3.”
BackgroundMethodInfluentialLearning the syntax of plant assemblages
Leblanc, C., Bonnet, P., Servajean, M., Thuiller, W., Chytrý, M., Aćić, S., Argagnon, O., Biurrun, I., Bonari, G., Bruelheide, H., et al. (2025) · Nature Plants · doi:10.1038/s41477-025-02105-7 · Cited by 10
Neves, M., Klippert, A., Knöspel, F., Rudeck, J., Stolz, A., Bán, Z., Becker, M., Diederich, K., Grune, B., Kahnau, P., et al. (2023) · Journal of Biomedical Semantics · doi:10.1186/s13326-023-00292-w · Cited by 12
S2abEL: A Dataset for Entity Linking from Scientific Tables
Lou, Y., Kuehl, B., Bransom, E., Feldman, S., Naik, A., & Downey, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.186 · Cited by 16
Bhaskar, A., & Stodden, V. (2024) · ACM-REP · doi:10.1145/3641525.3663629 · Cited by 15
“…abstracts followed by the metrics and phrases as learning examples • Insert abstract of corresponding preprint We also also assessed the feasibility of BERT based models such as SciBERT [3] and SPECTRE [25] in extracting and interpreting reproducibility metrics from scientific research manuscripts.”
A Hierarchical Context Augmentation Method to Improve Retrieval-Augmented LLMs on Scientific Papers
Che, T.-Y., Mao, X.-L., Lan, T., & Huang, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671847 · Cited by 16
“Scientific papers are the main carrier for storing, disseminating and learning professional knowledge, and there have been a lot of meaningful researches on this domain, such as SciBERT[5], SciERC [23] and so on.”
Ma, C., & Wolfinger, R. D. (2024) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.4c00134 · Cited by 10
Hierarchical Graph Latent Diffusion Model for Conditional Molecule Generation
Bian, T., Niu, Y., Chang, H., Yan, D., Huang, J., Rong, Y., Xu, T., Li, J., & Cheng, H. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679547 · Cited by 20
“We optimize the objec-tive function , which is the expected squared error between the denoising neural network’s output and the original noise, weighted by the reweighting terms 𝑤 ( 𝑡 ) , where S is the condition embedding matrix obtained by encoding numerical property scores using MLP or text-based property descripti…”
InfluentialNer4Opt: named entity recognition for optimization modelling from natural language
Kadıoğlu, S., Dakle, P. P., Uppuluri, K., Politi, R., Raghavan, P., Rallabandi, S. K., & Srinivasamurthy, R. (2024) · Constraints · doi:10.1007/s10601-024-09376-5 · Cited by 16
“For domain-specific tasks (e.g., sentiment analysis in finance), performance can be improved further using domain-specific corpora to fine-tune pre-trained models [48–51].”
Can Pre-trained Language Models Generate Titles for Research Papers?
Rehman, T., Sanyal, D. K., & Chattopadhyay, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0865-2_13 · Cited by 15
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 11
Improved paper recommendation model incorporating knowledge graph embedding with IRGAN model
Thierry, N., Chatelain, I. B. C., Ali, Z., & Kefalas, P. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3710898 · Cited by 12
“Similarly, Lu et al. [38] introduced an approach that utilizes DPCNN to take advantage of deep semantic relationships among sentences during the step of feature extraction and they utilize SciBERT [5] to learn the scientific paper content.”
ContReviews: A content-based recommendation system for updating Living Evidences in health care
Tenti, P., Thomas, J., Peñaloza, R., & Pasi, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112981 · Cited by 11
Attention-based interactive multi-level feature fusion for named entity recognition
Xu, Y., & Chen, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-86718-0 · Cited by 14
“For example, Beltagy et al. 20 proposed a BERT-based method called SciBERT, which integrates token, segment, and positional embeddings into NER models.”
Kallergis, G., Asgari, E., Empting, M., Hirsch, A. K. H., Klawonn, F., & McHardy, A. C. (2025) · Communications Chemistry · doi:10.1038/s42004-025-01484-4 · Cited by 11
Sadruddin, S., D’Souza, J., Poupaki, E., Watkins, A., Giglou, H. B., Rula, A., Karasulu, B., Auer, S., Mackus, A. J. M., & Kessels, W. M. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94578-6_14 · Cited by 17
Adapting language models for mental health analysis on social media
Aragón, M. E., López-Monroy, A. P., Montes-y-Gómez, M., & Losada, D. E. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103217 · Cited by 13
Fan, J., Wang, D., & Zheng, Y. (2025) · Sustainability · doi:10.3390/su17156971 · Cited by 11
Automatic Generation of Job Safety Reports with Explainable RAG-Based LLMs
Bernardi, M. L., Cimitile, M., Panella, G., Pecori, R., & Simoncelli, G. (2025) · Information Systems Frontiers · doi:10.1007/s10796-025-10634-x · Cited by 14
Khambete, M., Su, W., Garcia, J., & Badgeley, M. A. (2020) · PubMed · doi:10.48550/arxiv.2008.06606 · Cited by 11
Fine-Grained Named Entity Recognition with Distant Supervision in COVID-19 Literature
Wang, X., Song, X., Li, B., Zhou, K., Li, Q., & Han, J. (2020) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313126 · Cited by 10
“For SciBERT, since it does not release its pre-trained models, we use its SciBERT embeddings 6 and re-trained the model on BC5CDR.”
MethodInfluentialDomain-oriented Language Modeling with Adaptive Hybrid Masking and Optimal Transport Alignment
Zhang, D., Yuan, Z., Liu, Y., Liu, H., Zhuang, F., Xiong, H., & Chen, H. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467215 · Cited by 10
“For instance, BioBERT [13] and SciBERT [2] are pretrained leveraging large-scale domain-specific corpora for biomedical and scientific domain tasks respectively.”
We Can Explain Your Research in Layman's Terms: Towards Automating Science Journalism at Scale
Dangovski, R., Shen, M., Byrd, D., Jing, L., Tsvetkova, D., Nakov, P., & Soljacic, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17507 · Cited by 18
“Beltagy, Lo, and Cohan (2019) fine-tuned BERT on scientific articles and improved the baselines for some downstream scientific tasks.”
BackgroundMethodInfluentialDebunking health fake news with domain specific pre-trained model
Kumari, S., Reddy, H. K., Kulkarni, C. S., & Gowthami, V. (2021) · Global Transitions Proceedings · doi:10.1016/j.gltp.2021.08.038 · Cited by 14
Re-Ranking System with BERT for Biomedical Concept Normalization
Cho, H., Choi, D., & Lee, H. (2021) · IEEE Access · doi:10.1109/access.2021.3108445 · Cited by 11
Graphine: A Dataset for Graph-aware Terminology Definition Generation
Liu, Z., Wang, S., Gu, Y., Zhang, R., Zhang, M., & Wang, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.278 · Cited by 10
“Within the three domain-specific pretrained language models, BioBERT and SciBERT obtained the most prominent performance.”
BackgroundMethodResultInfluentialSchmidt, T., Dennerlein, K., & Wolff, C. G. (2021) · Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL) · doi:10.18653/v1/2021.latechclfl-1.8 · Cited by 10
An empirical study of the design choices for local citation recommendation systems
Medić, Z., & Šnajder, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116852 · Cited by 10
Balancing Precision and Recall for Neural Biomedical Event Extraction
Su, F., Zhang, Y., Li, F., & Ji, D. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3161146 · Cited by 10
Cross-document attention-based gated fusion network for automated medical licensing exam
Liu, J., Ren, J., Lu, Z., He, W., Cui, M., Zhang, Z., & Bai, R. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117588 · Cited by 10
Sentiment and intent classification of in-text citations using BERT
Visser, R., & Dunaiski, M. (2022) · EPiC series in computing · doi:10.29007/wk21 · Cited by 14
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.