SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“These tend to perform better than general purpose models on scientific domain tasks, and serve as a foundation for learning dense embeddings of scientific documents.”

“Tay et al. (2022) use control codes to switch be-tween three de-noising modes during pre-training, and associate every downstream task with a mode during fine-tuning.”

“Aribandi et al. (2022) pre-train T5 (Raffel et al., 2020) on C4 span denoising and 107 other tasks across 8 task families.”

“Representing Scientific Documents Prior works have produced large scale language models pre-trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022).”

“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”

“, 2020)) can be tailored to produce high-quality representations of documents with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“Within scientific embeddings, SPECTER [Cohan et al., 2020], SPECTER2 [Singh et al., 2023], and SciNCL [Osten-dorff et al., 2022] use direct citations as a contrastive training signal; one of our findings is that this training does not transfer to cosine neighborhoods that match a well-constructed…”

“To compare embedding models, we construct a sequential tour from 3M arXiv title-abstract embeddings produced by four models spanning 2023– 2026: SPECTER2 [35], BGE-M3 [11], Nomic Embed Text v2 [30], and F2LLM-v2-8B [47] (top-ranked on clustering benchmarks), each reduced to 2D with default UMAP.”

“Our model selection considers general-purpose language models (RoBERTa-base and RoBERTa-large (Liu et al., 2019)), and scientific language models pre-trained on academic documents (SPECTER2 (Cohan et al., 2020; Singh et al., 2022) and ScholarBERT (Hong et al., 2022)).”

“We ship the three-component hybrid (whitened SPECTER2 plus concept TF-IDF plus venue LDA) rather than raw SPECTER2, even though a small ablation we ran on this corpus gives raw a roughly one-to two-percentage-point edge on phantom-collaborator precision at 𝐾 = 20 .”

“A fundamental limitation of all these systems is that they optimise for topical relevance : any cited paper is a positive label, with no distinction be-tween a core experimental baseline and tangential background.”

“BERTopic and its scientific variants (e.g, SciBERT [24], Specter2 [25]) represents a more recent advancement in topic modeling post their not very sophisticated semantic predecessor LSA (Latent Semantic Analysis) [26].”

“To reduce the costs of human annotation and obtain high-quality relevance assessments, some studies have leveraged indirect relevance signals, such as citation data, treating authors as implicit annotators who indicate relevance through their citations [14, 20].”

“Fine-Grained.”

“SPECTER2 embeddings of 191,570 sepsis-related text nodes were projected to two dimensions using PCA followed by UMAP and clustered with the Leiden algorithm.”

“Furthermore, it explicitly models citation links as signals of deeper semantic relatedness between documents [18].”

“After training, we chose our fine-tune of the Specter2 model, which was initially trained entirely on scientific literature from the Semantic Scholar database [8].”

“The best-performing algorithms for assigning reviewers to papers in peer review (OpenRe-view 2025), such as SPECTER2 (Singh et al. 2023) and SciNCL (Ostendorff et al. 2022), crucially rely on citation networks and past publications.”

“We generate dense embeddings from award text using SPECTER2 [Singh et al., 2022], a powerful science-domain adapted embedding model.”

“In contrast, PAT-SPECTER is based on SPECTER2 (Singh et al., 2022), a transformer model pretrained on scientific articles using citation supervision, and is similarly fine-tuned on examiner citation triplets to adapt it to the patent domain.”

“For example, Cohan et al.[18] and Singh et al.[19] developed the SPECTER framework series, which uses citation graphs to build triplet supervision signals and trains SciBERT [20] to obtain high-quality document representations with strong transferability.”

“SPECTER2 is a transformer model pre-trained on scientific texts and is suitable for various NLP tasks involving academic texts [35].”

“Services such as the Semantic Scholar literature graph (Ammar et al., 2018) rely on embeddings produced by SPECTER-2 (Singh et al., 2022 relationship between scientific documents and construct literature feeds for researchers that align with their interests.”

“We encode each query with the SPECTER-2 ad-hoc query adapter and recorded the top 50 results when queried against the index.”

“We embed both the questions and text chunks using the SPECTER2 model (Singh et al., 2023) specifically for scientific documents pretrained on 4 See Appendix E for full prompts. citation relations, compute cosine similarity, and select the top three relevant chunks per question.”

“In recent years, SPECTER-RM (Cohan et al. 2020) and its latest version, SPECTER2-RM (Singh et al. 2023b), generate embeddings of scientific papers using citation-aware fine-tuning on SciBERT.”

“Utilizing semantic information from paper titles and abstracts [13], our system maps a two-dimensional semantic space, indicating the relative positions of researchers and datasets within the CM4AI knowledge domain.”

“SPECTER introduced citation-supervised transformer representations for scientific papers [1], SciNCL improved robustness via neighborhood contrastive learning [2], and subsequent benchmarks have consolidated best practices for scientific representation evaluation [3].”

“We also use the recent Specter2 version (Singh et al., 2022).”

“For the evidence retrieval task with QASPER, Specter 2 is used with the AdHoc Query Adapter for questions and the Proximity Adapter for paragraphs (Singh et al., 2022).”

“While scientific benchmarks such as MMLU (Hendrycks et al., 2020), SciBench (Wang et al., 2023), SciQ (Welbl et al., 2017), SciRepE-val (Singh et al., 2022), SciQA (Auer et al., 2023), and QASA (Lee et al., 2023) are used for multi-disciplinary scientific QA evaluations, field-specific benchmarks…”