SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“These tend to perform better than general purpose models on scientific domain tasks, and serve as a foundation for learning dense embeddings of scientific documents.”

“Tay et al. (2022) use control codes to switch be-tween three de-noising modes during pre-training, and associate every downstream task with a mode during fine-tuning.”

“Aribandi et al. (2022) pre-train T5 (Raffel et al., 2020) on C4 span denoising and 107 other tasks across 8 task families.”

“Representing Scientific Documents Prior works have produced large scale language models pre-trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022).”

“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”

“, 2020)) can be tailored to produce high-quality representations of documents with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“First, the name and description are used to generate a vector representation of the Concept [Singh et al., 2023].”

“Content-side alternatives such as SPECTER and SciRepEval (Cohan et al., 2020; Singh et al., 2023) motivate the LLM-rater family.”

“Graph structure has long been used to model scientific knowledge, from Literature-Based Discovery to citation-graph mining, document representation, and impact forecasting [26, 27, 24, 7].”

“Notably, previous state-of-the-art retrieval systems (SPECTER and SPECTER2 [8, 43] achieved 0% accuracy on CiteME, and the authors identify three distinct error categories: misunderstanding the excerpt (50%), stopping at approximately matching papers (32%), and finding the correct citation in…”

“We employ two representative dense re-trievers: SPECTER2 [51] and Contriever-MS [19].”

“For comparison, we also evaluated other embedding models, including Specter2 [15], which is specifically trained on scientific citation data, and a traditional TF-IDF with cosine similarity baseline.”

“Our visualization application is highly adaptable, incorporating node representations via SPECTER2, a scientific document embedding model (Singh et al., 2022), dimensionality reduction techniques such as t-SNE and UMAP (McInnes et al., 2020; van der Maaten & Hinton, 2008), and large-scale…”

“Our work leverages training data from the Search task within SciRepEval 3 (Singh et al., 2023), which contains about 478k queries from real users on Semantic Scholar.”

“In Figure 7, panel (a), we use the embedding model from (Singh et al., 2023), specifically pretrained on scientific papers’ citation relations us-ing contrastive learning.”

“In [11] a similar approach is discussed for scientific paper in English.”

“…suggested by recent literature: (i) citation-aware scientific encoders capture content signals that correlate with long-term influence [1], [2], [3]; (ii) responsible features should be auditable, time-capped, and, where possible, normalized within venue/discipline to reduce field effects [5];…”

“To obtain semantically rich document embeddings, we utilize SPECTER2-base (Singh et al., 2023), a pretrained transformer model designed for scientific documents.”

“…shown in Figure 2, the mean cosine similarity between abstract embeddings and the number of shared references correlate well, confirming the us-SPORENLP ability of cosine similarity as a base metric for the recommendation system, as already indicated by the benchmarks shown in (Singh et al., 2022).”

“We also include four models specialized for scientific domains: SciMult (Zhang et al., 2023), SPECTER 2.0 (Singh et al., 2023), Med-CPT (Jin et al., 2023), and BMR ETRIEVER series (410M/2B/7B) (Xu et al., 2024).”

“Recent work uses LLMs to extract claims [35], methods [36], and contributions [37] from scientific text.”

“Papers were embedded with SPECTER2 [36], developed based on SPECTER [37].”

“SPECTER2 [Singh et al., 2023] uses a SciBERT-based document encoding learned from scientific texts in which training examples were chosen using citation relationships as a strong similarity signal.”

“It serves various downstream applications, including knowledge base and knowledge graph construction (Zhang et al., 2024a; Wei et al., 2024), question answering (Choudhary et al., 2023), and domain-specific document understanding (Singh et al., 2023; Nandi and Agrawal, 2024).”

“More recent studies further extend this direction using transformer-based encoder models [11], [12] (e.g., PubMed-BERT [4], SPECTER2 [13]).”

“…including domain-adapted trans-former encoders such as BioBERT, PubMedBERT, SapBERT, E5-Large, and SPECTER2, have demonstrated strong performance in semantic representation and biomedical retrieval tasks (Lee et al. 2020, Gu et al. 2021, Singh et al. 2023, Khattab et al. 2026, Liu et al. n.d.).”

“They are commonly evaluated on benchmarks such as SciRepEval [19].”

“We use specter2 base [22] as encoder due to its better performance observed with citation networks [5].”

“To assess the robustness of our analysis, we do the same analysis with the SPECTER2 (Singh et al., 2023) embedding model which results in a 768 -dimensional embedding vector, see Appendix Figure 8 and Figure 10.”

“First, at the semantic level, we used 2 natural language models—SPECTER2 and PubMedBERT—to vectorize titles and abstracts, aggregating them at the keyword level [66-68].”

“To contextualize the strength of the observed correlations, we further compare our results against several widely used scientific embedding models: SPECTER (Cohan et al., 2020), SPECTER2 Proximity (Singh et al., 2023), SciNCL (Osten-dorff et al., 2022b), SemCSE (Brinner and Zarriess, 2025), and the invasion biology-sp…”

“Previous work utilizes pairs of articles, formed by either citation links [9, 43] (i.e. one paper cite the other) or co-citations [34] (i.e. cited in the same sentence within another paper).”

“…Semantic Scholar (Any article from the S2 collection for comparison purpose, ACL articles only, and most cited ACL articles) as a baseline, as well as sparse (BM25 (Robertson and Zaragoza, 2009)) and dense (SPECTER2 (Singh et al., 2023)) retrieval models (see Table 5, full details in Table 9).”

“For classifying articles, we employed two fine-tuned language models for classifying publications: a SPECTER2-based model (Singh et al., 2023) for multi-label topic classification based on the used NLP taxonomy (Schopf et al., 2023; Schopf and Matthes, 2024), and a SciNCL-based model (Ostendorff et…”

“…propose a citation-based contrastive pre-training paradigm, and they are evaluated on classification and recommendation tasks; more recent models such as SPECTER 2.0 [44] and SciMult [61] devise multi-task pre-training frameworks, which are evaluated on more diverse tasks such as literature search.”

“5 [16], as it demonstrated higher accuracy and relevance compared to a model specifically trained for scientific publications (SPECTER [14]) in preliminary experiments.”

“Furthermore, for a significant portion of the scores, the rationale was very general and abstract, often merely reciting the evaluation characteristic guidelines provided in the system prompt without offering specific insights.”

“6B , BGE-M3 [5], GTE-large [21], NV-Embed-v2 [20], and Qwen3-Embedding-8B [52]; scientific re-trievers including SPECTER2 [40], SciNCL [32], PubMedBERT-base-embeddings [14], BioClinical-ModernBERT [43], ChemicalBERT-uncased 6 , ChEmbed-prog [18], and MatSciBERT [15]; and reasoning-oriented…”

“Similarity compares rewritten reviews with the originals using SPECTER2 and masked writing-style similarity.”

“Article and Concept texts are encoded separately (bi-encoder) using the SPECTER2 base model 2 , a scientific document encoder pretrained with a citation-aware contrastive objective [Singh et al., 2023].”

“FinMTEB [Tang and Yang, 2024], ChemTEB [Shiraee Kasmaee et al., 2024], MLEB [Butler et al., 2025], and SciRepEval [Singh et al., 2023] each demonstrate that high aggregate MTEB scores do not predict performance on specialised professional text, motivating purpose-built benchmarks.”

“BioMINTREL uses SPECTER2, a scientific text encoder [54], to encode the two retrieval texts separately and calculates the cosine similarity between the corresponding data-side and method-side texts.”

“Following (Hu et al., 2026), we used Spector2 (Singh et al., 2023), a specialized sentence embedding model finetuned on abstracts of scientific papers.”

“Existing benchmarks for scientific document representations, including SciDocs [Cohan et al., 2020] and SciRepEval [Singh et al., 2023], evaluate representations of papers only.”

“Following pretraining on 6M citation triplets, it is also fine-tuned using task-specific adapters trained on the SciRepEval benchmark [18].”

“The benchmark does not provide SPECTER2 vectors.”

“Domain-specific models such as SPECTER2 (Singh et al., 2023) represent a natural extension, however, our corpus spans multiple do - mains (medical imaging, fault diagnosis, graph networks) rather than a single scientific field, making a general-purpose encoder a reasonable default.”

“…& Zaragoza, 2009; Thakur et al., 2021), as well as literature graphs, structured corpora, scientific encoders, and citation-aware representations or recommendation (Ammar et al., 2018; Lo et al., 2020; Beltagy et al., 2019; Cohan et al., 2020; Singh et al., 2023; Bhagavatula et al., 2018).”

“To compare the semantic distance between two papers, we use embedding representations extracted from SPECTER2 [37] for each paper based on their titles and compute the distance between them.”

“Among multiple tested pre-trained transformer models, fine-tuning the SPECTER2 model [37,38] for each category yielded the best performance as measured by the macro-averaged F1 score, both regarding cross-validation as well as test error.”

“SciDocs (Cohan et al., 2020) and SciRepEval (Singh et al., 2023) evaluate retrieval relevance from citation-graph labels, measuring similarity holistically without distinguishing which dimensions overlap.”

“We compare EvoTree with representative flat, hierarchical, citation-aware, and taxonomy-induction baselines, including HAC-SPECTER2 (Murtagh and Contreras, 2012), KMeans-flat (Ahmed et al., 2020), HAC-4level (Murtagh and Contreras, 2012), CitRank+HAC (Woods, 2024), TaxoGen-style (Zhang et al., 2018), Hu-CiteTaxo (Hu e…”

“…1) Topic-ILP represents each paper by a topic distribution inferred with a topic model [5] and compares distributions in that space; 2) Embedding-ILP represents each paper by a dense embedding from SPECTER2 [42], a language model widely adopted for scientific text, and uses cosine similarity.”

“Dense methods encode titles and abstracts as vector embeddings (Reimers and Gurevych, 2019), and scientific-document models use citation structure to learn task-specific representations (Cohan et al., 2020; Singh et al., 2022; Ostendorff et al., 2022).”

“The encoder variant retained the maximum statistic but replaced MPNet with SPECTER2 (Singh et al., 2023).”

“…for review generation (Kang et al., 2018; Dycke et al., 2023; Yuan et al., 2022; Shen et al., 2022; Lou et al., 2025) and document reasoning (Singh et al., 2023; Dasigi et al., 2021; Cohan et al., 2019) interleave stated and unstated weaknesses, letting models inflate scores by…”

“The statistical details of all subsets are presented in For R ext , we select for each p ∈ P , the most semantically similar cited paper using SPECTER2 embeddings (Singh et al., 2023), extracting claims and QA pairs from each p .”

“We measure academic closeness using SPECTER2, a large language model trained to capture semantic similarity among scientific texts 17 , calculating the cosine similarity between each proposal’s title and abstract and a given reviewer’s 20 most-cited papers 18 (Methods).”

“For D5, we compare HexEval-D5 with TF–IDF (Salton and Buckley 1988), SPECTER2 (Singh et al. 2023), and direct zero-shot LLM scoring.”

“…into the geometry of the space; SciNCL [Ostendorff et al., 2022] extends this with contrastive learning in the citation neighbourhood, and SPECTER2 [Singh et al., 2023] adds task-specific adapters and achieved state-of-the-art performance on the SciRepEval benchmark at the time of its publication.”

“The first two are general-domain retrieval encoders trained on MS MARCO (Bajaj et al., 2018), while the SPECTER2 variants (Singh et al., 2023) are scientific paper encoders trained with citation-based supervision, building on the original SPECTER model (Cohan et al., 2020a).”

“Building on SciBERT as its foundation, SPECTER (Cohan et al. , 2020) learns embeddings of titles and abstracts using citation-based triplets, and SPECTER2 (Singh et al. , 2023) extends this approach through multi-format training with adapters.”

“…bids with an affinity score A ij estimating how well reviewer i matches paper j , computed from TF-IDF similarity between the re-viewer’s publication record and the submission [4] or, in current deployments, from document embeddings such as SPECTER [6], [14] wired into the venue’s matching service.”

“Candidate paper groups were formed as dynamic topical neighborhoods using SPECTER2 similarity over titles and document descriptions, combined with keyword overlap.”

“SPECTER2 extended this direction through the SciRepEval benchmark and adapter-based training across multiple scientific tasks, supporting paper similarity and expertise scoring (Singh et al., 2023).”

“The citation-premium analysis (§7) relies on paper abstracts (for the SPECTER2 embeddings) and Semantic Scholar citation counts; both are available for the large majority of papers in all three families (Table 9).”

“Each Dataset node is initialized with a 768-dim SPECTER2 abstract embedding (Singh et al., 2023; Cohan et al., 2020); non-dataset types receive Xavier-initialized learnable embeddings of the same dimension.”

“Scientific text representation models can also encode title, abstract, and structured document signals using representations trained for scientific literature (Cohan et al., 2020; Singh et al., 2023).”

“These include general-purpose dense retrievers such as Contriever (Izacard et al. 2021), GTR-L (Ni et al. 2022), InstructOR-L (Su et al. 2023), SGPT (Muen - nighoff 2022), SPECTER 2.0 (Singh et al. 2023), ColBERT (Wang et al. 2023) and Dragon (Lin et al. 2023).”

“More recent systems increasingly combine metadata, abstracts, full text, and language models to characterize research aims, topics, and disciplinary relations (Wu et al., 2025; Beltagy et al., 2019; Lo et al., 2020; Singh et al., 2023; Zhang et al., 2024).”

“We also include SPECTER (Cohan et al., 2020), SciNCL (Osten-dorff et al., 2022), and SPECTER2 with the PRX adapter (Singh et al., 2023), all of which are scientific document encoders trained on citation links between papers.”

“The representation baselines encompass GloVe, Word2Vec, SSCI-SciBERT-e4 (a modified version of SciBERT for domain-specific applications) (Shen et al., 2023), and SPECTER2 (an advanced adaptation of SciBERT tailored for representing scientific documents) (Singh et al., 2022).”