SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“These tend to perform better than general purpose models on scientific domain tasks, and serve as a foundation for learning dense embeddings of scientific documents.”

“Tay et al. (2022) use control codes to switch be-tween three de-noising modes during pre-training, and associate every downstream task with a mode during fine-tuning.”

“Aribandi et al. (2022) pre-train T5 (Raffel et al., 2020) on C4 span denoising and 107 other tasks across 8 task families.”

“Representing Scientific Documents Prior works have produced large scale language models pre-trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022).”

“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”

“, 2020)) can be tailored to produce high-quality representations of documents with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“, 2018), bi-encoder networks (Conneau et al., 2017) and BERT-based methods (Reimers & Gurevych, 2019).”

“We also introduce three new large-scale training datasets aimed at predicting same-authors, citations (via triplets) as in Cohan et al. (2020), and influential citations, which we define as four or more citations of the same paper in the text of a single paper.”

“We also introduce three new large-scale training datasets aimed at predicting same-authors, citations (via triplets) as in Cohan et al. (2020), and influential citations, which we define as four or more citations of the same paper in the text of a single paper.”

“Our benchmark includes diverse tasks with a variety of different evaluation metrics, and following previous work (e.g. Wang et al. (2019)) we report an average of the individual metrics (each ranging from 0-100).”

“NFCorpus This is a set of non technical English queries and associated results compiled from Nutri-tionFacts.org (Boteva et al., 2016 (Wang et al., 2020b) along with their relevance scores on a scale of 0-2.”

“Further, we also tried loss weighting strategies (Chen et al., 2018; Liu et al., 2019a) but our preliminary experiments produced better re-sults without any scaling so we didn’t explore it further.”

“We explore two state-of-the-art methods to generate format-specific representations: control codes (Keskar et al., 2019; Raffel et al., 2020) as input signal; and parameter-efficient adapter methods (Pfeiffer et al., 2021; Stickland and Murray, 2019), where a separate network module is introduced…”

“Multi-Task Learning Across Formats Multi-task learning (Caruana, 1993) with deep neural networks has shown improvements upon single-task training for related objectives (Liu et al., 2015, 2019b).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“NFCorpus This is a set of non technical English queries and associated results compiled from Nutri-tionFacts.org (Boteva et al., 2016 (Wang et al., 2020b) along with their relevance scores on a scale of 0-2.”

“While there has been significant progress in evaluating generalizability of NLP models (Ye et al., 2021; Sanh et al., 2021), evaluation of scientific document representations has remained limited.”

“Cohan et al. (2020) and Ostendorff et al. (2022b) fine-tune SciBERT (Beltagy et al., 2019) with a triplet loss that encourages papers citing each other to have similar embeddings, using the title and abstract of research papers as the input.”

“These methods are pre-trained on a set of over 1B query-candidate pairs including scientific text, and are two of the best BERT-base sized models on the recent MTEB benchmark (Muennighoff et al., 2022) for general-purpose text embeddings.”

“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”

“Multi-Task Learning Across Formats Multi-task learning (Caruana, 1993) with deep neural networks has shown improvements upon single-task training for related objectives (Liu et al., 2015, 2019b).”

“• Liu et al. (2014), an extension of Mimno and McCallum (2007) which adds 766 additional paper-review annotations.”

“We include TREC-CoVID (Voorhees et al., 2021) and NFCorpus (Boteva et al., 2016) as test tasks for this format.”

“The models variants can be further sub-categorized as retrieval based on best aspect matching (TS ASPIRE) and a weighted sum of the similarity score among all the aspects based on Optimal Transport (OT ASPIRE) between the query and candidates.”

“We train our models in a multi-task setup with task-heterogeneous batching (Aghajanyan et al., 2021).”

“Cohan et al. (2020) and Ostendorff et al. (2022b) fine-tune SciBERT (Beltagy et al., 2019) with a triplet loss that encourages papers citing each other to have similar embeddings, using the title and abstract of research papers as the input.”

“…between query and candidate papers which is a similar setting to our proximity tasks, so we only evaluate and report its results for that subset in Appendix C. Finally, we also evaluate two recent general-purpose text embedding methods, E5 v2 (Wang et al., 2022) and MPNet (Song et al., 2020).”

“When training on multiple tasks simultaneously, it is important to choose a combination of tasks that does not display negative transfer (Aribandi et al., 2022; Padmakumar et al., 2022; Fifty et al., 2021).”

“Pytrec_eval (Van Gysel and de Rijke, 2018) is used to calculate the ranking metrics.”

“Representing Scientific Documents Prior works have produced large scale language models pre-trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022).”

“In contrast, SciRepEval provides more challenging and diverse tasks to help motivate methods for producing scientific document representations that can generalize well across tasks.”

“…the S2AND author disambiguation task (Subrama-nian et al., 2021) with paper similarity features; Paper-Reviewer Matching (Mimno and McCallum, 2007; Liu et al., 2014; Zhao et al., 2022), where candidate reviewers are ranked by expert annotators based on the similarity of their papers to the query…”

“We include a total of ten proximity-based tasks, including four nearest neighbor tasks from SciDocs (predicting citations, co-citations, co-viewed or co-read papers), and three others from previous work: the S2AND author disambiguation task (Subrama-nian et al., 2021) with paper similarity features; Paper-Reviewer Mat…”

“We include a total of eleven proximitybased tasks, including four evaluation tasks from SciDocs (predicting citations and co-citations, and predicting co-viewed or co-read papers), and two others from previous work: the S2AND author disambiguation task (Subramanian et al., 2021) with paper similarity features, and Pap…”

“Existing benchmarks either focus on document similarity (Mysore et al., 2021; Voorhees et al., 2021) or tasks that are highly correlated and not diverse (Cohan et al., 2020).”

“However, SPECTER2 Base and MTL CLS are on par even though the latter is trained on in-domain datasets.”

“Supporting the findings in Padmakumar et al. (2022), pre-fine-tuning on the tasks from each individual format’s training data gives better or similar performance on downstream tasks when compared to training on all the formats at once, except on out-of-train classification and search.”

“…the two SciDocs tasks ( MAG and MeSH Diseases ), we have four others; including a binary task to predict whether a paper is relevant to biomimicry (Shyam et al., 2019), two biomedical classification tasks, namely DRSM from Burns (2022) and MeSH Descriptors classification (Lipscomb, 2000), and a…”

“Representing Scientific Documents Earlier work aimed at document embeddings used word vectors (J et al., 2016; Le & Mikolov, 2014; Wu et al., 2018), convolutions (Liu et al.”

“Aghajanyan et al. (2021) pre-fine-tune language models on 46 tasks from 4 task types before fine-tuning on the downstream task.”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“For evaluation, we consider predicting three numeric attributes related to prominence or quality: Tweet Mentions (Jain and Singh, 2021), and two new datasets predicting peer review rating and maximum h-index of authors for a set of ICLR papers from OpenReview 1 .”

“Recent approaches further modify the Transformer architecture in LLMs with Adapters (Singh et al., 2023), GNN-nested Transformers (Jin et al., 2023b), and Mixture-of-Experts Transformers (Zhang et al., 2023f) to better capture graph signals.”

“To identify AI papers in various fields across eras, we fine-tune BERT 32,33 , an established language model 34–36 , on articles published in explicitly AI-oriented scientific journals and conferences to automatically extract and interpret information from context.”

“Several works have designed specialized retrieval models for biomedical domains (Mohan et al., 2017; Liu et al., 2021; Jin et al., 2023; Luo et al., 2022a; Singh et al., 2023; Zhang et al., 2023).”

“The code, models, and data for SPECTER are available at https://github.com/allenai/SPECTER and for SciRep-Eval at https://github.com/allenai/SciRepEval.”

“Work in the last 10 years has sought to make the space of literature more manageable for scientists, with the introduction of keyword search systems [6, 7], vector similarity embeddings [8, 9, 10, 11] and recommender systems [12, 13].”

“In our study using the LLMs4OM framework, we conduct extensive evaluations, beginning with the RAG module where we explore four retrieval methods: TFIDF [38], sentence-BERT [37], SPECTER2 [42], and Ope-nAI text-embedding-ada [33].”

“For pre-trained language models, we selected BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), SciBERT (Beltagy et al., 2019), SPECTER (Cohan et al., 2020), SPECTER2, THExt-cs-sciBERT (La Qua-tra and Cagliero, 2022), and CS_RoBERTa 3 (Singh et al., 2022).”

“In conclusion, we are optimistic that the S ci RIFF data and evaluations S ci RIFF-E val , as well as the model checkpoints, will serve as valuable resources to build systems for scientific researchers.”

“For example, SPECTER [15, 51] leveraged citations between documents to train dense vector representations to encode the content-similarity of research papers.”

“However, recent studies [8, 44, 50] indicate that LLMs face challenges when specializing in scientific literature understanding, particularly in context understanding and question answering.”

“To develop our OOD test set, we first embed all 100 research topics by running SPECTER2 , a scientific paper embedding model using citation graph (Singh et al., 2022), on the title and abstract of the Cochrane systematic review associated with each topic.”

“SciRepEval (Singh et al., 2023), a multi-format benchmark for scientific document representations, includes 24 realistic tasks across four formats: classification, regression, ranking, and search.”

“Selection of few-shot examples: Prior work has shown that in-context learning can benefit from sophisticated strategies for selecting exemplars, e.g. based on diversity (Hongjin et al., 2022) or informativeness (Wu et al., 2023) (Singh et al., 2022).”

“However, this model is unsuitable for comparing patent titles and abstracts because of its focus is on the patent claims.”

“These two ideas are different from the techniques ( e.g., adapters (Houlsby et al., 2019) and control codes (Keskar et al., 2019)) explored in previous studies (Singh et al., 2022) for multi-task scientific text representation learning.”

“The second important choice is the size of reviewers’ profiles.”

“We initialize the paper encoder with a pre-trained scientific domain language model (e.g., SPECTER-v2 (Singh et al., 2022)).”

“Recently, instruction tuning that distils various skills and knowledge from existing strong LLMs (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022; Wang et al., 2022c; Singh et al., 2022), has received significant research interests.”

“…preserving the semantics (Niu et al., 2021), question generation that aims to generate a fluid and relevant question given a passage and a target answer (Song et al., 2018), or dialogue-response generation which aims to generate natural-looking text relevant to the prompt (Zhang et al., 2020).”

“…narrowly-defined tasks, such as evaluating performance on short-form question answering from document context [11, 75, 18, 35, 72, 36], generating summaries from multiple documents [45, 39, 33], as well as tasks like information extraction [41], hypothesis generation [40], and retrieval [16, 70].”

“We further adapt SPECTER2 to our hierarchical multi-label classification task, fine-tuning it for each update of our evolving taxonomy.”

“SPECTER-v2 is trained via multi-task learning us-ing paper metadata from SCIBERT (Beltagy et al., 2019), and Contriever-MS is fine-tuned via massive training queries (MS MARCO) from BERT base uncased (Devlin et al., 2018).”

“…state-of-the-art models as well as several that have been extensively benchmarked on other tasks: E5-Large-v2 (Wang et al., 2022), SimCSE-Large (Gao et al., 2021), SPECTER-v2 (Singh et al., 2022), ROBERTA-Large (Liu et al., 2019), SPLADE-v2 (Formal et al., 2021), and SPLADE++ (Formal et al., 2022).”

“The first benchmark is the SciRepEval (Singh et al. 2022) which provides 25 challenging tasks across four formats: classification, regression, ranking, and search.”

“• SciNCL [34] is also a scientific PLM initialized from SciBERT and trained on citation links.”

“Embedding-based Ranking We generate embeddings for all collected papers using SPECTER v2 (Singh et al., 2022) on concatenated titles and abstracts.”

“…(Li et al., 2024b; Wang et al., 2024; Nigam et al., 2024; Pu et al., 2024; Kumar et al., 2024; Qi et al., 2023), either employ sentence-based semantic similarity (Reimers and Gurevych, 2019; Beltagy et al., 2019) or embed generic inter-document relationship (Cohan et al., 2020; Singh et al., 2022).”

“Given the current state of rapid adoption of MLIPs 29 – 31 , developing the processes of benchmarking and evaluating MLIP models is increasingly important.”

“We encode all documents using SPECTER2 (Co-han et al., 2020; Singh et al., 2023), a transformer pre-trained on citation graphs (768-dim CLS embeddings).”

“[SEP] The MDAS had good internal consistency, with Cronbach alpha of .89 and Guttman split-half coefficient of 0.71,.. [14] task.”

“We obtain qualitatively similar results with a different embeddings model (SPECTER2 [84]) and by only using the title of the focal paper (Appendix Figure A8).”

“MTEB (Muennighoff et al., 2023); C-MTEB (Xiao et al., 2024c); MMTEB (Enevoldsen et al., 2025); ArabicMTEB (Bhatia et al., 2024); PL-MTEB (Po´swiata et al., 2024); FaMTEB (Zinvandi et al., 2025); SciRepEval (Singh et al., 2023); FinMTEB (Tang & Yang, 2025); ChemTEB (Kasmaee et al., 2024); MIEB (Xiao et al., 2025); ATEB…”

“…own work relates to other work [31] by contextualizing embedded citations [6], drawing connections to follow-on work directly in the paper document [41], and making recommendations based on a paper’s metadata such as citations and authorship [8, 24, 40, 43] or its semantic similarity [4, 18, 24].”

“In this work, we focus on the widely used SPECTER algorithm [6].”

“These capabilities enhance the depth and breadth of literature reviews [44], improve the precision of research impact analysis [8], and support the organization of scientific knowledge [45].”

“In contrast, CCA outperforms SPECTER2 by leveraging network structure and research papers’ content.”

“We also include the current SotA methods SciBERT (Beltagy et al., 2019) and SciNCL (Ostendorff et al., 2022) as additional SciRepEval baselines.”

“FoRC Shared Task I represents a task of scholarly document classification, a task where BERT [3], an encoder model for text embeddings, has demonstrated excellent results in a variety of benchmarks [12].”

“Researchers have also proposed learning representations for retrieval of relevant scientific articles (Singh et al. 2022; Cohan et al. 2020; Ostendorff et al. 2022; Mysore, Cohan, and Hope 2021), literature Review Generation (Hu and Wan 2014; Kasanishi et al. 2023; Chen et al. 2021), Question…”

“…for scientific question answering that was created by using knowledge graphs of academic articles and with the help of human-made templates, and SciRepEval(Singh et al., 2022), a collection of several scientific document tasks across four types: classification, regression, proximity, and searching.”

“Similarly, the ZB-MED-DSS team experiment with the following BERT-based models: SciBERT, SciNCL, and SPECTER2 [37].”

“As suggested in SciREpEval models [23], we utilized both regression and classification formats for the IFT of the HOC task, which is a document classification task.”

“Given a scientific text ω or c , we compute embeddings using the SPECTER2 model 6 with the regression adapter (Singh et al., 2023).”

“Most of the existing tools focus on notifying and recommending researchers with relevant liter-Figure (Singh et al., 2022; Cohan et al., 2020; Ostendorff et al., 2022; Mysore et al., 2021), literature Review Generation (Hu and Wan, 2014; Kasanishi et al., 2023; Chen et al., 2021), Question…”

“We embed only the first 512 tokens of each sampled document using the specter2-base model provided by Singh et al. [65].”

“We chose the SPECTER2 embedding (Singh et al., 2023), a multi-format embedding developed specifically to represent scientific documents.”

“…texts in the form of SciBERT (Beltagy et al., 2019), SciDeBERTa (Kim et al., 2023), SPECTER (Cohan et al., 2020a), SciNCL (Ostendorff et al., 2022), and SPECTER2 (Singh et al., 2023), as well as several state-of-the-art general-purpose embedding models commonly used for document retrieval.”

“For matching on topic embeddings, we use SPECTER2 (Singh et al., 2022).”

“We demonstrate the potential of the affine model transfer through two case studies: (i) the prediction of feed-forward torque at seven joints of the robot arm [23], and (ii) the prediction of review scores and decisions of scientific papers [24].”

“More recently, Singh et al. (2022) further extended the contrastive learning approach of SPECTER to better deal with multiple tasks of relevance to scientific papers, including citation prediction, leading to SPECTER2.”

“3”

“For bioRxiv, we choose SPECTER2 (Singh et al., 2022) 15 which is finetuned on abstracts of scientific papers.”

“SPECTER (Singh et al., 2022) improves relatedness scoring via citation-informed embeddings, but does not extract or leverage specific content from papers.”

“While scientific benchmarks such as MMLU (Hendrycks et al., 2020), SciBench (Wang et al., 2023), SciQ (Welbl et al., 2017), SciRepE-val (Singh et al., 2022), SciQA (Auer et al., 2023), and QASA (Lee et al., 2023) are used for multi-disciplinary scientific QA evaluations, field-specific benchmarks…”

“0 (Singh et al., 2022) PRX model trained on scientific titles and abstracts.”

“SciBERT (Beltagy et al., 2019) is a variant of BERT fully trained on 1 et al., 2020) and (Singh et al., 2022).”

“We used the SPECTER2 model [11] to generate yearly embeddings for papers to analyze the emergence and diffusion of CSS over time.”

“Recent works on encoding scientific texts include models such as SPECTER [3], SciNCL [16] and SPECTER2 [19].”

“Specter2 uses the same architecture, but is trained on 6M triples [67].”

“This includes short one-sentence summaries of publications (TLDRs), SPECTER2 embeddings of publications, author information, as well as citations and references.”

“As an alternative, we ground our experiments on the SPECTER language model [28, 85] that we use as a zero-shot ranker without task-specific fine-tuning.”

“It could denote that it is not always required to utilize full-text information for the tasks present in S CI R EP E VAL benchmark.”

“We present the results of SPECTER [21] and SPECTER2 [77] on CiteME as our baselines.”

“The second category includes SciBERT [31], METAG [6], PATTON, SciPATTON [5], SPECTER [3, 32], SciNCL [4], and SciMult [33].”

“We embed these candidates using SPECTER2 (Singh et al., 2022) and compute their co-sine similarity to i ∗ .”

“For the social sciences dataset, we used SPECTER2 (Singh et al., 2023), a transformer-based model pretrained on a broad scientific corpus and further optimized with citation-informed objectives to project abstracts into a 768-dimensional embedding space.”

“Each encoder ini-tializes from SPECTER2 (Singh et al., 2023) to inherit general scientific knowledge, then fine-tunes to specialize in its designated dimension.”

“SciRepEval [16] and PaperQA [17] assess paper classification and question answering.”

“Representation learning aims to project scientific documents into a low-dimensional vector space, where the semantic or structural relationships between documents can be effectively captured and computed (Singh et al., 2023).”

“1 54 (70B), Mixtral 55 (7Bx8E), Gemma 4 (31B); and Llama 4 Scout (17Bx16E), evaluated in few-shot settings (3, 15, or 30 examples) ● A fine-tuned small Open LLM: LLaMA-3.”

“In the context of scientometrics, existing relationships which have been used as ground truths include ‘paper-cites-paper’ (Ali et al., 2022), ‘author-cites-paper’ (Jiang et al., 2023), and ‘paper-co-read-with-paper’ (Singh, D’Arcy, Cohan, Downey, & Feldman, 2023) information.”

“Although these specialized models on scientific texts demonstrate success in the scientific literature understanding benchmarks such as MAPLE (Zhang et al., 2023d) and SciRepEval (Singh et al., 2023), the benchmarks and the corpora used in the training of these scientific language models predominantly focus on textual…”

“In contrast to the existing saturated scientific benchmarks like SciDocs [Cohan et al., 2020] and SciRepEval [Singh et al., 2022], LEGOB-ENCH can be used to evaluate the full-text encoding capabilities of existing (SciBERT [Beltagy et al., 2019b], SPECTER [Cohan et al., 2020], SciNCL [Ostendorff et…”

“Of this portion, 50% is drawn from FoS categories Medicine, Pharma (Singh et al., 2023), and 50% consists of English instances from the PharmaChem domain, sampled randomly to ensure representation across all customers.”

“Embedding methods make it possible to represent a scientific document as a point in a high-dimensional space and compare that position with earlier documents (Cohan et al. 2020; Shibayama et al. 2021; Singh et al. 2023).”

“Human verification indicates that GPT-5 consistently preserves the JSON format, and analyzing the review and response jointly produces better results than separate analysis, likely due to the additional cross-segment context enabling more reliable linking and reasoning.”

“Following prior work [4, 13, 15], using the title and abstract separated by the [ SEP ] token as input effectively balances input length constraints with high-quality representation.”

“These include general-purpose dense retrievers such as Contriever [19], GTR-L [31], InstructOR-L [41], SGPT [29], SPECTER 2.0 [38], and Dragon [26].”

“…2024), and have therefore been applied to a variety of tasks within the scientific domain, including literature search and similarity assessment (Singh et al., 2023), classification (Ros-tam and Kert´esz, 2024), and summarization (Sefid and Giles, 2022), with similar pretrained models having…”

“A series of works on paper representation learning aims to obtain numerical representations of papers that can be used for information retrieval or recommendation (Ostendorff et al., 2022; Singh et al., 2023).”