SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“These tend to perform better than general purpose models on scientific domain tasks, and serve as a foundation for learning dense embeddings of scientific documents.”

“Tay et al. (2022) use control codes to switch be-tween three de-noising modes during pre-training, and associate every downstream task with a mode during fine-tuning.”

“Aribandi et al. (2022) pre-train T5 (Raffel et al., 2020) on C4 span denoising and 107 other tasks across 8 task families.”

“Representing Scientific Documents Prior works have produced large scale language models pre-trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022).”

“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”

“, 2020)) can be tailored to produce high-quality representations of documents with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“Recent approaches further modify the Transformer architecture in LLMs with Adapters (Singh et al., 2023), GNN-nested Transformers (Jin et al., 2023b), and Mixture-of-Experts Transformers (Zhang et al., 2023f) to better capture graph signals.”

“To identify AI papers in various fields across eras, we fine-tune BERT 32,33 , an established language model 34–36 , on articles published in explicitly AI-oriented scientific journals and conferences to automatically extract and interpret information from context.”

“Several works have designed specialized retrieval models for biomedical domains (Mohan et al., 2017; Liu et al., 2021; Jin et al., 2023; Luo et al., 2022a; Singh et al., 2023; Zhang et al., 2023).”

“The code, models, and data for SPECTER are available at https://github.com/allenai/SPECTER and for SciRep-Eval at https://github.com/allenai/SciRepEval.”

“Work in the last 10 years has sought to make the space of literature more manageable for scientists, with the introduction of keyword search systems [6, 7], vector similarity embeddings [8, 9, 10, 11] and recommender systems [12, 13].”

“In our study using the LLMs4OM framework, we conduct extensive evaluations, beginning with the RAG module where we explore four retrieval methods: TFIDF [38], sentence-BERT [37], SPECTER2 [42], and Ope-nAI text-embedding-ada [33].”

“For pre-trained language models, we selected BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), SciBERT (Beltagy et al., 2019), SPECTER (Cohan et al., 2020), SPECTER2, THExt-cs-sciBERT (La Qua-tra and Cagliero, 2022), and CS_RoBERTa 3 (Singh et al., 2022).”

“In conclusion, we are optimistic that the S ci RIFF data and evaluations S ci RIFF-E val , as well as the model checkpoints, will serve as valuable resources to build systems for scientific researchers.”

“For example, SPECTER [15, 51] leveraged citations between documents to train dense vector representations to encode the content-similarity of research papers.”

“However, recent studies [8, 44, 50] indicate that LLMs face challenges when specializing in scientific literature understanding, particularly in context understanding and question answering.”

“To develop our OOD test set, we first embed all 100 research topics by running SPECTER2 , a scientific paper embedding model using citation graph (Singh et al., 2022), on the title and abstract of the Cochrane systematic review associated with each topic.”

“SciRepEval (Singh et al., 2023), a multi-format benchmark for scientific document representations, includes 24 realistic tasks across four formats: classification, regression, ranking, and search.”

“Selection of few-shot examples: Prior work has shown that in-context learning can benefit from sophisticated strategies for selecting exemplars, e.g. based on diversity (Hongjin et al., 2022) or informativeness (Wu et al., 2023) (Singh et al., 2022).”

“However, this model is unsuitable for comparing patent titles and abstracts because of its focus is on the patent claims.”

“These two ideas are different from the techniques ( e.g., adapters (Houlsby et al., 2019) and control codes (Keskar et al., 2019)) explored in previous studies (Singh et al., 2022) for multi-task scientific text representation learning.”

“The second important choice is the size of reviewers’ profiles.”

“We initialize the paper encoder with a pre-trained scientific domain language model (e.g., SPECTER-v2 (Singh et al., 2022)).”

“Recently, instruction tuning that distils various skills and knowledge from existing strong LLMs (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022; Wang et al., 2022c; Singh et al., 2022), has received significant research interests.”

“…preserving the semantics (Niu et al., 2021), question generation that aims to generate a fluid and relevant question given a passage and a target answer (Song et al., 2018), or dialogue-response generation which aims to generate natural-looking text relevant to the prompt (Zhang et al., 2020).”

“…narrowly-defined tasks, such as evaluating performance on short-form question answering from document context [11, 75, 18, 35, 72, 36], generating summaries from multiple documents [45, 39, 33], as well as tasks like information extraction [41], hypothesis generation [40], and retrieval [16, 70].”

“We further adapt SPECTER2 to our hierarchical multi-label classification task, fine-tuning it for each update of our evolving taxonomy.”

“SPECTER-v2 is trained via multi-task learning us-ing paper metadata from SCIBERT (Beltagy et al., 2019), and Contriever-MS is fine-tuned via massive training queries (MS MARCO) from BERT base uncased (Devlin et al., 2018).”

“…state-of-the-art models as well as several that have been extensively benchmarked on other tasks: E5-Large-v2 (Wang et al., 2022), SimCSE-Large (Gao et al., 2021), SPECTER-v2 (Singh et al., 2022), ROBERTA-Large (Liu et al., 2019), SPLADE-v2 (Formal et al., 2021), and SPLADE++ (Formal et al., 2022).”

“The first benchmark is the SciRepEval (Singh et al. 2022) which provides 25 challenging tasks across four formats: classification, regression, ranking, and search.”

“• SciNCL [34] is also a scientific PLM initialized from SciBERT and trained on citation links.”

“Embedding-based Ranking We generate embeddings for all collected papers using SPECTER v2 (Singh et al., 2022) on concatenated titles and abstracts.”

“…(Li et al., 2024b; Wang et al., 2024; Nigam et al., 2024; Pu et al., 2024; Kumar et al., 2024; Qi et al., 2023), either employ sentence-based semantic similarity (Reimers and Gurevych, 2019; Beltagy et al., 2019) or embed generic inter-document relationship (Cohan et al., 2020; Singh et al., 2022).”

“Given the current state of rapid adoption of MLIPs 29 – 31 , developing the processes of benchmarking and evaluating MLIP models is increasingly important.”

“We encode all documents using SPECTER2 (Co-han et al., 2020; Singh et al., 2023), a transformer pre-trained on citation graphs (768-dim CLS embeddings).”

“[SEP] The MDAS had good internal consistency, with Cronbach alpha of .89 and Guttman split-half coefficient of 0.71,.. [14] task.”

“We obtain qualitatively similar results with a different embeddings model (SPECTER2 [84]) and by only using the title of the focal paper (Appendix Figure A8).”

“MTEB (Muennighoff et al., 2023); C-MTEB (Xiao et al., 2024c); MMTEB (Enevoldsen et al., 2025); ArabicMTEB (Bhatia et al., 2024); PL-MTEB (Po´swiata et al., 2024); FaMTEB (Zinvandi et al., 2025); SciRepEval (Singh et al., 2023); FinMTEB (Tang & Yang, 2025); ChemTEB (Kasmaee et al., 2024); MIEB (Xiao et al., 2025); ATEB…”

“…own work relates to other work [31] by contextualizing embedded citations [6], drawing connections to follow-on work directly in the paper document [41], and making recommendations based on a paper’s metadata such as citations and authorship [8, 24, 40, 43] or its semantic similarity [4, 18, 24].”

“In this work, we focus on the widely used SPECTER algorithm [6].”

“These capabilities enhance the depth and breadth of literature reviews [44], improve the precision of research impact analysis [8], and support the organization of scientific knowledge [45].”

“In contrast, CCA outperforms SPECTER2 by leveraging network structure and research papers’ content.”

“We also include the current SotA methods SciBERT (Beltagy et al., 2019) and SciNCL (Ostendorff et al., 2022) as additional SciRepEval baselines.”

“FoRC Shared Task I represents a task of scholarly document classification, a task where BERT [3], an encoder model for text embeddings, has demonstrated excellent results in a variety of benchmarks [12].”

“Researchers have also proposed learning representations for retrieval of relevant scientific articles (Singh et al. 2022; Cohan et al. 2020; Ostendorff et al. 2022; Mysore, Cohan, and Hope 2021), literature Review Generation (Hu and Wan 2014; Kasanishi et al. 2023; Chen et al. 2021), Question…”

“…for scientific question answering that was created by using knowledge graphs of academic articles and with the help of human-made templates, and SciRepEval(Singh et al., 2022), a collection of several scientific document tasks across four types: classification, regression, proximity, and searching.”

“Similarly, the ZB-MED-DSS team experiment with the following BERT-based models: SciBERT, SciNCL, and SPECTER2 [37].”

“As suggested in SciREpEval models [23], we utilized both regression and classification formats for the IFT of the HOC task, which is a document classification task.”

“Given a scientific text ω or c , we compute embeddings using the SPECTER2 model 6 with the regression adapter (Singh et al., 2023).”

“Most of the existing tools focus on notifying and recommending researchers with relevant liter-Figure (Singh et al., 2022; Cohan et al., 2020; Ostendorff et al., 2022; Mysore et al., 2021), literature Review Generation (Hu and Wan, 2014; Kasanishi et al., 2023; Chen et al., 2021), Question…”

“We embed only the first 512 tokens of each sampled document using the specter2-base model provided by Singh et al. [65].”

“We chose the SPECTER2 embedding (Singh et al., 2023), a multi-format embedding developed specifically to represent scientific documents.”

“…texts in the form of SciBERT (Beltagy et al., 2019), SciDeBERTa (Kim et al., 2023), SPECTER (Cohan et al., 2020a), SciNCL (Ostendorff et al., 2022), and SPECTER2 (Singh et al., 2023), as well as several state-of-the-art general-purpose embedding models commonly used for document retrieval.”

“For matching on topic embeddings, we use SPECTER2 (Singh et al., 2022).”

“We demonstrate the potential of the affine model transfer through two case studies: (i) the prediction of feed-forward torque at seven joints of the robot arm [23], and (ii) the prediction of review scores and decisions of scientific papers [24].”

“More recently, Singh et al. (2022) further extended the contrastive learning approach of SPECTER to better deal with multiple tasks of relevance to scientific papers, including citation prediction, leading to SPECTER2.”

“3”

“For bioRxiv, we choose SPECTER2 (Singh et al., 2022) 15 which is finetuned on abstracts of scientific papers.”

“SPECTER (Singh et al., 2022) improves relatedness scoring via citation-informed embeddings, but does not extract or leverage specific content from papers.”

“While scientific benchmarks such as MMLU (Hendrycks et al., 2020), SciBench (Wang et al., 2023), SciQ (Welbl et al., 2017), SciRepE-val (Singh et al., 2022), SciQA (Auer et al., 2023), and QASA (Lee et al., 2023) are used for multi-disciplinary scientific QA evaluations, field-specific benchmarks…”

“0 (Singh et al., 2022) PRX model trained on scientific titles and abstracts.”

“SciBERT (Beltagy et al., 2019) is a variant of BERT fully trained on 1 et al., 2020) and (Singh et al., 2022).”

“We used the SPECTER2 model [11] to generate yearly embeddings for papers to analyze the emergence and diffusion of CSS over time.”

“Recent works on encoding scientific texts include models such as SPECTER [3], SciNCL [16] and SPECTER2 [19].”

“Specter2 uses the same architecture, but is trained on 6M triples [67].”

“This includes short one-sentence summaries of publications (TLDRs), SPECTER2 embeddings of publications, author information, as well as citations and references.”

“As an alternative, we ground our experiments on the SPECTER language model [28, 85] that we use as a zero-shot ranker without task-specific fine-tuning.”

“It could denote that it is not always required to utilize full-text information for the tasks present in S CI R EP E VAL benchmark.”

“We present the results of SPECTER [21] and SPECTER2 [77] on CiteME as our baselines.”

“The second category includes SciBERT [31], METAG [6], PATTON, SciPATTON [5], SPECTER [3, 32], SciNCL [4], and SciMult [33].”

“We embed these candidates using SPECTER2 (Singh et al., 2022) and compute their co-sine similarity to i ∗ .”

“For the social sciences dataset, we used SPECTER2 (Singh et al., 2023), a transformer-based model pretrained on a broad scientific corpus and further optimized with citation-informed objectives to project abstracts into a 768-dimensional embedding space.”

“Each encoder ini-tializes from SPECTER2 (Singh et al., 2023) to inherit general scientific knowledge, then fine-tunes to specialize in its designated dimension.”

“SciRepEval [16] and PaperQA [17] assess paper classification and question answering.”

“Representation learning aims to project scientific documents into a low-dimensional vector space, where the semantic or structural relationships between documents can be effectively captured and computed (Singh et al., 2023).”

“1 54 (70B), Mixtral 55 (7Bx8E), Gemma 4 (31B); and Llama 4 Scout (17Bx16E), evaluated in few-shot settings (3, 15, or 30 examples) ● A fine-tuned small Open LLM: LLaMA-3.”

“In the context of scientometrics, existing relationships which have been used as ground truths include ‘paper-cites-paper’ (Ali et al., 2022), ‘author-cites-paper’ (Jiang et al., 2023), and ‘paper-co-read-with-paper’ (Singh, D’Arcy, Cohan, Downey, & Feldman, 2023) information.”

“Although these specialized models on scientific texts demonstrate success in the scientific literature understanding benchmarks such as MAPLE (Zhang et al., 2023d) and SciRepEval (Singh et al., 2023), the benchmarks and the corpora used in the training of these scientific language models predominantly focus on textual…”

“In contrast to the existing saturated scientific benchmarks like SciDocs [Cohan et al., 2020] and SciRepEval [Singh et al., 2022], LEGOB-ENCH can be used to evaluate the full-text encoding capabilities of existing (SciBERT [Beltagy et al., 2019b], SPECTER [Cohan et al., 2020], SciNCL [Ostendorff et…”

“Of this portion, 50% is drawn from FoS categories Medicine, Pharma (Singh et al., 2023), and 50% consists of English instances from the PharmaChem domain, sampled randomly to ensure representation across all customers.”

“Embedding methods make it possible to represent a scientific document as a point in a high-dimensional space and compare that position with earlier documents (Cohan et al. 2020; Shibayama et al. 2021; Singh et al. 2023).”

“Human verification indicates that GPT-5 consistently preserves the JSON format, and analyzing the review and response jointly produces better results than separate analysis, likely due to the additional cross-segment context enabling more reliable linking and reasoning.”

“Following prior work [4, 13, 15], using the title and abstract separated by the [ SEP ] token as input effectively balances input length constraints with high-quality representation.”

“These include general-purpose dense retrievers such as Contriever [19], GTR-L [31], InstructOR-L [41], SGPT [29], SPECTER 2.0 [38], and Dragon [26].”

“…2024), and have therefore been applied to a variety of tasks within the scientific domain, including literature search and similarity assessment (Singh et al., 2023), classification (Ros-tam and Kert´esz, 2024), and summarization (Sefid and Giles, 2022), with similar pretrained models having…”

“A series of works on paper representation learning aims to obtain numerical representations of papers that can be used for information retrieval or recommendation (Ostendorff et al., 2022; Singh et al., 2023).”