SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“These tend to perform better than general purpose models on scientific domain tasks, and serve as a foundation for learning dense embeddings of scientific documents.”

“Tay et al. (2022) use control codes to switch be-tween three de-noising modes during pre-training, and associate every downstream task with a mode during fine-tuning.”

“Aribandi et al. (2022) pre-train T5 (Raffel et al., 2020) on C4 span denoising and 107 other tasks across 8 task families.”

“Representing Scientific Documents Prior works have produced large scale language models pre-trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022).”

“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”

“, 2020)) can be tailored to produce high-quality representations of documents with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani et al., 2017).”

“, 2018), bi-encoder networks (Conneau et al., 2017) and BERT-based methods (Reimers & Gurevych, 2019).”

“We also introduce three new large-scale training datasets aimed at predicting same-authors, citations (via triplets) as in Cohan et al. (2020), and influential citations, which we define as four or more citations of the same paper in the text of a single paper.”

“We also introduce three new large-scale training datasets aimed at predicting same-authors, citations (via triplets) as in Cohan et al. (2020), and influential citations, which we define as four or more citations of the same paper in the text of a single paper.”

“Our benchmark includes diverse tasks with a variety of different evaluation metrics, and following previous work (e.g. Wang et al. (2019)) we report an average of the individual metrics (each ranging from 0-100).”

“Further, we also tried loss weighting strategies (Chen et al., 2018; Liu et al., 2019a) but our preliminary experiments produced better re-sults without any scaling so we didn’t explore it further.”

“Multi-Task Learning Across Formats Multi-task learning (Caruana, 1993) with deep neural networks has shown improvements upon single-task training for related objectives (Liu et al., 2015, 2019b).”

“We explore two state-of-the-art methods to generate format-specific representations: control codes (Keskar et al., 2019; Raffel et al., 2020) as input signal; and parameter-efficient adapter methods (Pfeiffer et al., 2021; Stickland and Murray, 2019), where a separate network module is introduced…”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“While there has been significant progress in evaluating generalizability of NLP models (Ye et al., 2021; Sanh et al., 2021), evaluation of scientific document representations has remained limited.”

“NFCorpus This is a set of non technical English queries and associated results compiled from Nutri-tionFacts.org (Boteva et al., 2016 (Wang et al., 2020b) along with their relevance scores on a scale of 0-2.”

“Cohan et al. (2020) and Ostendorff et al. (2022b) fine-tune SciBERT (Beltagy et al., 2019) with a triplet loss that encourages papers citing each other to have similar embeddings, using the title and abstract of research papers as the input.”

“These methods are pre-trained on a set of over 1B query-candidate pairs including scientific text, and are two of the best BERT-base sized models on the recent MTEB benchmark (Muennighoff et al., 2022) for general-purpose text embeddings.”

“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”

“Multi-Task Learning Across Formats Multi-task learning (Caruana, 1993) with deep neural networks has shown improvements upon single-task training for related objectives (Liu et al., 2015, 2019b).”

“• Liu et al. (2014), an extension of Mimno and McCallum (2007) which adds 766 additional paper-review annotations.”

“We include TREC-CoVID (Voorhees et al., 2021) and NFCorpus (Boteva et al., 2016) as test tasks for this format.”

“The models variants can be further sub-categorized as retrieval based on best aspect matching (TS ASPIRE) and a weighted sum of the similarity score among all the aspects based on Optimal Transport (OT ASPIRE) between the query and candidates.”

“We train our models in a multi-task setup with task-heterogeneous batching (Aghajanyan et al., 2021).”

“Cohan et al. (2020) and Ostendorff et al. (2022b) fine-tune SciBERT (Beltagy et al., 2019) with a triplet loss that encourages papers citing each other to have similar embeddings, using the title and abstract of research papers as the input.”

“…between query and candidate papers which is a similar setting to our proximity tasks, so we only evaluate and report its results for that subset in Appendix C. Finally, we also evaluate two recent general-purpose text embedding methods, E5 v2 (Wang et al., 2022) and MPNet (Song et al., 2020).”

“When training on multiple tasks simultaneously, it is important to choose a combination of tasks that does not display negative transfer (Aribandi et al., 2022; Padmakumar et al., 2022; Fifty et al., 2021).”

“Pytrec_eval (Van Gysel and de Rijke, 2018) is used to calculate the ranking metrics.”

“Representing Scientific Documents Prior works have produced large scale language models pre-trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022).”

“…the S2AND author disambiguation task (Subrama-nian et al., 2021) with paper similarity features; Paper-Reviewer Matching (Mimno and McCallum, 2007; Liu et al., 2014; Zhao et al., 2022), where candidate reviewers are ranked by expert annotators based on the similarity of their papers to the query…”

“In contrast, SciRepEval provides more challenging and diverse tasks to help motivate methods for producing scientific document representations that can generalize well across tasks.”

“We include a total of ten proximity-based tasks, including four nearest neighbor tasks from SciDocs (predicting citations, co-citations, co-viewed or co-read papers), and three others from previous work: the S2AND author disambiguation task (Subrama-nian et al., 2021) with paper similarity features; Paper-Reviewer Mat…”

“We include a total of eleven proximitybased tasks, including four evaluation tasks from SciDocs (predicting citations and co-citations, and predicting co-viewed or co-read papers), and two others from previous work: the S2AND author disambiguation task (Subramanian et al., 2021) with paper similarity features, and Pap…”

“Existing benchmarks either focus on document similarity (Mysore et al., 2021; Voorhees et al., 2021) or tasks that are highly correlated and not diverse (Cohan et al., 2020).”

“Aghajanyan et al. (2021) pre-fine-tune language models on 46 tasks from 4 task types before fine-tuning on the downstream task.”

“However, SPECTER2 Base and MTL CLS are on par even though the latter is trained on in-domain datasets.”

“Supporting the findings in Padmakumar et al. (2022), pre-fine-tuning on the tasks from each individual format’s training data gives better or similar performance on downstream tasks when compared to training on all the formats at once, except on out-of-train classification and search.”

“…the two SciDocs tasks ( MAG and MeSH Diseases ), we have four others; including a binary task to predict whether a paper is relevant to biomimicry (Shyam et al., 2019), two biomedical classification tasks, namely DRSM from Burns (2022) and MeSH Descriptors classification (Lipscomb, 2000), and a…”

“Representing Scientific Documents Earlier work aimed at document embeddings used word vectors (J et al., 2016; Le & Mikolov, 2014; Wu et al., 2018), convolutions (Liu et al.”

“Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with contrastive learning (Xu et al., 2021; Gao et al., 2021; Neelakantan et al., 2022).”

“For evaluation, we consider predicting three numeric attributes related to prominence or quality: Tweet Mentions (Jain and Singh, 2021), and two new datasets predicting peer review rating and maximum h-index of authors for a set of ICLR papers from OpenReview 1 .”

“NFCorpus This is a set of non technical English queries and associated results compiled from Nutri-tionFacts.org (Boteva et al., 2016 (Wang et al., 2020b) along with their relevance scores on a scale of 0-2.”

“Recent approaches further modify the Transformer architecture in LLMs with Adapters (Singh et al., 2023), GNN-nested Transformers (Jin et al., 2023b), and Mixture-of-Experts Transformers (Zhang et al., 2023f) to better capture graph signals.”

“To identify AI papers in various fields across eras, we fine-tune BERT 32,33 , an established language model 34–36 , on articles published in explicitly AI-oriented scientific journals and conferences to automatically extract and interpret information from context.”

“Several works have designed specialized retrieval models for biomedical domains (Mohan et al., 2017; Liu et al., 2021; Jin et al., 2023; Luo et al., 2022a; Singh et al., 2023; Zhang et al., 2023).”

“The code, models, and data for SPECTER are available at https://github.com/allenai/SPECTER and for SciRep-Eval at https://github.com/allenai/SciRepEval.”