SPECTER: Document-level Representation Learning using Citation-informed Transformers
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 1 of the 26 sources its claims cite were checked for support.
Abstract
Representation learning is a critical ingredient for natural language processing systems.Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token-and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power.For applications on scientific documents, such as classification and recommendation, the embeddings power strong performance on end tasks.We propose SPECTER, a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph.Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning.Additionally, to encourage further research on document-level models, we introduce SCIDOCS, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation.We show that SPECTER outperforms a variety of competitive baselines on the benchmark.1
Overview
CoChat Summary
TL;DR
SPECTER learns document-level embeddings of scientific papers by pretraining a SciBERT Transformer on citation-informed triplets, using only title and abstract at inference. It introduces the SCIDOCS seven-task benchmark and reports an average score of 80.0 across tasks/metrics, a 3.1-point absolute improvement over the next-best baseline.
Key findings
- SPECTER achieves an average of 80.0 across all SCIDOCS tasks/metrics, 3.1 points absolute over the next-best baseline. On MeSH classification it reaches F1 86.4 (+2.3 over best baseline) and on MAG F1 82.0 (+1.5); on co-view MAP 83.8 and co-read MAP 84.5, improving over Citeomatic by 2.7 and 4.0 points; on co-citation nDCG 94.8, +2.3 over SGC; on recommendation nDCG 53.9.
- In an online A/B test with 4,113 clicks, the SPECTER-based ranker improved clickthrough rate over the baseline by 46.5%.
- Ablations show removing the abstract drops average performance to 68.1, adding authors drops it to 67.3, using no hard negatives drops it to 78.4, and initializing from BERT-Large instead of SciBERT drops it to 77.5. Fixed SPECTER embeddings also outperform SciBERT fine-tuned on task-specific signals (SPECTER average 80.0 vs. multitask fine-tuning 78.0).
- t-SNE clustering quality for SPECTER is homogeneity 0.41 and completeness 0.72, compared with SciBERT’s 0.19 and 0.63.
Methods
- SPECTER initializes from SciBERT, encodes the concatenated title and abstract, and uses the final [CLS] pooled output as the paper vector. It pretrains with a triplet margin loss on the citation graph: query paper, positive cited paper, and negative uncited paper, using L2 distance and margin m=1. Hard negatives are citations of citations; training uses 2 hard negatives and 3 easy negatives per query.
- Training data: a Semantic Scholar subset of about 146K query papers (~26.7M tokens) and 32K validation papers, yielding about 684K training triples and 145K validation triples. Training uses Adam with LR 2e-5, slanted triangular LR, 2 epochs, batch size 4 with gradient accumulation for effective batch size 32, on a single Titan V GPU.
- Evaluation: SCIDOCS covers seven tasks—MeSH classification (23K papers, 11 disease classes), MAG topic classification (25K papers, 19 level-1 classes), direct citation prediction (~30K total papers; 1K queries, up to 5 cited and 25 uncited candidates), co-citation (30K), co-views (~30K), co-reads (~30K), and recommendation (22K clickthrough events; 20K/1K/1K split; propensity-adjusted P@1 and nDCG). Baselines include SIF, SciBERT, Sent-BERT, Citeomatic, and SGC; classification uses a linear SVM, and recommendation uses a feedforward ranking network with ten features including cosine embedding distance.
Limitations
- Stated: SPECTER uses only title and abstract; adding full text could improve accuracy, but full text is often not freely available and Transformer memory limits would require new techniques. Venue metadata was not normalized, and author names were sparse/out-of-vocabulary; adding authors or venues did not help and authors hurt performance.
- Stated: On the recommendation task, differences between embedding methods are generally smaller because embeddings are used alongside several other informative features, limiting their impact on overall performance.
- Evident from the text: evaluation is limited to scientific documents and the released SCIDOCS tasks, with no evidence for other domains or languages; the online A/B test covers one production recommender and 4,113 clicks, so broader deployment evidence is not provided.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
1 Valid · 2 Not valid · 17 Not checked · 56 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“This is because SciBERT, like other pretrained language models, is trained via language modeling objectives, which only predict words or sentences given their in-document, nearby textual context.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > Document Representation
What the sources say
- Supportedcites Beltagy et al., 2019 matched to “SciBERT: A Pretrained Language Model for Scientific Text” (2019) source
The Methods (Sec. 2, Background) states that SciBERT follows BERT's architecture and pretraining tasks — 'predicting randomly masked tokens and predicting whether two sentences follow each other' — i.e. word-level and sentence-level objectives. The Corpus paragraph of Sec. 2 and Sec. 3.3 show these instances are built from the full text of papers with sentences split and packed up to 128/512 tokens, so the context the objectives use is nearby, in-document text. The Introduction likewise groups SciBERT with ELMo, GPT and BERT as unsupervised pretrained language models, covering the 'like other pretrained language models' clause. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the recommendations task, as opposed to previous experiments, the differences in method scores are generally smaller.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 5 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that these pretrained embeddings substantially outperform alternative off-theshelf embeddings.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 3. Fasttext-Sum
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we show in our experiments ( §6), including hard negatives results in more accurate embeddings compared to using random negatives alone.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 2.4 Selecting Negative Distractors
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We achieve substantial improvements over the strongest of a wide variety of baselines, demonstrating the effectiveness of our model.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 8 Conclusions and Future Work
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 76 claims and to verify the 18 references still open.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 29
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Covered model families span content-and embedding-based methods (e.g.,TF-IDF [63], BM25 [59] , SPECTER/SciBERT [17] ), graph-and HIN/GNN-based methods (GraphSAGE, Graph Attention Networks (GATs), meta-path reasoning), and hybrid/LLM pipelines (prompting, re-ranking, RAG).”
From A Critical Review on Citation Recommendation Systems Through a Graph-Centric Approach (2026) · cites this paper as [17]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853
(1996) · Cited by 20,179
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 2
- William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017b. Inductive representation learning on large graphs. In NIPS.
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training. arXiv.
Cited by
Items 201–300 of 1,111
Xian, X., Wang, G., Bi, X., Srinivasa, J., Kundu, A., Fleming, C. B., Hong, M., & Ding, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.17275 · Cited by 20
Is Semantic Chunking Worth the Computational Cost?Preprint
Qu, R., Bao, F. S., & Tu, R. (2024) · Qeios · doi:10.32388/m7ykdz · Cited by 19
A general method for real-time detection of information threats with a Ukraine case study
Lipianina-Honcharenko, K., Komar, M., Bohuta, H., Ihnatiev, I., Yurkiv, K., Illiashenko, O. A., & Bilovus, L. I. (2025) · RADIOELECTRONIC AND COMPUTER SYSTEMS · doi:10.32620/reks.2025.3.15 · Cited by 18
Huang, J., Huang, W.-B., Bu, Y., Cao, Q., Shen, H., & Cheng, X. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101427 · Cited by 18
Measuring the commercial potential of science
Masclans, R., Hasan, S., & Cohen, W. M. (2025) · Strategic Management Journal · doi:10.1002/smj.3720 · Cited by 18
Keyphrase Generation Beyond the Boundaries of Title and Abstract
Garg, K., Chowdhury, J. R., & Caragea, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.427 · Cited by 18
Wang, Y., Liu, J., Xu, X., Ke, X., Wu, T., & Gou, X. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3578365 · Cited by 18
Dynamic Context Extraction for Citation Classification
Kunnath, S. N., Pride, D., & Knoth, P. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.41 · Cited by 18
Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation
Boytsov, L., Akinpelu, D., Katyal, N., Lin, T., Gao, F., Zhao, Y., Huang, J., & Nyberg, E. (2025) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2025.ijcnlp-long.46 · Cited by 18
Persistent Hierarchy in Contemporary International CollaborationPreprint
Miao, L., Larivière, V., Lee, B., Ahn, Y., & Sugimoto, C. R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.13020 · Cited by 18
F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source DataPreprint
Zhang, Z., Liao, Z.-H., Yu, H., Di, P., & Wang, R. (2025) · arXiv.org · doi:10.48550/arxiv.2510.02294 · Cited by 8
Nechakhin, V., D’Souza, J., & Eger, S. (2024) · Information · doi:10.3390/info15060328 · Cited by 17
Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review
Kuznetsov, I., Buchmann, J., Eichler, M., & Gurevych, I. (2022) · Computational Linguistics · doi:10.1162/coli_a_00455 · Cited by 17
TR-MTEB: A Comprehensive Benchmark and Embedding Model Suite for Turkish Sentence Representations
Baysan, M. S., & Güngör, T. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.471 · Cited by 17
PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&A
Martin, A., Leckey, C. A. C., Brown, M. C., & Kaur, H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791101 · Cited by 18
Predicting substantive biomedical citations without full text
Hoppe, T. A., Arabi, S., & Hutchins, B. I. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2213697120 · Cited by 17
SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval
Wu, S., Li, Y., Zhu, K., Zhang, G., Liang, Y., Ma, K., Xiao, C., Zhang, H., Yang, B., Chen, W., et al. (2024) · Research Explorer (The University of Manchester) · doi:10.48550/arxiv.2401.13478 · Cited by 17
Exploring the evolution of research topics during the COVID-19 pandemic
Invernici, F., Bernasconi, A., & Ceri, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124028 · Cited by 17
exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem
Ebrahimi, S., Salamat, S., Arabzadeh, N., Bashari, M., & Bagheri, E. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_1 · Cited by 18
Unsupervised extreme multi label classification of stack overflow posts
Devine, P., & Blincoe, K. (2022) · IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) · doi:10.1145/3528588.3528652 · Cited by 17
Scientific Paper Extractive Summarization Enhanced by Citation Graphs
Chen, X., Li, M., Gao, S., Yan, R., Gao, X., & Zhang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.270 · Cited by 17
I3: I ntent- I ntrospective Retrieval Conditioned on I nstructions
Pan, K., Li, J., Wang, W., Fei, H., Song, H., Ji, W., Lin, J., Liu, X., Chua, T., & Tang, S. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657745 · Cited by 18
Seed-Guided Topic Discovery with Out-of-Vocabulary Seeds
Zhang, Y., Yu, M., Wang, X., Wang, S., & Han, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.21 · Cited by 17
NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear DomainPreprint
Acharya, A., Munikoti, S., Hellinger, A., Smith, S. E., Wagle, S., & Horawalavithana, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.10920 · Cited by 17
Unify Graph Learning with Text: Unleashing LLM Potentials for Session Search
Wu, S., Tu, Q., Liu, H., Xu, J., Liu, Z., Zhang, G., Wang, R., Chen, X., & Yan, R. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645574 · Cited by 16
Elliott, O. P., & Clark, J. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672512 · Cited by 16
Shen, Y., Zhang, Y., Zhang, Y., & Han, J. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.167 · Cited by 16
Effective distributed representations for academic expert search
Berger, M., Zavrel, J., & Groth, P. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.7 · Cited by 16
DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based QueriesPreprint
Wang, J., Wang, K., Wang, X., Naidu, P., Bergen, L., & Paturi, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.04678 · Cited by 16
SPAR: Scholar Paper Retrieval with LLM-based Agents for Enhanced Academic SearchPreprint
Shi, X., Li, Y., Kou, Q., Yu, L., Xie, J., & Zhou, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.15245 · Cited by 16
Trends and applications of variational autoencoders in medical imaging analysis
Yeoh, P. S. Q., Hasikin, K. B., Wu, X., Goh, S. L., & Lai, K. W. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102647 · Cited by 15
RottenReviews: Benchmarking Review Quality with Human and LLM-Based Judgments
Ebrahimi, S., Sadeghian, S., Ghorbanpour, A., Arabzadeh, N., Salamat, S., Li, M., Le, H. S., Bashari, M., & Bagheri, E. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761506 · Cited by 15
Scientific document processing: challenges for modern learning methods
Kashyap, A. R., Yang, Y., & Kan, M. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00352-7 · Cited by 15
Scientific paper recommender system using deep learning and link prediction in citation network
Li, W. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e34685 · Cited by 15
CorpusStudio: Surfacing Emergent Patterns In A Corpus Of Prior Work While Writing
Dang, H., Swoopes, C., Buschek, D., & Glassman, E. L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713974 · Cited by 15
Bevendorff, J., Dementieva, D., Fröbe, M., Gipp, B., Greiner-Petter, A., Karlgren, J., Mayerl, M., Nakov, P., Panchenko, A., Potthast, M., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_21 · Cited by 16
Chen, J., Sun, X., & Brown, V. (2023) · Journal of Advanced Computing Systems · doi:10.69987/jacs.2023.30102 · Cited by 15
Counterfactual Evaluation of Peer-Review Assignment Policies
Saveski, M., Jecmen, S., Shah, N. B., & Ugander, J. H. (2023) · neural information processing systems · doi:10.52202/075280-2563 · Cited by 16
HITS-based attentional neural model for abstractive summarization
Cai, X., Shi, K., Jiang, Y., Yang, L., & Liu, S. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106996 · Cited by 15
Cross-lingual citations in English papers: a large-scale analysis of prevalence, usage, and impact
Saier, T., Färber, M., & Tsereteli, T. (2021) · International Journal on Digital Libraries · doi:10.1007/s00799-021-00312-z · Cited by 15
Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector
Nguyen, T., Lei, Y.-B., Ju, J.-H., Yang, E., & Yates, A. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2510.00671 · Cited by 19
A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
Huang, Y., Tang, K., Chen, M., & Wang, B. (2024) · arXiv · doi:10.48550/arxiv.2404.15777 · Cited by 14
Labor Space: A Unifying Representation of the Labor Market via Large Language Models
Kim, S., Ahn, Y., & Park, J. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645464 · Cited by 14
A Hybrid Citation Recommendation Model With SciBERT and GraphSAGE
Dinh, T. N., Pham, P., Giang, N. L., Nguyên, N. T., & Vo, B. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3490774 · Cited by 14
Arhiliuc, C., Guns, R., Daelemans, W. M. P., & Engels, T. (2024) · Scientometrics · doi:10.1007/s11192-024-05217-7 · Cited by 15
State Space Models are Strong Text Rerankers
Xu, Z., Yan, J., Gupta, A., & Srikumar, V. (2025) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2025.repl4nlp-1.12 · Cited by 14
Efficient Re-ranking with Cross-encoders via Early Exit
Busolin, F., Lucchese, C., Nardini, F. M., Orlando, S., Perego, R., Trani, S., & Veneri, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729962 · Cited by 14
ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research
Lin, J., Dai, L., Han, R., Sui, Y., Wang, R., Sun, X., Wu, Q., Feng, M., Liu, H., & Xiong, H. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737432 · Cited by 14
MA-DPR: Manifold-aware Distance Metrics for Dense Passage Retrieval
Liu, Y., Wen, Q., Zhao, M., Liang, J., & Sanner, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1582 · Cited by 14
What Should I Cite? A RAG Benchmark for Academic Citation Prediction
Zheng, L., Zhang, J., Chen, C., Wang, C., Li, H., Li, Y., Mao, Y., Yan, S., Song, Z., Feng, Z., et al. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792075 · Cited by 14
LIMEADE: From AI Explanations to Advice Taking
Lee, B. C. G., Downey, D., Lo, K., & Weld, D. S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3589345 · Cited by 14
Self-citationDialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources
Hwang, Y., Kim, Y., Bae, H., Lee, H., Bang, J., & Jung, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.545 · Cited by 14
Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories
Ge, S., Xiong, C., Rosset, C., Overwijk, A., Han, J., & Bennett, P. N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.111 · Cited by 14
PubGraph: A Large-Scale Scientific Knowledge GraphPreprint
Ahrabian, K., Du, X., Myloth, R. D., Ananthan, A. B. S., & Pujara, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.02231 · Cited by 14
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T. (2024) · arXiv · doi:10.48550/arxiv.2407.18940 · Cited by 13
Schopf, T., Gerber, E., Ostendorff, M., & Matthes, F. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_113 · Cited by 13
Reviewerly: Modeling the Reviewer Assignment Task as an Information Retrieval Problem
Arabzadeh, N., Ebrahimi, S., Salamat, S., Bashari, M., & Bagheri, E. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679081 · Cited by 13
MixGR: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularity
Cai, F., Zhao, X., Chen, T., Chen, S., Zhang, H., Gurevych, I., & Koeppl, H. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.579 · Cited by 13
Hou, J., Zheng, B., Li, H., & Li, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04617-1 · Cited by 13
Tracing the evolution of household carbon emission research by machine learning
Long, Y., Huang, L., Montagna, S., Chen, Z., Ding, X., Shigetomi, Y., Zhao, T., Asatani, K., Sakata, I., & Yoshida, Y. (2025) · Environmental Impact Assessment Review · doi:10.1016/j.eiar.2025.108039 · Cited by 14
Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation
Zhang, H., Tang, M., Bi, K., Guo, J., Liu, S., Shi, D., Yin, D., & Cheng, X. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.88 · Cited by 13
CiteBART: Learning to Generate Citations for Local Citation Recommendation
Çelik, E. Y., & Tekır, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.89 · Cited by 13
Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation
Zhang, H., Bi, K., Guo, J., Zhang, J., Wang, S., Yin, D., & Cheng, X. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769496 · Cited by 15
Agentic memory-augmented retrieval and evidence grounding for medical question-answering tasks
Jia, S., Bit, S., Jasodanand, V. H., Liu, Y., & Kolachalama, V. B. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106339 · Cited by 14
Leveraging full-text article exploration for citation analysis
Quatra, M. L., Cagliero, L., & Baralis, E. (2021) · Scientometrics · doi:10.1007/s11192-021-04117-4 · Cited by 13
ULTRA: A Data-driven Approach for Recommending Team Formation in Response to Proposal Calls
Srivastava, B., Koppel, T., Paladi, S. T., Valluru, S. L., Sharma, R., & Bond, O. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00130 · Cited by 13
Gerasimenko, N. A., Chernyavsky, A. S., & Никифорова, М. А. (2022) · Doklady Mathematics · doi:10.1134/s1064562422060072 · Cited by 13
DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning
Wang, X., He, J., Wang, P., Zhou, Y., Sun, T., & Qiu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29886 · Cited by 13
LEGOBench: Scientific Leaderboard Generation Benchmark
Singh, S., Alam, S., Malwat, H., & Singh, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.855 · Cited by 13
COVIDScholar: An automated COVID-19 research aggregation and analysis platform
Dagdelen, J. M., Trewartha, A., Huo, H., Fei, Y., He, T., Cruse, K., Wang, Z., Subramanian, A., Justus, B., Ceder, G., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281147 · Cited by 13
Embedding Recycling for Language Models
Saad-Falcon, J., Singh, A., Soldaini, L., D’Arcy, M., Cohan, A., & Downey, D. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.145 · Cited by 13
Self-citationAugmenting Scientific Creativity with Retrieval across Knowledge DomainsPreprint
Kang, H. B., Mysore, S., Huang, K., Chang, H.-S., Prein, T., McCallum, A., Kittur, A., & Olivetti, E. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.01328 · Cited by 13
MACQ: A Holistic View of Model Acquisition TechniquesPreprint
Callanan, E., Venezia, R. D., Armstrong, V., Paredes, A., Chakraborti, T., & Muise, C. J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.06530 · Cited by 13
Bias Evaluation and Mitigation in Retrieval-Augmented Medical Question-Answering SystemsPreprint
Ji, Y., Zhang, H., & Wang, Y. (2025) · PubMed Central · doi:10.48550/arxiv.2503.15454 · Cited by 13
AI Research Agents Narrow Scientific ExplorationPreprint
Tang, Y., & Yang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.27905 · Cited by 13
Unsupervised Domain Adaptation for Keyphrase Generation using Citation Contexts
Boudin, F., & Aizawa, A. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.33 · Cited by 12
Hierarchical Attention Transformer Networks for Long Document Classification
Hu, Y., Chen, P., Liu, T., Gao, J., Sun, Y., & Yin, B. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533869 · Cited by 12
Metricbert: Text Representation Learning Via Self-Supervised Triplet Training
Malkiel, I., Ginzburg, D., Barkan, O., Caciularu, A., Weill, Y., & Koenigstein, N. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746018 · Cited by 12
FeedLens: Polymorphic Lenses for Personalizing Exploratory Search over Knowledge Graphs
Kaur, H., Downey, D., Singh, A., Cheng, E. Y.-Y., Weld, D. S., & Bragg, J. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545631 · Cited by 12
Self-citationGarcía, G. S., Khalaf, R. A., Invernici, F., Ceri, S., & Bernasconi, A. (2022) · GigaScience · doi:10.1093/gigascience/giad036 · Cited by 12
Overview of the Context24 Shared Task on Contextualizing Scientific Claims
Chan, C. S. J., Naik, A., Akamatsu, M., Bekele, H., Bransom, E., Campbell, I., & Sparks, J. (2024) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2024.sdp-1.3 · Cited by 12
A Survey of Citation Recommendation Tasks and Methods
Medić, Z., & Šnajder, J. (2021) · Journal of Computing and Information Technology · doi:10.20532/cit.2020.1005160 · Cited by 12
On the Interpolation of Contextualized Term-based Ranking with BM25 for Query-by-Example Retrieval
Abolghasemi, A., Askari, A., & Verberne, S. (2022) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3539813.3545133 · Cited by 12
Bøgh, S., Hain, D. S., Hansen, E. B., Jensen, S. B., Tvedebrink, T., & Jurowetzki, R. (2022) · Springer eBooks · doi:10.1007/978-3-031-15428-7_22 · Cited by 12
Topic Segmentation Model Focusing on Local ContextPreprint
Lee, J., Han, J., Baek, S., & Song, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.01935 · Cited by 12
IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question AnsweringPreprint
Shorten, C., Skaburskas, A., Jones, D. M., Pierse, C., Esposito, R., Trengrove, J., Dilocker, E., & van Luijt, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17687 · Cited by 12
Unlocking LLM Creativity in Science through Analogical ReasoningPreprint
Shen, A., Druckmann, S., & Zou, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.11258 · Cited by 12
Improved paper recommendation model incorporating knowledge graph embedding with IRGAN model
Thierry, N., Chatelain, I. B. C., Ali, Z., & Kefalas, P. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3710898 · Cited by 12
Kong, M., Zhang, Y., Sheng, L., & Hong, K. (2025) · Scientometrics · doi:10.1007/s11192-025-05356-5 · Cited by 11
ContReviews: A content-based recommendation system for updating Living Evidences in health care
Tenti, P., Thomas, J., Peñaloza, R., & Pasi, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112981 · Cited by 11
Investigating Task Arithmetic for Zero-Shot Information Retrieval
Braga, M., Kasela, P., Raganato, A., & Pasi, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730216 · Cited by 11
A supervised machine learning approach for assessing grant peer review reports
Okasa, G., León, A. R., Strinzel, M., Jorstad, A., Milzow, K., Egger, M., & Müller, S. (2025) · Quantitative Science Studies · doi:10.1162/qss.a.23 · Cited by 11
Naik, A., Lehman, J. F., & Rosé, C. P. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00500 · Cited by 11
QUADRo: Dataset and Models for QUestion-Answer Database Retrieval
Campese, S., Lauriola, I., & Moschitti, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1042 · Cited by 11
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 11
Analysis of Patterns and Trends in COVID-19 Research
Dornick, C., Kumar, A., Seidenberger, S., Seidle, E., & Mukherjee, P. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.05.032 · Cited by 11
Textomics: A Dataset for Genomics Data Summary Generation
Wang, M., Liu, Z., & Wang, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.335 · Cited by 11
MS2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · arXiv · doi:10.48550/arxiv.2104.06486 · Cited by 10
Self-citationMa, C., & Wolfinger, R. D. (2024) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.4c00134 · Cited by 10
Large Language Models for Predicting Empathic Accuracy Between a Designer and a User
Fabunmi, O., Halgamuge, S. K., Beck, D., & Hölttä‐Otto, K. (2024) · Journal of Mechanical Design · doi:10.1115/1.4067227 · Cited by 10
Metadata
- Authors
- Cohan, A., Feldman, S., Beltagy, I., Downey, D., & Weld, D.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2270-2282
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3016913119
- Cited by
- 1,108
- References
- 62
- Citations by year
- 2026: 364 · 2025: 244 · 2024: 178 · 2023: 157 · 2022: 113 · 2021: 44 · 2020: 8
- FWCI
- 21.99
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (cc-by)
- Funders
- National Science Foundation (1936940)
- University of Washington
- Office of Naval Research (N00014- 18-1-2193)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- citation networks, citation prediction, scientific documents, transformer language models, pre-training, document classification, recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

