SPECTER: Document-level Representation Learning using Citation-informed Transformers
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 1 of the 26 sources its claims cite were checked for support.
Abstract
Representation learning is a critical ingredient for natural language processing systems.Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token-and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power.For applications on scientific documents, such as classification and recommendation, the embeddings power strong performance on end tasks.We propose SPECTER, a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph.Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning.Additionally, to encourage further research on document-level models, we introduce SCIDOCS, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation.We show that SPECTER outperforms a variety of competitive baselines on the benchmark.1
Overview
CoChat Summary
TL;DR
SPECTER learns document-level embeddings of scientific papers by pretraining a SciBERT Transformer on citation-informed triplets, using only title and abstract at inference. It introduces the SCIDOCS seven-task benchmark and reports an average score of 80.0 across tasks/metrics, a 3.1-point absolute improvement over the next-best baseline.
Key findings
- SPECTER achieves an average of 80.0 across all SCIDOCS tasks/metrics, 3.1 points absolute over the next-best baseline. On MeSH classification it reaches F1 86.4 (+2.3 over best baseline) and on MAG F1 82.0 (+1.5); on co-view MAP 83.8 and co-read MAP 84.5, improving over Citeomatic by 2.7 and 4.0 points; on co-citation nDCG 94.8, +2.3 over SGC; on recommendation nDCG 53.9.
- In an online A/B test with 4,113 clicks, the SPECTER-based ranker improved clickthrough rate over the baseline by 46.5%.
- Ablations show removing the abstract drops average performance to 68.1, adding authors drops it to 67.3, using no hard negatives drops it to 78.4, and initializing from BERT-Large instead of SciBERT drops it to 77.5. Fixed SPECTER embeddings also outperform SciBERT fine-tuned on task-specific signals (SPECTER average 80.0 vs. multitask fine-tuning 78.0).
- t-SNE clustering quality for SPECTER is homogeneity 0.41 and completeness 0.72, compared with SciBERT’s 0.19 and 0.63.
Methods
- SPECTER initializes from SciBERT, encodes the concatenated title and abstract, and uses the final [CLS] pooled output as the paper vector. It pretrains with a triplet margin loss on the citation graph: query paper, positive cited paper, and negative uncited paper, using L2 distance and margin m=1. Hard negatives are citations of citations; training uses 2 hard negatives and 3 easy negatives per query.
- Training data: a Semantic Scholar subset of about 146K query papers (~26.7M tokens) and 32K validation papers, yielding about 684K training triples and 145K validation triples. Training uses Adam with LR 2e-5, slanted triangular LR, 2 epochs, batch size 4 with gradient accumulation for effective batch size 32, on a single Titan V GPU.
- Evaluation: SCIDOCS covers seven tasks—MeSH classification (23K papers, 11 disease classes), MAG topic classification (25K papers, 19 level-1 classes), direct citation prediction (~30K total papers; 1K queries, up to 5 cited and 25 uncited candidates), co-citation (30K), co-views (~30K), co-reads (~30K), and recommendation (22K clickthrough events; 20K/1K/1K split; propensity-adjusted P@1 and nDCG). Baselines include SIF, SciBERT, Sent-BERT, Citeomatic, and SGC; classification uses a linear SVM, and recommendation uses a feedforward ranking network with ten features including cosine embedding distance.
Limitations
- Stated: SPECTER uses only title and abstract; adding full text could improve accuracy, but full text is often not freely available and Transformer memory limits would require new techniques. Venue metadata was not normalized, and author names were sparse/out-of-vocabulary; adding authors or venues did not help and authors hurt performance.
- Stated: On the recommendation task, differences between embedding methods are generally smaller because embeddings are used alongside several other informative features, limiting their impact on overall performance.
- Evident from the text: evaluation is limited to scientific documents and the released SCIDOCS tasks, with no evidence for other domains or languages; the online A/B test covers one production recommender and 4,113 clicks, so broader deployment evidence is not provided.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
1 Valid · 2 Not valid · 17 Not checked · 56 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“This is because SciBERT, like other pretrained language models, is trained via language modeling objectives, which only predict words or sentences given their in-document, nearby textual context.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > Document Representation
What the sources say
- Supportedcites Beltagy et al., 2019 matched to “SciBERT: A Pretrained Language Model for Scientific Text” (2019) source
The Methods (Sec. 2, Background) states that SciBERT follows BERT's architecture and pretraining tasks — 'predicting randomly masked tokens and predicting whether two sentences follow each other' — i.e. word-level and sentence-level objectives. The Corpus paragraph of Sec. 2 and Sec. 3.3 show these instances are built from the full text of papers with sentences split and packed up to 128/512 tokens, so the context the objectives use is nearby, in-document text. The Introduction likewise groups SciBERT with ELMo, GPT and BERT as unsupervised pretrained language models, covering the 'like other pretrained language models' clause. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the recommendations task, as opposed to previous experiments, the differences in method scores are generally smaller.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 5 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that these pretrained embeddings substantially outperform alternative off-theshelf embeddings.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 3. Fasttext-Sum
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we show in our experiments ( §6), including hard negatives results in more accurate embeddings compared to using random negatives alone.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 2.4 Selecting Negative Distractors
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We achieve substantial improvements over the strongest of a wide variety of baselines, demonstrating the effectiveness of our model.”
§ SPECTER: Document-level Representation Learning using Citation-informed Transformers > 8 Conclusions and Future Work
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 76 claims and to verify the 18 references still open.
- It may state two separate claims as one. 22
- Probably not a substantial claim. 29
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Covered model families span content-and embedding-based methods (e.g.,TF-IDF [63], BM25 [59] , SPECTER/SciBERT [17] ), graph-and HIN/GNN-based methods (GraphSAGE, Graph Attention Networks (GATs), meta-path reasoning), and hybrid/LLM pipelines (prompting, re-ranking, RAG).”
From A Critical Review on Citation Recommendation Systems Through a Graph-Centric Approach (2026) · cites this paper as [17]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853
(1996) · Cited by 20,179
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 2
- William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017b. Inductive representation learning on large graphs. In NIPS.
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training. arXiv.
Cited by
Items 101–200 of 1,111
Long, D. X., Nguyen, N.-H., Sim, T., Dao, H. T., Joty, S., Kawaguchi, K., Chen, N. F., & Kan, M. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.15 · Cited by 43
Token Pooling: Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact
Clavié, B., Chaffin, A., Adams, G., & Kato, M. P. (2026) · Journal of Natural Language Processing · doi:10.5715/jnlp.33.1484 · Cited by 42
Prompting Strategies for Citation Classification
Kunnath, S. N., Pride, D., & Knoth, P. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615018 · Cited by 42
SciSpace Literature Review: Harnessing AI for Effortless Scientific Discovery
Jain, S., Kumar, A., Roy, T., Shinde, K., Vignesh, G., & Tondulkar, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_28 · Cited by 41
Enhancing citation recommendation using citation network embedding
Pornprasit, C., Liu, X., Kiattipadungkul, P., Kertkeidkachorn, N., Kim, K., Noraset, T., Hassan, S., & Tuarob, S. (2022) · Scientometrics · doi:10.1007/s11192-021-04196-3 · Cited by 41
CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by ExamplePreprint
Mysore, S., O’Gorman, T., McCallum, A., & Zamani, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12906 · Cited by 42
The Effect of Public Science on Corporate R&D
Arora, A., Belenzon, S., Cioaca, L. C., Sheer, L., & Zhang, H. (2023) · National Bureau of Economic Research · doi:10.3386/w31899 · Cited by 41
Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers
Chen, S. H., Gutiérrez, B. J., & Su, Y. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.02642 · Cited by 42
Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration
Dai, S., Liu, W., Zhou, Y., Pang, L., Ruan, R., Wang, G., Dong, Z., Xu, J., & Wen, J.-R. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.421 · Cited by 41
SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks
Zhao, Y., Zhang, K., Hu, T., Wu, S., Bras, R. J. L., Liu, Y., Tang, R., Chang, J. C., Dodge, J., Bragg, J., et al. (2025) · neural information processing systems · doi:10.52202/085713-3532 · Cited by 40
Self-citationParameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers
Tam, W., Liu, X., Ji, K., Xue, L., Liu, J., Li, T., Dong, Y., & Tang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.874 · Cited by 40
Streamlining Cross-Document Coreference Resolution: Evaluation and ModelingPreprint
Cattan, A., Eirew, A., Stanovsky, G., Joshi, M., & Dagan, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11032 · Cited by 40
SciReviewGen: A Large-scale Dataset for Automatic Literature Review Generation
Kasanishi, T., Isonuma, M., Mori, J., & Sakata, I. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.418 · Cited by 39
PaECTER: Patent-level representation learning using citation-informed transformers
Ghosh, M., Rose, M. E., Erhardt, S., Buunk, E., & Harhoff, D. (2026) · PLoS ONE · doi:10.1371/journal.pone.0358188 · Cited by 39
OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining
Zhang, F., Shi, S., Zhu, Y., Chen, B., Cen, Y., Yu, J., Chen, Y., Wang, L., Zhao, Q., Cheng, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672354 · Cited by 38
Specialized document embeddings for aspect-based similarity of research papers
Ostendorff, M., Blume, T., Ruas, T., Gipp, B., & Rehm, G. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530912 · Cited by 38
Hierarchical Multi-Label Classification of Scientific Documents
Sadat, M., & Caragea, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.610 · Cited by 40
EdgeRAG: Online-Indexed RAG for Edge DevicesPreprint
Seemakhupt, K., Liu, S., & Khan, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.21023 · Cited by 40
Text-Attributed Graph Representation Learning: Methods, Applications, and Challenges
Zhang, D. C., Yang, M., Ying, R., & Lauw, H. W. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3641255 · Cited by 37
Exploring the Landscape of Natural Language Processing Research
Schopf, T., Arabi, K., & Matthes, F. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_111 · Cited by 38
When Large Language Models Meet Citation: A SurveyPreprint
Zhang, Y., Wang, Y., Wang, K., Sheng, Q. Z., Yao, L., Mahmood, A., Zhang, W. E., & Zhao, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.09727 · Cited by 38
Citation recommendation employing heterogeneous bibliographic network embedding
Ali, Z., Qi, G., Muhammad, K., Bhattacharyya, S., Ullah, I., & Abro, W. A. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06135-y · Cited by 36
SciXGen: A Scientific Paper Dataset for Context-Aware Text Generation
Chen, H., Takamura, H., & Nakayama, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.128 · Cited by 36
Heterogeneous Graph Neural Networks for Keyphrase Generation
Ye, J., Cai, R., Gui, T., & Zhang, Q. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.213 · Cited by 36
Ginzburg, D., Malkiel, I., Barkan, O., Caciularu, A., & Koenigstein, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.272 · Cited by 35
Moro, G., & Valgimigli, L. (2021) · Sensors · doi:10.3390/s21196430 · Cited by 35
The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study
Zhang, Y., Jin, B., Zhu, Q., Yu, M., & Han, J. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583354 · Cited by 35
Autonomous LLM-Driven Research — from Data to Human-Verifiable Research Papers
Ifargan, T., Hafner, L., Kern, M., Alcalay, O., & Kishony, R. (2025) · NEJM AI · doi:10.1056/aioa2400555 · Cited by 34
Hierarchical Graph Convolutional Networks for Structured Long Document Classification
Liu, T., Hu, Y., Wang, B., Sun, Y., Gao, J., & Yin, B. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3185295 · Cited by 35
Beyond Summarization: Designing AI Support for Real-World Expository Writing TasksPreprint
Shen, Z., August, T., Siangliulue, P., Lo, K., Bragg, J., Hammerbacher, J., Downey, D., Chang, J. C., & Sontag, D. A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02623 · Cited by 37
Self-citationKim, H., Sohn, J., Gilson, A., Cochran-Caggiano, N., Applebaum, S. S., Jin, H., Park, S., Park, Y., Park, J., Choi, S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.06738 · Cited by 35
Self-citationSciMMIR: Benchmarking Scientific Multi-modal Information Retrieval
Wu, S., Li, Y., Zhu, K., Zhang, G., Liang, Y., Ma, K., Xiao, C., Zhang, H., Yang, B., Chen, W., et al. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.746 · Cited by 33
Social-RAG: Retrieving from Group Interactions to Socially Ground AI Generation
Wang, R., Zhou, X., Qiu, L., Chang, J. C., Bragg, J., & Zhang, A. X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713749 · Cited by 35
He, J., Zhang, B., Rouhizadeh, H., Chen, Y., Yang, R., Lu, J. S., Chen, X., Liu, N., & Teodoro, D. (2026) · Artificial Intelligence Review · doi:10.21203/rs.3.rs-8330917/v1 · Cited by 35
SPR-SMN: scientific paper recommendation employing SPECTER with memory network
Ali, Z., Qi, G., Kefalas, P., Khusro, S., Khan, I. U., & Muhammad, K. (2022) · Scientometrics · doi:10.1007/s11192-022-04425-3 · Cited by 33
An anatomization of research paper recommender system: Overview, approaches and challenges
Sharma, R., Gopalani, D., & Meena, Y. K. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105641 · Cited by 33
WhiteningBERT: An Easy Unsupervised Sentence Embedding ApproachPreprint
Huang, J., Tang, D., Zhong, W., Lu, S., Shou, L., Gong, M., Jiang, D., & Duan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01767 · Cited by 33
From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP
Mosbach, M., Gautam, V., Browne, T. V., Klakow, D., & Geva, M. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.181 · Cited by 30
Jin, J. (2025) · International Journal of Graphic Design · doi:10.51903/ijgd.v3i2.3698 · Cited by 30
MUVERA: Multi-Vector Retrieval via Fixed Dimensional EncodingsPreprint
Dhulipala, L., Hadian, M., Jayaram, R., Lee, J., & Mirrokni, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.19504 · Cited by 31
Your Mixture-of-Experts LLM Is Secretly an Embedding Model For FreePreprint
Li, Z., & Zhou, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.10814 · Cited by 29
Invernici, F., Curati, F., Jakimov, J., Samavi, A., & Bernasconi, A. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01189-4 · Cited by 28
Instruct-DeBERTa: A Hybrid Approach for Aspect-based Sentiment Analysis on Textual Reviews
Jayakody, D., Malkith, A. V. A., Isuranda, K., Thenuwara, V., de Silva, N., Ponnamperuma, S. R., Sandamali, G. G. N., & Sudheera, K. L. K. (2025) · International Journal on Advances in ICT for Emerging Regions (ICTer) · doi:10.4038/icter.v18i2.7290 · Cited by 28
Length-Induced Embedding Collapse in PLM-based Models
Zhou, Y., Dai, S., Cao, Z., Zhang, X., & Xu, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1396 · Cited by 29
Literature-Grounded Novelty Assessment of Scientific Ideas
Shahid, S., Radensky, M., Fok, R., Siangliulue, P., Weld, D. S., & Hope, T. (2025) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2025.sdp-1.9 · Cited by 28
Self-citationRELiC: Retrieving Evidence for Literary Claims
Thai, K., Chang, Y., Krishna, K., & Iyyer, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.517 · Cited by 28
SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
Choi, E., Lee, S., Choi, M., Ko, H., Song, Y.-I., & Lee, J. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557456 · Cited by 28
Lu, Y., Yuan, M., Liu, J., & Chen, M. (2023) · Scientometrics · doi:10.1007/s11192-022-04566-5 · Cited by 28
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Zhang, Y., Cheng, H., Shen, Z., Liu, X., Wang, Y., & Gao, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.820 · Cited by 28
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and AnalysisPreprint
Yu, J., Ding, Z., Tan, J., Luo, K., Weng, Z., Gong, C., Zeng, L., Cui, R., Han, C., Sun, Q., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.12857 · Cited by 28
HiBench: Benchmarking LLMs Capability on Hierarchical Structure Reasoning
Jiang, Z., Wu, P., Liang, Z., Chen, P. Q., Xu, Y., Jia, Y., Tu, J., Li, C., Ng, P. H. F., & Li, Q. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737378 · Cited by 27
Corpus Poisoning via Approximate Greedy Gradient Descent
Su, J., Nakov, P., & Cardie, C. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.222 · Cited by 28
Weakly Supervised Multi-Label Classification of Full-Text Scientific Papers
Zhang, Y., Jin, B., Chen, X., Shen, Y., Zhang, Y., Yu, M., & Han, J. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599544 · Cited by 27
Dense Retrieval Adaptation using Target Domain Description
Hashemi, H., Zhuang, Y., Kothur, S. S. R., Prasad, S., Meij, E., & Croft, W. B. (2023) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3578337.3605127 · Cited by 27
Papeos: Augmenting Research Papers with Talk Videos
Kim, T. S., Latzke, M., Bragg, J., Zhang, A. X., & Chang, J. C. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606770 · Cited by 27
Complex online harms and the smart home: A scoping review
Olabode, S. A., Owens, R., Zhang, V. N., Copilah-Ali, J., Kolomeets, M., Wu, H., Malviya, S., Markeviciute, K., Spiliotopoulos, T., Neesham, C., et al. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.08.019 · Cited by 27
A dataset of mentorship in bioscience with semantic and demographic estimations
Ke, Q., Liang, L., Ding, Y., David, S. V., & Acuna, D. E. (2022) · Scientific Data · doi:10.1038/s41597-022-01578-x · Cited by 27
HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction
Hao, Q., Fan, J., Xu, F., Yuan, J., & Li, Y. (2024) · neural information processing systems · doi:10.52202/079017-1527 · Cited by 26
Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training
Xu, R., Yu, Y., Ho, J. C., & Yang, C. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592085 · Cited by 26
A weakly supervised textual entailment approach to zero-shot text classification
Pàmies, M., Palao, J. L., Multari, F., Durán-Silva, N., Parra-Rojas, C. A., González-Agirre, A., Massucci, F. A., & Villegas, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.22 · Cited by 26
When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews
Kumar, S., Ghosal, T., & Ekbal, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1038 · Cited by 26
A Survey of Pre-trained Language Models for Processing Scientific TextPreprint
Ho, X., Nguyen, A.-K. D., Dao, A., Jiang, J., Chida, Y., Sugimoto, K., To, H. Q., Boudin, F., & Aizawa, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.17824 · Cited by 26
Mitigating Bias in RAG: Controlling the Embedder
Kim, T., Springer, J. M., Raghunathan, A., & Sap, M. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.974 · Cited by 26
Aspect-based Sentiment Analysis Techniques: A Comparative Study
Jayakody, D., Isuranda, K., Malkith, A. V. A., de Silva, N., Ponnamperuma, S. R., Sandamali, G. G. N., & Sudheera, K. L. K. (2024) · Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon63886.2024.10688631 · Cited by 25
Chain-of-Factors Paper-Reviewer Matching
Zhang, Y., Shen, Y., Kang, S., Chen, X., Jin, B., & Han, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714708 · Cited by 25
HindSight: Evaluating LLM-Generated Research Ideas via Future ImpactPreprint
Jiang, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.15164 · Cited by 26
Simons, A., Zichert, M., & Wüthrich, A. (2026) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2026.102151 · Cited by 24
ComLittee: Literature Discovery with Personal Elected Author Committees
Kang, H. B., Soliman, N., Latzke, M., Chang, J. C., & Bragg, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581371 · Cited by 24
Modeling Information Change in Science Communication with Semantically Matched Paraphrases
Wright, D., Pei, J., Jurgens, D., & Augenstein, I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.117 · Cited by 24
Graph-based methods for Author Name Disambiguation: a survey
Bonis, M. D., Falchi, F., & Manghi, P. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1536 · Cited by 24
Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling
Chang, H.-S., Sun, R.-Y., Ricci, K., & McCallum, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.48 · Cited by 24
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.840 · Cited by 23
Deep learning for journal recommendation system of research papers
Gündoğan, E., Kaya, M., & Daud, A. Y. (2022) · Scientometrics · doi:10.1007/s11192-022-04535-y · Cited by 23
Researching Alignment Research: Unsupervised AnalysisPreprint
Kirchner, J. H., Smith, L. T., Thibodeau, J., McDonell, K., & Reynolds, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.02841 · Cited by 23
Woo, S., & Walsh, J. P. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00303 · Cited by 22
Xu, S., Pang, L., Shen, H., & Cheng, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.365 · Cited by 22
The Birth of a New Discipline: Data Science Education
Mike, K., Kimelfeld, B., & Hazzan, O. (2023) · Harvard Data Science Review · doi:10.1162/99608f92.280afe66 · Cited by 22
Enhancing Lexicon-Based Text Embeddings with Large Language Models
Lei, Y., Shen, T., Cao, Y. H., & Yates, A. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.930 · Cited by 22
Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment
Deng, J., Jiang, Z., Pang, L., Wei, Z., Chen, L., Xu, K., Yang, S., Shen, H., & Cheng, X. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.639 · Cited by 22
F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual WorldPreprint
Zhang, Z., Liao, Z.-H., Yu, H., Di, P., & Wang, R. (2026) · arXiv.org · doi:10.48550/arxiv.2603.19223 · Cited by 17
Academic Expert Finding via $(k, \mathcal{P})$-Core based Embedding over Heterogeneous Graphs
Xu, X., Liu, J., Wang, Y., & Ke, X. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00030 · Cited by 21
Choudhury, M. H., Salsabil, L., Jayanetti, H. R., Wu, J. F., Ingram, W. A., & Fox, E. A. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00019 · Cited by 21
Wang, Z., Zhang, H., Chen, H., Feng, Y., & Ding, J. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102119 · Cited by 21
Chaudhary, A., Raman, K., & Bendersky, M. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.107 · Cited by 21
An automated approach to identify scientific publications reporting pharmacokinetic parameters
Gonzalez Hernandez, F., Carter, S. J., Iso-Sipilä, J., Goldsmith, P., Almousa, A. A., Gastine, S., Lilaonitkul, W., Kloprogge, F., & Standing, J. F. (2021) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16718.1 · Cited by 17
A New Query Expansion Approach via Agent-Mediated Dialogic InquiryPreprint
Seo, W., An, H., & Lee, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.08557 · Cited by 21
MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers
Kalra, J., Zhao, X., Kim, T. E., Cai, F., Díaz, F., & Wu, T. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.601 · Cited by 20
Efficient Domain Adaptation of Sentence Embeddings Using Adapters
Schopf, T., Schneider, D. N., & Matthes, F. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_112 · Cited by 20
Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy
Kang, S., Agarwal, S., Jin, B., Lee, D., Yu, H., & Han, J. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645512 · Cited by 20
Routing-Guided Learned Product Quantization for Graph-Based Approximate Nearest Neighbor Search
Yue, Q., Xu, X., Wang, Y., Tao, Y., & Luo, X. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00370 · Cited by 20
Bazaga, A., Lió, P., & Micklem, G. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.537 · Cited by 20
Learning Domain Semantics and Cross-Domain Correlations for Paper Recommendation
Xie, Y., Sun, Y., & Bertino, E. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462975 · Cited by 20
A Study of Computational Reproducibility using URLs Linking to Open Access Datasets and Software
Salsabil, L., Wu, J. F., Choudhury, M. H., Ingram, W. A., Fox, E. A., Rajtmajer, S. M., & Giles, C. L. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524658 · Cited by 20
Multi2Claim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking
Tan, N. Ö., Nguyen, T., Bensemann, J., Peng, A. Y., Bao, Q., Chen, Y., GAHEGAN, M. N., & Witbrock, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.194 · Cited by 20
Enhancing local citation recommendation with recurrent highway networks and SciBERT-based embedding
Dinh, T. N., Pham, P., Giang, N. L., & Vo, B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122911 · Cited by 19
Citation Recommendation for Research Papers via Knowledge Graphs
Brack, A., Hoppe, A., & Ewerth, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86324-1_20 · Cited by 19
RPT: Toward Transferable Model on Heterogeneous Researcher Data via Pre-Training
Qiao, Z., Fu, Y., Wang, P., Xiao, M., Ning, Z., Zhang, D., Du, Y., & Zhou, Y. (2022) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3152386 · Cited by 19
A comparative evaluation of biomedical similar article recommendation
Zhang, L., Lu, W. P., Chen, H., Huang, Y., & Cheng, Q. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104106 · Cited by 19
Metadata
- Authors
- Cohan, A., Feldman, S., Beltagy, I., Downey, D., & Weld, D.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2270-2282
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3016913119
- Cited by
- 1,108
- References
- 62
- Citations by year
- 2026: 364 · 2025: 244 · 2024: 178 · 2023: 157 · 2022: 113 · 2021: 44 · 2020: 8
- FWCI
- 21.99
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (cc-by)
- Funders
- National Science Foundation (1936940)
- University of Washington
- Office of Naval Research (N00014- 18-1-2193)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- citation networks, citation prediction, scientific documents, transformer language models, pre-training, document classification, recommendations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

