RankGen: Improving Text Generation with Large Ranking Models
Open accessAbstract
Given an input sequence (or prefix), modern language models often assign high probabilities to output sequences that are repetitive, incoherent, or irrelevant to the prefix; as such, model-generated text also contains such artifacts.To address these issues we present RANKGEN, a 1.2B parameter encoder model for English that scores model generations given a prefix.RANKGEN can be flexibly incorporated as a scoring function in beam search and used to decode from any pretrained language model.We train RANKGEN using large-scale contrastive learning to map a prefix close to the ground-truth sequence that follows it and far away from two types of negatives:(1) random sequences from the same document as the prefix, and (2) sequences generated from a large language model conditioned on the prefix.Experiments across four different language models (345M-11B parameters) and two domains show that RANKGEN significantly outperforms decoding algorithms like nucleus, top-k, and typical sampling on both automatic metrics (85.0 vs 77.3 MAUVE) as well as human evaluations with English writers (74.5% human preference over nucleus sampling).Analysis reveals that RANKGEN outputs are more relevant to the prefix and improve continuity and coherence compared to baselines.We release our model checkpoints, code, and human preference data with explanations to facilitate future research.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The randomness in LLMs typically comes from the sampling methods used during text generation, such as top-k sampling or nucleus sampling.
“The randomness in LLMs typically comes from the sampling methods used during text generation, such as top-k sampling or nucleus sampling [31, 49].”
From An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2025) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
(2016) · neural information processing systems · Cited by 1,572 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Dense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279 · see citing works
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232 · see citing works
Hobbs, J. R. (1979) · Cognitive Science · doi:10.1207/s15516709cog0301_4 · Cited by 907 · see citing works
Learning Distributed Representations of Sentences from Unlabelled Data
Hill, F., Cho, K., & Korhonen, A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1162 · Cited by 566 · see citing works
(2020) · International Conference on Machine Learning · Cited by 358 · see citing works
Sharp nearby, fuzzy far away: How neural language models use context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303 · see citing works
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285 · see citing works
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · see citing works
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTPreprint
Khattab, O., & Zaharia, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12832 · Cited by 270 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated TextPreprint
Gehrmann, S., Clark, E., & Sellam, T. (2022) · Journal of Artificial Intelligence Research · doi:10.48550/arxiv.2202.06935 · Cited by 263 · see citing works
Scaling Up Models and Data with t5x and seqioPreprint
Roberts, A., Chung, H. W., Mishra, G., Levskaya, A., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., et al. (2022) · Journal of machine learning research · doi:10.48550/arxiv.2203.17189 · Cited by 232 · see citing works
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207 · see citing works
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works
Discriminative Reranking for Machine Translation
Shen, L., Sarkar, A., & Och, F. J. (2004) · Cited by 202 · see citing works
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192 · see citing works
Internet-Augmented Dialogue Generation
Komeili, M., Shuster, K., & Weston, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.579 · Cited by 192 · see citing works
Do Massively Pretrained Language Models Make Better Storytellers?
See, A., Pappu, A., Saxena, R., Yerukola, A., & Manning, C. D. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1079 · Cited by 139 · see citing works
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127 · see citing works
Hurdles to Progress in Long-form Question Answering
Krishna, K., Roy, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.393 · Cited by 114 · see citing works
Self-citationDiscourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
Jernite, Y., Bowman, S. R., & Sontag, D. (2017) · arXiv · doi:10.48550/arxiv.1705.00557 · Cited by 101 · see citing works
A Contrastive Framework for Neural Text GenerationPreprint
Su, Y., Lan, T., Wang, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06417 · Cited by 83 · see citing works
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Bakhtin, A., Gross, S., Ott, M., Deng, Y., Ranzato, M., & Szlam, A. (2019) · arXiv · doi:10.48550/arxiv.1906.03351 · Cited by 77 · see citing works
Tackling the Story Ending Biases in The Story Cloze Test
Sharma, R., Allen, J. F., Bakhshandeh, O., & Mostafazadeh, N. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2119 · Cited by 59 · see citing works
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56 · see citing works
Self-citationTwin Networks: Matching the Future for Sequence GenerationPreprint
Serdyuk, D., Ke, N. R., Sordoni, A., Trischler, A., Pal, C., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06742 · Cited by 48 · see citing works
Do Long-Range Language Models Actually Use Long-Range Context?
Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.62 · Cited by 43 · see citing works
Self-citationLearning to Decode for Future SuccessPreprint
Li, J., Monroe, W., & Jurafsky, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.06549 · Cited by 40 · see citing works
CoNT: Contrastive Neural Text GenerationPreprint
An, C., Feng, J., Lv, K., Kong, L., Qiu, X., & Huang, X. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.14690 · Cited by 37 · see citing works
On Losses for Modern Language Models
Aroca-Ouellette, S., & Rudzicz, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.403 · Cited by 29 · see citing works
Prediction with a short memory
Sharan, V., Kakade, S. M., Liang, P., & Valiant, G. J. (2018) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/3188745.3188954 · Cited by 24 · see citing works
Toward Better Storylines with Sentence-Level Language Models
Ippolito, D., Grangier, D., Eck, D., & Callison-Burch, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.666 · Cited by 20 · see citing works
RELiC: Retrieving Evidence for Literary Claims
Thai, K., Chang, Y., Krishna, K., & Iyyer, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.517 · Cited by 11 · see citing works
Self-citationChapterBreak: A Challenge Dataset for Long-Range Language Models
Sun, S., Thai, K., & Iyyer, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.271 · Cited by 7 · see citing works
Self-citationBRIO: Bringing Order to Abstractive SummarizationPreprint
Liu, Y., Liu, P., Radev, D., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.16804 · Cited by 4 · see citing works
Peter: A Novel of Which He is Not the Hero
Smith, F. H. (1908) · Medical Entomology and Zoology · Cited by 1 · see citing works
Cut the CARP: Fishing for zero-shot story evaluationPreprint
Matiana, S., Smith, Jr., Teehan, R., Castricato, L., Biderman, S., Gao, L., & Frazier, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03111 · Cited by 1 · see citing works
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732
Flagged first, then by citations
Research works printed but not linked: 37
- Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for se- quence prediction with recurrent neural networks. NeurIPS, 28.
- Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and An- drew McCallum. 2021. Energy-based reranking: Improving neural machine translation using energy- based models. In ACL-IJCNLP. · DOI
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference of Ma- chine Learning.
- Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 7282-7296, Online. Association for Computational Linguistics. · DOI · Paper page
- Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language mod- els: a simple approach to controlled text generation. In Proceedings of the International Conference on Learning Representations.
- Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. 2022. The efficiency misnomer. In International Conference on Learning Representations.
- Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc'Aurelio Ranzato. 2020. Residual energy- based models for text generation. In International Conference on Learning Representations.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Conference of the North American Chapter of the Association for Computational Lin- guistics, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
- Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah Smith, and Yejin Choi. 2022. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 7250-7274, Dublin, Ireland. Associ- ation for Computational Linguistics.
- Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial exam- ples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Com- putational Linguistics. · DOI
- Sergey Edunov, Myle Ott, Michael Auli, David Grang- ier, and Marc'Aurelio Ranzato. 2018. Classical structured prediction losses for sequence to se- quence learning. In Conference of the North Amer- ican Chapter of the Association for Computational Linguistics, pages 355-364. Association for Compu- tational Linguistics. · DOI
- Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi- erarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics. · DOI · Paper page
- Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
- Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visu- alization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations, pages 111-116, Florence, Italy. Association for Computa- tional Linguistics. · DOI · Paper page
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative ad- versarial nets. Advances in neural information pro- cessing systems, 27.
- Barbara J Grosz, Scott Weinstein, and Aravind K Joshi. 1995. Centering: a framework for modeling the lo- cal coherence of discourse. Computational Linguis- tics, 21(2):203-225.
- Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric J Horvitz, and Stefano Ermon. 2019. Bias correction of learned generative models using likelihood-free importance weighting. Advances in neural information processing systems, 32. 209
- Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. Ad- vances in neural information processing systems, 28.
- Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proceed- ings of the First Workshop on Neural Machine Trans- lation, pages 28-39, Vancouver. Association for Computational Linguistics. · DOI · Paper page
- Ann Lee, Michael Auli, and Marc'Aurelio Ranzato. 2021. Discriminative reranking for neural machine translation. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers), pages 7250-7264, Online. Association for Computa- tional Linguistics. · DOI
- Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics. · DOI · Paper page
- Lajanugen Logeswaran and Honglak Lee. 2018. An ef- ficient framework for learning sentence representa- tions. In International Conference on Learning Rep- resentations.
- Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, and Sebastian Riedel. 2020. How decoding strategies affect the verifiability of gener- ated text. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pages 223-235, Online. Association for Computational Linguistics. · DOI
- Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics. · DOI · Paper page
- Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022. Typical decoding for natural language generation. arXiv preprint arXiv:2202.00666. · arXiv
- Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research soft- ware platform. In Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84. · DOI
- Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg-Kirkpatrick. 2022. Mix and match: Learning- free controllable text generation using energy lan- guage models. In Proceedings of the Association for Computational Linguistics.
- Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A cor- pus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguis- tics. · DOI · Paper page
- Tetiana Parshakova, Jean-Marc Andreoli, and Marc Dymetman. 2019. Global autoregressive models for data-efficient sequence learning. In Proceed- ings of the 23rd Conference on Computational Nat- ural Language Learning (CoNLL), pages 900-909, Hong Kong, China. Association for Computational Linguistics. · DOI
- Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap be- tween neural text and human text using divergence frontiers. Advances in Neural Information Process- ing Systems, 34.
- Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401-2410, Online. Associa- tion for Computational Linguistics. · DOI
- Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. Cold decoding: Energy-based constrained text generation with langevin dynamics. In Proceedings of Advances in Neural Information Processing Systems.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th Interna- tional Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
- Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604. PMLR.
- Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evalua- tion of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 2).
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of Advances in Neural In- formation Processing Systems, pages 5998-6008.
Cited by
Sorted by citations
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · see citing works
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
A Critical Evaluation of Evaluations for Long-form Question Answering
Xu, F., Song, Y., Iyyer, M., & Choi, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.181 · Cited by 33 · see citing works
Self-citation
- Authors
- Krishna, K., Chang, Y., Wieting, J., & Iyyer, M.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Published
- 2022
- Pages
- 199-232
- OpenAlex
- W4385572722
- Cited by
- 30
- References
- 47
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 6 · 2023: 14 · 2022: 2
- FWCI
- 3.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- text generation, large language models, ranking models, beam search, contrastive learning, nucleus sampling, top-k sampling, text coherence, text relevance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
