Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding

Open access
Zeng, Hansi, Luo, Chen, Zamani, Hamed · 2024 · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Abstract

This paper introduces PAG-a novel optimization and decoding approach that guides autoregressive generation of document identifiers in generative retrieval models through simultaneous decoding. To this aim, PAG constructs a set-based and sequential identifier for each document. Motivated by the bag-of-words assumption in information retrieval, the set-based identifier is built on lexical tokens. The sequential identifier, on the other hand, is obtained via quantizing relevance-based representations of documents. Extensive experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin (e.g., 15.6% MRR improvements on MS MARCO), while achieving 22× speed up in terms of query latency.

Overview

Integrity signals

Registry

Crossref · 10.1145/3626772.3657746

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 29 references

Research works printed but not linked: 57

  1. Dmitry Baranchuk, Artem Babenko, and Yury Malkov. 2018. Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighbors. ArXiv, Vol. abs/1802.02422. https://api.semanticscholar.org/CorpusID:3602418 · arXiv
  2. Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen tau Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive Search Engines: Generating Substrings as Document Identifiers. ArXiv, Vol. abs/2204.10628. https://api.semanticscholar.org/CorpusID:248366293 · arXiv
  3. Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. ArXiv, Vol. abs/1611.09268. https://api.semanticscholar.org/CorpusID:1289517 · arXiv
  4. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive Entity Retrieval. ArXiv, Vol. abs/2010.00904. https://api.semanticscholar.org/CorpusID:222125277 · arXiv
  5. Xilun Chen, Kushal Lakhotia, Barlas O?uz, Anchit Gupta, Patrick Lewis, Stanislav Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen tau Yih. 2021. Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? ArXiv, Vol. abs/2110.06918. https://api.semanticscholar.org/CorpusID:238744204 · arXiv
  6. Yongjian Chen, Tao Guan, and Cheng Wang. 2010. Approximate Nearest Neighbor Search by Residual Vector Quantization. Sensors (Basel, Switzerland), Vol. 10, 11259--11273. https://api.semanticscholar.org/CorpusID:33774240
  7. David R. Cheriton. 2019. From doc2query to docTTTTTquery. https://api.semanticscholar.org/CorpusID:208612557
  8. Hyung Won Chung et al. 2022. Scaling Instruction-Finetuned Language Models. ArXiv Vol. abs/2210.11416. https://api.semanticscholar.org/CorpusID:253018554 · arXiv
  9. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2019. Overview of the TREC 2019 Deep Learning Track. In TREC.
  10. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. 2021. Overview of the TREC 2020 Deep Learning Track. ArXiv, Vol. abs/2102.07662. https://api.semanticscholar.org/CorpusID:212737158 · arXiv
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics. https://api.semanticscholar.org/CorpusID:52967399
  12. Thibault Formal, C. Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021a. SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. ArXiv, Vol. abs/2109.10086. https://api.semanticscholar.org/CorpusID:237581550 · arXiv
  13. Alex Graves. 2012. Sequence Transduction with Recurrent Neural Networks. ArXiv, Vol. abs/1211.3711. https://api.semanticscholar.org/CorpusID:17194112 · arXiv
  14. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning. PMLR, 3929--3938.
  15. Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. ArXiv, Vol. abs/2010.02666. https://api.semanticscholar.org/CorpusID:222141041 · arXiv
  16. 10.1109/TBDATA.2019.2921572 · DOI · Paper page
  17. Vladimir Karpukhin, Barlas O?uz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Conference on Empirical Methods in Natural Language Processing. https://api.semanticscholar.org/CorpusID:215737187
  18. O. Khattab and Matei A. Zaharia. 2020a. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. https://api.semanticscholar.org/CorpusID:216553223
  19. O. Khattab and Matei A. Zaharia. 2020b. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. https://api.semanticscholar.org/CorpusID:216553223
  20. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR, Vol. abs/1412.6980. https://api.semanticscholar.org/CorpusID:6628106 · arXiv
  21. Hyunji Lee, Jaeyoung Kim, Hoyeon Chang, Hanseok Oh, Sohee Yang, Vladimir Karpukhin, Yi Lu, and Minjoon Seo. 2022. Nonparametric Decoding for Generative Retrieval. In Annual Meeting of the Association for Computational Linguistics. https://api.semanticscholar.org/CorpusID:258959550
  22. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. ArXiv, Vol. abs/2005.11401. https://api.semanticscholar.org/CorpusID:218869575 · arXiv
  23. Yongqing Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2023 a. Learning to Rank in Generative Retrieval. ArXiv, Vol. abs/2306.15222. https://api.semanticscholar.org/CorpusID:259262395 · arXiv
  24. Yongqing Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2023 b. Multiview Identifiers Enhanced Generative Retrieval. In Annual Meeting of the Association for Computational Linguistics. https://api.semanticscholar.org/CorpusID:258947148
  25. Jimmy J. Lin and Xueguang Ma. 2021. A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques. ArXiv, Vol. abs/2106.14807. https://api.semanticscholar.org/CorpusID:235658149 · arXiv
  26. Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy J. Lin. 2020. Distilling Dense Representations for Ranking using Tightly-Coupled Teachers. ArXiv, Vol. abs/2010.11386. https://api.semanticscholar.org/CorpusID:225041183 · arXiv
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, Vol. abs/1907.11692. https://api.semanticscholar.org/CorpusID:198953378 · arXiv
  28. Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Jinfeng Rao, Marc Najork, Emma Strubell, and Donald Metzler. 2022. DSI: Updating Transformer Memory with New Documents. ArXiv, Vol. abs/2212.09744. https://api.semanticscholar.org/CorpusID:254854290 · arXiv
  29. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. ArXiv, Vol. abs/1901.04085. https://api.semanticscholar.org/CorpusID:58004692 · arXiv
  30. Long Ouyang et al. 2022. Training language models to follow instructions with human feedback. ArXiv Vol. abs/2203.02155. https://api.semanticscholar.org/CorpusID:246426909 · arXiv
  31. Biswajit Paria, Chih-Kuan Yeh, Ning Xu, Barnabás Póczos, Pradeep Ravikumar, and Ian En-Hsu Yen. 2020a. Minimizing FLOPs to Learn Efficient Sparse Representations. ArXiv, Vol. abs/2004.05665. https://api.semanticscholar.org/CorpusID:211107043 · arXiv
  32. Biswajit Paria, Chih-Kuan Yeh, Ning Xu, Barnabás Póczos, Pradeep Ravikumar, and Ian En-Hsu Yen. 2020b. Minimizing FLOPs to Learn Efficient Sparse Representations. ArXiv, Vol. abs/2004.05665. https://api.semanticscholar.org/CorpusID:211107043 · arXiv
  33. Jay M. Ponte and W. Bruce Croft. 1998. A language modeling approach to information retrieval. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. https://api.semanticscholar.org/CorpusID:14103653
  34. Ronak Pradeep, Kai Hui, Jai Gupta, Ádám Dániel Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q. Tran. 2023. How Does Generative Retrieval Scale to Millions of Passages? ArXiv, Vol. abs/2305.11841. https://api.semanticscholar.org/CorpusID:258822999 · arXiv
  35. Ronak Pradeep, Rodrigo Nogueira, and Jimmy J. Lin. 2021. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. ArXiv, Vol. abs/2101.05667. https://api.semanticscholar.org/CorpusID:231603106 · arXiv
  36. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In North American Chapter of the Association for Computational Linguistics. https://api.semanticscholar.org/CorpusID:231815627
  37. Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H. Keshavan, Trung Hieu Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Maheswaran Sathiamoorthy. 2023. Recommender Systems with Generative Retrieval. ArXiv, Vol. abs/2305.05065. https://api.semanticscholar.org/CorpusID:258564854 · arXiv
  38. Ruiyang Ren, Wayne Xin Zhao, J. Liu, Huaqin Wu, Ji rong Wen, and Haifeng Wang. 2023. TOME: A Two-stage Approach for Model-based Retrieval. ArXiv, Vol. abs/2305.11161. https://api.semanticscholar.org/CorpusID:258762633 · arXiv
  39. 10.5555/275537.275701 · doi:10.5555/275537.275701
  40. Stephen E. Robertson and Steve Walker. 1997. On relevance weights with little relevance information. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. https://api.semanticscholar.org/CorpusID:16829071
  41. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2024 b. LaMP: When Large Language Models Meet Personalization. arxiv: 2304.11406 [cs.CL] · arXiv
  42. Chris Samarinas, Atharva Nijasure Pracha Promthaw, Hansi Zeng, Julian Killingback, and Hamed Zamani. 2024. Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models. In arXiv preprint arXiv:2404.14772. · arXiv
  43. Felix Stahlberg and Bill Byrne. 2019. On NMT Search Errors and Model Errors: Cat Got Your Tongue? ArXiv, Vol. abs/1908.10090. https://api.semanticscholar.org/CorpusID:201646223 · arXiv
  44. Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, M. de Rijke, and Zhaochun Ren. 2023. Learning to Tokenize for Generative Retrieval. ArXiv, Vol. abs/2304.04171. https://api.semanticscholar.org/CorpusID:258048596 · arXiv
  45. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. ArXiv, Vol. abs/1409.3215. https://api.semanticscholar.org/CorpusID:7961699 · arXiv
  46. Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer Memory as a Differentiable Search Index. ArXiv, Vol. abs/2202.06991. https://api.semanticscholar.org/CorpusID:246863488 · arXiv
  47. Shuai Wang, Shengyao Zhuang, and G. Zuccon. 2021. BERT-based Dense Retrievers Require Interpolation with BM25 for Effective Passage Retrieval. Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. https://api.semanticscholar.org/CorpusID:237366133
  48. Yujing Wang, Ying Hou, Hong Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A Neural Corpus Indexer for Document Retrieval. ArXiv, Vol. abs/2206.02743. https://api.semanticscholar.org/CorpusID:249395549 · arXiv
  49. Tianxin Wei Bowen Jin Ruirui Li Hansi Zeng Zhengyang Wang Jianhui Sun Qingyu Yin Hanqing Lu Suhang Wang Jingrui He and Xianfeng Tang. 2024. Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=khAE1sTMdX
  50. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. ArXiv, Vol. abs/2007.00808. https://api.semanticscholar.org/CorpusID:220302524 · arXiv
  51. Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik G. Learned-Miller, and J. Kamps. 2018. From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing. Proceedings of the 27th ACM International Conference on Information and Knowledge Management. https://api.semanticscholar.org/CorpusID:52229883
  52. Cheng Xiang Zhai and John D. Lafferty. 2001. Model-based feedback in the language modeling approach to information retrieval. In International Conference on Information and Knowledge Management. https://api.semanticscholar.org/CorpusID:1043470
  53. Peitian Zhang, Zheng Liu, Yujia Zhou, Zhicheng Dou, and Zhao Cao. 2023. Term-Sets Can Be Strong Document Identifiers For Auto-Regressive Search Engines. ArXiv, Vol. abs/2305.13859. https://api.semanticscholar.org/CorpusID:258841428 · arXiv
  54. R. Zhou and Eric A. Hansen. 2005. Beam-Stack Search: Integrating Backtracking with Beam Search. In International Conference on Automated Planning and Scheduling. https://api.semanticscholar.org/CorpusID:11314454
  55. Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Yu Wu, Peitian Zhang, and Ji rong Wen. 2022. Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer. ArXiv, Vol. abs/2208.09257. https://api.semanticscholar.org/CorpusID:251710261 · arXiv
  56. Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2022a. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://api.semanticscholar.org/CorpusID:252993059
  57. Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, G. Zuccon, and Daxin Jiang. 2022b. Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation. ArXiv, Vol. abs/2206.10128. https://api.semanticscholar.org/CorpusID:249890267 · arXiv

Cited by

See all 23

Metadata

Authors
Zeng, H., Luo, C., & Zamani, H.
Affiliations
  • University of Massachusetts Amherst (US)
  • Amazon (United States) (US)
Venue
Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2024-07-10
Pages
469-480
OpenAlex
W4400528921
Cited by
22
References
29
Citations by year
2026: 9 · 2025: 11 · 2024: 2
FWCI
8.38
Type
conference paper
Language
en
Open access
Locations
  • International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
generative retrieval, autoregressive generation, document identifiers, query response time, periaqueductal gray

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding. https://app.cochat.ai/papers/10.1145/3626772.3657746
The paper
Zeng, H., Luo, C., & Zamani, H. (2024). Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3626772.3657746

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.