CoChat
Sign Up
PreprintCited by 223Topic ModelingInformation Retrieval and Search BehaviorMultimodal Machine Learning Applications

Enabling Large Language Models to Generate Text with Citations

Open access
Gao, Tianyu, Yen, Howard, Yu, Jiatong, Chen, Danqi·2023·arXiv
DOI 10.48550/arxiv.2305.14627arXiv 2305.14627
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) have emerged as a widely-used tool for information seeking, but their generated outputs are prone to hallucination. In this work, our aim is to allow LLMs to generate text with citations, improving their factual correctness and verifiability. Existing work mainly relies on commercial search engines and human evaluation, making it challenging to reproduce and compare different modeling approaches. We propose ALCE, the first benchmark for Automatic LLMs' Citation Evaluation. ALCE collects a diverse set of questions and retrieval corpora and requires building end-to-end systems to retrieve supporting evidence and generate answers with citations. We develop automatic metrics along three dimensions -- fluency, correctness, and citation quality -- and demonstrate their strong correlation with human judgements. Our experiments with state-of-the-art LLMs and novel prompting strategies show that current systems have considerable room for improvement -- For example, on the ELI5 dataset, even the best models lack complete citation support 50% of the time. Our analyses further highlight promising future directions, including developing better retrievers, advancing long-context LLMs, and improving the ability to synthesize information from multiple sources.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.14627

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Paragraph-level evaluation masks claim-level errors.

    “Current benchmarks fail in three ways: paragraph-level evaluation masks claimlevel errors [12, 13], Wikipedia datasets suffer LLM contamination [14, 15], and metrics ignore epistemic safety [16].”

    From Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citations (2024) · cites this paper as [13]

    Section 3.2 and Appendix A say that ROUGE does not reflect correctness well and show that a high ROUGE-L score can mask missing aspects of an ELI5 answer that claim recall detects. This supports the concern about answer-level metrics missing claim-level problems, but the paper does not establish that paragraph-level evaluation generally masks claim-level errors across current benchmarks. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The output of RAG is not guaranteed to be consistent with retrieved relevant passages because models are not explicitly trained to leverage and follow facts from provided passages.

    “Moreover, the output is not guaranteed to be consistent with retrieved relevant passages (Gao et al., 2023) since the models are not explicitly trained to leverage and follow facts from provided passages.”

    From CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance (2023) · cites this paper as Gao et al., 2023

    Section 5.3 (Retrieval Analysis; page unavailable) reports that models' correctness can lag behind retrieval recall even when accurate answers are in context, supporting the claim that outputs are not guaranteed to follow retrieved passages. Section 4.2 says this study uses prompting without fine-tuning, but it does not establish that a lack of explicit training is the cause of the inconsistency. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • ChatGPT consistently exhibits superior efficacy in ASQA, surpassing smaller LMs.

    “Gao et al. (2023) found that ChatGPT consistently exhibits superior efficacy in this particular task, surpassing smaller LMs.”

    From CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance (2023) · cites this paper as Gao et al., 2023

    Table 7 (Section 5.2) shows ChatGPT outperforming several smaller models on ASQA correctness and citation quality. But its superiority is not consistent across measures or models: LLaMA-2-70B-Chat scores 41.5 on correctness versus ChatGPT’s 40.4, and Section 5.2 describes its correctness as comparable to the OpenAI models. The claim overstates the comparison. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

  • Even when systems cite their sources, the citations are usually coarse (pointing to entire documents or paragraphs) and are produced after generation, so a reader must sift through irrelevant text to confirm a statement.

    “even when systems cite their sources, the citations are usually coarse-pointing to entire documents or paragraphs-and are produced after generation, so a reader must still sift through irrelevant text to confirm any given statement (Gao et al., 2023;Slobodkin et al., 2024).”

    From Attributable by Construction: Claim-Anchored Provenance for Multi-Document Summarization (2026) · cites this paper as Gao et al., 2023

    Section 2 (Task Setup and Datasets; page unavailable) says commercial systems such as Bing Chat cite entire Web pages, whereas the paper uses 100-word passages because they are easier to verify. Section 3.3 links irrelevant citations to human review of extraneous passages. The paper evaluates post-hoc citing as one approach, but does not say citations are usually produced after generation or usually point to paragraphs. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037. · arXiv
  2. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Mag- giore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In Inter- national Conference on Machine Learning (ICML), volume 162, pages 2206-2240.
  3. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP).
  4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Process- ing Systems (NeurIPS).
  5. Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Association for Computational Linguistics (ACL), pages 3558-3567. · DOI · Paper page
  6. Martin Funkquist, Ilia Kuznetsov, Yufang Hou, and Iryna Gurevych. 2022. CiteBench: A benchmark for Scientific Citation Text Generation. arXiv preprint arXiv:2212.09577. · arXiv
  7. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- cent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2023. RARR: Researching and revising what language models say, using language models. In Association for Computational Linguistics (ACL).
  8. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval- augmented language model pre-training. In Interna- tional Conference on Machine Learning (ICML).
  9. Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00303. · arXiv
  10. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations (ICLR).
  11. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (NAACL-HLT), pages 3905-3920. · DOI
  12. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  13. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Atlas: Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299. · arXiv
  14. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38. · DOI · Paper page
  15. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Ac- tive retrieval augmented generation. arXiv preprint arXiv:2305.06983. · arXiv
  16. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-World En- tailment for Claims in Wikipedia. arXiv preprint arXiv:2303.01432. · arXiv
  17. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics. · DOI · Paper page
  18. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations (ICLR).
  19. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Conference on Artificial In- telligence (AAAI), volume 32.
  20. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4940-4957, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  21. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Openassistant conversations -de- mocratizing large language model alignment. arXiv preprint arXiv:2304.07327. · arXiv
  22. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Association for Com- putational Linguistics (ACL), pages 6086-6096. · DOI · Paper page
  23. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell- Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147. · arXiv
  24. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics. · DOI
  25. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. WebGPT: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332. · arXiv
  26. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Empirical Methods in Natural Language Processing (EMNLP), pages 9844-9855.
  27. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural In- formation Processing Systems (NeurIPS), 35:27730- 27744.
  28. Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. TALM: Tool augmented language models. arXiv preprint arXiv:2205.12255. · arXiv
  29. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics. · DOI · Paper page
  30. Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izac- ard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wen-tau Yih, et al. 2021. The Web Is Your Oyster- Knowledge-Intensive NLP against a Very Large Web Corpus. arXiv preprint arXiv:2112.09924. · arXiv
  31. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: Measuring the gap be- tween neural text and human text using divergence frontiers. In Advances in Neural Information Pro- cessing Systems.
  32. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. · arXiv
  33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Trans- former. The Journal of Machine Learning Research (JMLR), 21(140).
  34. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 2383-2392. · DOI · Paper page
  35. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring Attribution in Natural Lan- guage Generation Models. Computational Linguis- tics, pages 1-64. · DOI · Paper page
  36. Samuel Joseph Amouyal Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022. QAMPARI: An Open-domain Question An- swering Benchmark for Questions with Many An- swers from Multiple Paragraphs. arXiv preprint arXiv:2205.12665. · arXiv
  37. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761. · arXiv
  38. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with con- trastive evidence. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 624- 643. · DOI · Paper page
  39. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Interna- tional Conference on Machine Learning (ICML).
  40. Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 373-393.
  41. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Wei Chang. 2022. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 8273-8288, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. · DOI
  42. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 809-819. · DOI · Paper page
  43. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. LLaMA: Open and Effi- cient Foundation Language Models. arXiv preprint arXiv:2302.13971. · arXiv
  44. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5008-5020, Online. Asso- ciation for Computational Linguistics. · DOI
  45. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT).
  46. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  47. Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. 2023. Automatic evaluation of at- tribution by large language models. arXiv preprint arXiv:2305.06311. · arXiv
  48. Shiyue Zhang and Mohit Bansal. 2021. Finding a bal- anced degree of automation for summary evaluation. In Empirical Methods in Natural Language Process- ing (EMNLP), pages 6617-6632. · DOI
  49. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (NAACL-HLT), pages 1298-1308. · DOI
  50. Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. In Empirical Methods in Natural Language Processing (EMNLP), pages 5657-5673. Fluency Correct. Citation (MAUVE) (EM Rec.) Rec. Prec. ROUGE-L Length ChatGPT VANILLA (5-psg) 66.8 (2.0) 40.4 (0.6) 73.6 (1.1) 72.5 (1.8) 37.0 (0.4) 40.0 (3.1) w/ RERANK 77.0 (0.0) 40.2 (0.0) 84.8 (0.0) 81.6 (0.0) 36.9 (0.0) 40.8 (0.0) SUMM (10-psg) 70.0 (1.2) 43.3 (0.8) 68.8 (0.6) 61.8 (1.1) 36.9 (0.2) 49.8 (4.3) w/ INTERACT 69.0 (2.7) 39.1 (0.5) 73.4 (0.2) 66.5 (4.9) 35.7 (0.2) 34.0 (0.9) SNIPPET (10-psg) 69.8 (2.5) 41.4 (0.6) 65.3 (0.6) 57.4 (0.9) 36.4 (0.4) 43.0 (3.5) INLINESEARCH 58.7 (1.3) 32.4 (0.6) 58.3 (1.3) 58.3 (1.3) 58.2 (1.1) 23.7 (1.1) CLOSEDBOOK 52.7 (4.9) 38.2 (0.1) 26.7 (1.1) 26.7 (1.1) 37.1 (0.3) 61.1 (4.5) ORACLE(5-psg) 64.4 (0.6) 48.9 (1.2) 74.5 (0.6) 72.7 (1.0) 38.2 (1.0) 37.4 (3.0)

Cited by

Sorted by citations

Authors
Gao, T., Yen, H., Yu, J., & Chen, D.
Affiliations
Princeton University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
6465-6488
DOI
10.48550/arxiv.2305.14627
arXiv
2305.14627
OpenAlex
W4389520670
Cited by
223
References
1
Citations by year
2026: 94 · 2025: 63 · 2024: 56 · 2023: 10
FWCI
39.32
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2239290, CAREER)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, citation generation, hallucinations, automatic evaluation metrics, retrieval-augmented generation, factual accuracy, citation quality, prompt engineering, information seeking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review