Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 3

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Learning to Generate Answers with Citations via Factual Consistency Models

Aly, Rami, Tang, Zhiqiang, Tan, Samson, Karypis, George·2024·Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2024.acl-long.641
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.acl-long.641

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 2 A schematic view of our iterative citation fine-tuning method CaLF. It uses a factual consistency model to: i) create weakly supervised training instances by filtering diversely sampled responses, ii) adjust the loss contribution of each answer token according to its Shapley relevance for factual consistency prediction.
Figure 2 A schematic view of our iterative citation fine-tuning method CaLF. It uses a factual consistency model to: i) create weakly supervised training instances by filtering diversely sampled responses, ii) adjust the loss contribution of each answer token according to its Shapley relevance for factual consistency prediction.CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Figure 3 The computation of relevance weights W for rescaling the loss according to Eq. 2. We first use SHAP to measure the token importance for predicting ϕ ( s i , C i ) = o i . We adjust for differences in scale of W ϕ,i for sentences s i and differences in tokenization between the FCM and the LLM.
Figure 3 The computation of relevance weights W for rescaling the loss according to Eq. 2. We first use SHAP to measure the token importance for predicting ϕ ( s i , C i ) = o i . We adjust for differences in scale of W ϕ,i for sentences s i and differences in tokenization between the FCM and the LLM.CC-BY · From the PDF
Figure 4 Evaluation metrics and CaLF's dynamic stopping criterion over the number of iterations on ASQA.
Figure 4 Evaluation metrics and CaLF's dynamic stopping criterion over the number of iterations on ASQA.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 53

  1. Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella La- pata. 2023. Multilingual summarization with factual consistency evaluation. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 3562-3591, Toronto, Canada. Association for Com- putational Linguistics. · DOI
  2. Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Car- lier, Michael Noetel, and Andreas Stuhlmüller. 2021. RAFT: A real-world few-shot text classification benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 2).
  3. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  4. Steven Bird and Edward Loper. 2004. NLTK: The natu- ral language toolkit. In Proceedings of the ACL In- teractive Poster and Demonstration Sessions, pages 214-217, Barcelona, Spain. Association for Compu- tational Linguistics.
  5. Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aha- roni, Daniel Andor, Livio Baldini Soares, Massimil- iano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question an- swering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037. · arXiv
  6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics. · DOI · Paper page
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  8. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop, pages 177-190. Springer.
  9. Yifan Deng, Xingsheng Zhang, Heyan Huang, and Yue Hu. 2023. Towards faithful dialogues via focus learn- ing. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4554-4566, Toronto, Canada. Association for Computational Linguistics. · DOI
  10. Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  11. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a. RARR: Researching and revis- ing what language models say, using language mod- els. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477-16508, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  12. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b. Enabling large language models to generate text with citations. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465-6488, Singapore. Associa- tion for Computational Linguistics.
  13. Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. TrueTeacher: Learning factual consistency evaluation with large language models. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 2053-2070, Singapore. Associa- tion for Computational Linguistics.
  14. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PMLR.
  15. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
  16. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3905-3920, Seattle, United States. Association for Computational Lin- guistics. · DOI · Paper page
  17. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  18. Jie Huang and Kevin Chen-Chuan Chang. 2024. Cita- tion: A key to building responsible and accountable large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  19. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
  20. Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nan- dan Thakur, and Jimmy Lin. 2023. Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution. arXiv preprint arXiv:2307.16883. · arXiv
  21. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4940-4957, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  22. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  23. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
  24. Nelson Liu, Tianyi Zhang, and Percy Liang. 2023. Eval- uating verifiability in generative search engines. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 7001-7025, Singapore. Association for Computational Linguistics.
  25. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. Ex- pertQA: Expert-curated questions and attributed an- swers. In 2024 Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics.
  26. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 9802-9822, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  27. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell- Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147. · arXiv
  28. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, Singa- pore. Association for Computational Linguistics.
  29. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics. · DOI · Paper page
  30. Edoardo Mosca, Ferenc Szigeti, Stella Tragianni, Daniel Gallagher, and Georg Groh. 2022. SHAP-based ex- planation methods: A review for NLP interpretabil- ity. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 4593- 4603, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  31. Benjamin Muller, John Wieting, Jonathan Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Soares, Roee Aharoni, Jonathan Herzig, and Xinyi Wang. 2023. Evaluating and modeling attribution for cross-lingual question answering. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 144-157, Singapore. Association for Computational Linguistics.
  32. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332. · arXiv
  33. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2021. Large dual encoders are generalizable retriev- ers. arXiv preprint arXiv:2112.07899. · arXiv
  34. R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2. · arXiv
  35. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: Measuring the gap be- tween neural text and human text using divergence frontiers. In Advances in Neural Information Pro- cessing Systems.
  36. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Sys- tems.
  37. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring attribution in natural lan- guage generation models. Computational Linguistics, pages 1-64.
  38. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics. · DOI · Paper page
  39. Lloyd S Shapley. 1953. A value for n-person games. In Harold W. Kuhn and Albert W. Tucker, editors, Contributions to the Theory of Games II, pages 307- 317. Princeton University Press, Princeton.
  40. Chenglei Si, Navita Goyal, Sherry Tongshuang Wu, Chen Zhao, Shi Feng, Hal Daumé III, and Jordan Boyd-Graber. 2023. Large language models help hu- mans verify truthfulness-except when they are con- vincingly wrong. arXiv preprint arXiv:2310.12558. · arXiv
  41. Julius Steen, Juri Opitz, Anette Frank, and Katja Mark- ert. 2023. With a little push, NLI models can robustly and efficiently predict faithfulness. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 914-924, Toronto, Canada. Association for Computational Linguistics. · DOI
  42. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Wei Chang. 2022. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 8273-8288, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. · DOI · Paper page
  43. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.08239. · arXiv
  44. Katherine Tian, Eric Mitchell, Huaxiu Yao, Christo- pher D Manning, and Chelsea Finn. 2023. Fine- tuning language models for factuality. arXiv preprint arXiv:2311.08401. · arXiv
  45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  46. Prasetya Utama, Joshua Bambrick, Nafise Moosavi, and Iryna Gurevych. 2022. Falsesum: Generating document-level NLI examples for recognizing fac- tual inconsistency in summarization. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2763-2776, Seattle, United States. Association for Computational Linguistics. · DOI
  47. Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. Proceed- ings of the AAAI Conference on Artificial Intelligence, 32(1). · DOI · Paper page
  48. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3225-3245, Toronto, Canada. Association for Com- putational Linguistics. · DOI · Paper page
  49. Xi Ye, Ruoxi Sun, Sercan Ö Arik, and Tomas Pfister. 2024. Effective large language model adaptation for improved grounding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  50. Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. Automatic evaluation of attri- bution by large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4615-4635, Singapore. Association for Computational Linguistics.
  51. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328-11348, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  52. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI · Paper page
  53. Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. 2023. Weaker than you think: A critical look at weakly supervised learn- ing. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14229-14253, Toronto, Canada. Association for Computational Linguistics. 10 https://huggingface.co/meta-llama/ Llama-2-7b-chat-hf 11 https://huggingface.co/Open-Orca/ Mistral-7B-OpenOrca 12 https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.1 13 https://huggingface.co/yzha/AlignScore, and their repository https://github.com/yuh-zha/ AlignScore 14 https://huggingface.co/google/t5_xxl_true_ nli_mixture 15 https://huggingface.co/juliussteen/ DeBERTa-v3-FaithAug 16 https://huggingface.co/google/t5_11b_ trueteacher_and_anli 17 https://github.com/facebookresearch/llama/ blob/main/LICENSE 18 https://github.com/princeton-nlp/ALCE 19 https://huggingface.co/datasets/miracl/hagrid 20 https://pypi.org/project/rouge-score/ 21 https://pypi.org/project/mauve-text/ Method Rouge-L Fluency Correctness Grounded Correct. Citation Recall Citaion Precision Avg length ASQA ChatGPT (Gao et al., 2023b) - 66.6 40.4 - 73.6 · DOI

Cited by

See all 3

Metadata

Authors
Aly, R., Tang, Z., Tan, S., & Karypis, G.
Venue
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2024
Pages
11876-11896
DOI
10.18653/v1/2024.acl-long.641
Cited by
3
References
3

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning to Generate Answers with Citations via Factual Consistency Models. https://app.cochat.ai/papers/10.18653/v1/2024.acl-long.641
The paper
Aly, R., Tang, Z., Tan, S., & Karypis, G. (2024). Learning to Generate Answers with Citations via Factual Consistency Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2024.acl-long.641

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.