Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models
Open accessAbstract
Despite recent progress, it has been difficult to prevent semantic hallucinations in generative Large Language Models. One common solution to this is augmenting LLMs with a retrieval system and making sure that the generated output is attributable to the retrieved information. Given this new added constraint, it is plausible to expect that the overall quality of the output will be affected, for example, in terms of fluency. Can scaling language models help? Here we examine the relationship between fluency and attribution in LLMs prompted with retrieved evidence in knowledge-heavy dialog settings. Our experiments were implemented with a set of auto-metrics that are aligned with human preferences. They were used to evaluate a large set of generations, produced under varying parameters of LLMs and supplied context. We show that larger models tend to do much better in both fluency and attribution, and that (naively) using top-k retrieval versus top-1 retrieval improves attribution but hurts fluency. We next propose a recipe that could allow smaller models to both close the gap with larger models and preserve the benefits of top-k retrieval while avoiding its drawbacks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This heightened tendency to sample infrequently occurring tokens exacerbates the risk of hallucinations.
“Consequently, this heightened tendency to sample infrequently occurring tokens exacerbates the risk of hallucinations [5].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 17
- Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.T., Jin, A., Bos, T., Baker, L., Du, Y. and Li, Y., 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. · arXiv
- Adiwardana, D., Luong, M.T., So, D.R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y. and Le, Q.V., 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977. · arXiv
- Guu, K., Lee, K., Tung, Z., Pasupat, P. and Chang, M., 2020, November. Retrieval augmented language model pre- training. In International Conference on Machine Learning (pp. 3929-3938). PMLR.
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.T., Rocktäschel, T. and Riedel, S., 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33, pp.9459-9474.
- Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G.V.D., Lespiau, J.B., Damoc, B., Clark, A. and Casas, D.D.L., 2021. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426. · arXiv
- Li, Z., Guo, R. and Kumar, S., 2022. Decoupled context processing for context augmented language modeling. arXiv preprint arXiv:2210.05758. · arXiv
- Suzgun, M., Melas-Kyriazi, L. and Jurafsky, D., 2022. Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models. arXiv preprint arXiv:2205.11503. · arXiv
- Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S. and Chappidi, S., 2020. Open-domain question answering goes conversational via question rewriting. arXiv preprint arXiv:2010.04898. · arXiv
- Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D. and Chi, E.H., 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682. · arXiv
- Rashkin, H., Nikolaev, V., Lamm, M., Collins, M., Das, D., Petrov, S., Tomar, G.S., Turc, I. and Reitter, D., 2021. Measuring attribution in natural language generation models. arXiv preprint arXiv:2112.12870. · arXiv
- Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A. and Matias, Y., 2022. TRUE: Re-evaluating Factual Consistency Evaluation. arXiv preprint arXiv:2204.04991. · arXiv
- Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H. and Ba, J., 2022. Large language models are human- level prompt engineers. arXiv preprint arXiv:2211.01910. · arXiv
- Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A.T., Fan, Y., Zhao, V.Y., Lao, N., Lee, H., Juan, D.C. and Guu, K., 2022. Attributed text generation via post-hoc research and revision. arXiv preprint arXiv:2210.08726. · arXiv
- Laban, P., Schnabel, T., Bennett, P.N. and Hearst, M.A., 2021. SummaC: Re-Visiting NLI-based Models for Inconsis- tency Detection in Summarization. arXiv preprint arXiv:2111.09525. · arXiv
- Lee, N., Ping, W., Xu, P., Patwary, M., Shoeybi, M. and Catanzaro, B., 2022. Factuality Enhanced Language Models for Open-Ended Text Generation. arXiv preprint arXiv:2206.04624. · arXiv
- Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S. and Schuh, P., 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. · arXiv
- Thorne, J. and Vlachos, A., 2020. Evidence-based factual error correction. arXiv preprint arXiv:2012.15788. · arXiv
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180 · see citing works
“During inference, hallucination can also be caused by sampling randomness [1, 14], insufficient context attention [55], and softmax bottleneck [9, 73].”
BackgroundCalibrated Language Models Must Hallucinate
Kalai, A. T., & Vempala, S. S. (2024) · Proceedings of the 56th Annual ACM Symposium on Theory of Computing · doi:10.1145/3618260.3649777 · Cited by 91 · see citing works
The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S. (2026) · ACM Computing Surveys · doi:10.1145/3773080 · Cited by 58 · see citing works
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation
Daheim, N., Dziri, N., Sachan, M., Gurevych, I., & Ponti, E. M. (2023) · arXiv · doi:10.48550/arxiv.2303.17574
- Authors
- Aksitov, R., Chang, C.-C., Reitter, D., Shakeri, S., & Sung, Y.
- Affiliations
- Google (United States) (US)
- ORCID
- Reitter, David
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.05578
- OpenAlex
- W4320836790
- Cited by
- 9
- References
- 0
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- retrieval-augmented generation, large language models, semantic hallucination, attribution, fluency, top-k retrieval, model scaling, human preference alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
