One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations
Open accessAbstract
As Large Language Models (LLMs) are nondeterministic, the same input can generate different outputs, some of which may be incorrect or hallucinated. If run again, the LLM may correct itself and produce the correct answer. Unfortunately, most LLM-powered systems resort to single results which, correct or not, users accept. Having the LLM produce multiple outputs may help identify disagreements or alternatives. However, it is not obvious how the user will interpret conflicts or inconsistencies. To this end, we investigate how users perceive the AI model and comprehend the generated information when they receive multiple, potentially inconsistent, outputs. Through a preliminary study, we identified five types of output inconsistencies. Based on these categories, we conducted a study (N = 252) in which participants were given one or more LLM-generated passages to an information-seeking question. We found that inconsistency within multiple LLM-generated outputs lowered the participants’ perceived AI capacity, while also increasing their comprehension of the given information. Specifically, we observed that this positive effect of inconsistencies was most significant for participants who read two passages, compared to those who read three. Based on these findings, we present design implications that, instead of regarding LLM output inconsistencies as a drawback, we can reveal the potential inconsistencies to transparently indicate the limitations of these models and promote critical LLM usage.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM-based writing assistants have not reached the level of proficiency to be used without oversight.
“LLM-based writing assistants have also not reached the level of proficiency to be used without oversight [46, 70].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [46]
The Introduction (p. 1) warns that users may accept incorrect LLM responses, and the Conclusion (p. 10) recommends critical examination of generated outputs. This supports the need for oversight, but the study concerns information-seeking rather than the proficiency of LLM-based writing assistants specifically. (LLM, conf=0.84)
Experimental — accuracy not yet measuredExisting work on evaluating figure captions shows that people over-trust AI-generated information while AI can produce inaccurate responses.
“Existing work on evaluating figure captions shows that people over-trust AI-generated information [49] while at the same time, AI can produce inaccurate responses [46, 70].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1× · see citing works
“…hypothesis-driven (i.e., suggests evidence for decisions rather than suggesting decisions) [49], etc. Studies found that these explanations improve people’s understanding of the model [12, 37, 38, 43, 61] or enhance subjective perception of the AI and tendency to follow its suggestions [36, 54].”
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124 · see citing works
Question Asking During Tutoring
Graesser, A. C., & Person, N. K. (1994) · American Educational Research Journal · doi:10.3102/00028312031001104 · Cited by 900 · see citing works
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · see citing works
(2011) · Cited by 725 · see citing works
Wiśniewski, P., & Lu, Y. (2010) · Computers in Human Behavior · doi:10.1016/j.chb.2010.03.008 · Cited by 701 · see citing works
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · see citing works
<i>Why and why not</i> explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · see citing works
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · see citing works
KILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.200 · Cited by 357 · see citing works
Fair Statistical Communication in HCI
Dragicevic, P. (2016) · Human-computer interaction series · doi:10.1007/978-3-319-26633-6_13 · Cited by 339 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · see citing works
<scp>SummaC</scp>: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238 · see citing works
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220 · see citing works
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
Conceptual Metaphors Impact Perceptions of Human-AI Collaboration
Khadpe, P., Krishna, R., Fei-Fei, L., Hancock, J. T., & Bernstein, M. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415234 · Cited by 183 · see citing works
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172 · see citing works
Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
Suh, S., Min, B., Palani, S., & Xia, H. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606756 · Cited by 151 · see citing works
Panigutti, C., Beretta, A., Giannotti, F., & Pedreschi, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502104 · Cited by 145 · see citing works
A Taxonomy for Software Voting Algorithms Used in Safety-Critical Systems
Latif-Shabgahi, G. R., Bass, J. M., & Bennett, S. (2004) · IEEE Transactions on Reliability · doi:10.1109/tr.2004.832819 · Cited by 138 · see citing works
Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127 · see citing works
Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging
Fogliato, R., Chappidi, S., Lungren, M. P., Fisher, P., Wilson, D., Fitzke, M., Parkinson, M., Horvitz, E., Inkpen, K. M., & Nushi, B. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533193 · Cited by 115 · see citing works
Nourani, M., Kabir, S., Mohseni, S., & Ragan, E. D. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5284 · Cited by 104 · see citing works
You Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85 · see citing works
Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00494 · Cited by 72 · see citing works
Reducing conversational agents' overconfidence through linguistic calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2020) · arXiv · doi:10.48550/arxiv.2012.14983 · Cited by 72 · see citing works
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering
Asai, A., & Hajishirzi, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.499 · Cited by 66 · see citing works
Evaluating the Effects of Displaying Uncertainty in Context-Aware Applications
Antifakos, S., Schwaninger, A., & Schiele, B. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30119-6_4 · Cited by 50 · see citing works
Leiser, F., Eckhardt, S., Knaeble, M., Maedche, A., Schwabe, G., & Sunyaev, A. (2023) · Mensch und Computer 2023 · doi:10.1145/3603555.3603565 · Cited by 44 · see citing works
texSketch: Active Diagramming through Pen-and-Ink Annotations
Subramonyam, H., Seifert, C. M., Shah, P., & Adar, E. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376155 · Cited by 43 · see citing works
Self-citationHumans, AI, and Context: Understanding End-Users’ Trust in a Real-World Computer Vision Application
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593978 · Cited by 37 · see citing works
DAPIE: Interactive Step-by-Step Explanatory Dialogues to Answer Children’s Why and How Questions
Lee, Y., Kim, T. S., Kim, S., Yun, Y., & Kim, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581369 · Cited by 35 · see citing works
Self-citationAn Exploration of Location Error Estimation
Dearman, D., Varshavsky, A., de Lara, E., & Truong, K. N. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74853-3_11 · Cited by 28 · see citing works
How Accurate Does It Feel? – Human Perception of Different Types of Classification Mistakes
Papenmeier, A., Kern, D., Hienert, D., Kammerer, Y., & Seifert, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501915 · Cited by 25 · see citing works
Radensky, M., Séguin, J. A., Lim, J. S., Olson, K., & Geiger, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594043 · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 36
- [2][2] Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Francis Christiano, John Schulman, and Dandelion Mané. 2016. Concrete Problems in AI Safety. ArXiv abs/1606.06565 (2016). https://api.semanticscholar.org/CorpusID:10242377 - · arXiv
- [5][5] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S. Weld, Walter S. Lasecki, and Eric Horvitz. 2019. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. In AAAI Conference on Artificial Intelligence . https://api.semanticscholar.org/CorpusID:53997192 -
- [11][11] Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. 2023. RELIC: Investigating Large Language Model Responses using Self-Consistency. ArXiv abs/2311.16842 (2023). https://api. semanticscholar.org/CorpusID:265466244 - · arXiv
- [10][10] Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2019. Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations. In Annual Meeting of the Association for Computational Linguistics . https://api.semanticscholar.org/CorpusID: 203905467 -
- [14][14] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. ArXiv abs/1803.05457 (2018). https://api.semanticscholar.org/CorpusID:3922816 - · arXiv
- [13][13] Avishek Choudhury and Hamid Shamszare. 2023. Investigating the Impact of User Trust on the Adoption and Use of ChatGPT: Survey Analysis. Journal of Medical Internet Research 25 (2023). https://api.semanticscholar.org/CorpusID:258922988 -
- [15][15] Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. LM vs LM: Detecting Factual Errors via Cross Examination. In Conference on Empirical Methods in Natural Language Processing . https://api.semanticscholar.org/CorpusID: 258833288 -
- [16][16] Lynne M. Connelly. 2013. Grounded theory. Medsurg nursing : official journal of the Academy of Medical-Surgical Nurses 22 2 (2013), 124, 127. -
- [20][20] Alexander R. Fabbri, Chien Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021. QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization. In North American Chapter of the Association for Computational Linguistics . https://api.semanticscholar.org/CorpusID:245218667 -
- [24][24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2020. Measuring Massive Multitask Language Understanding. ArXiv abs/2009.03300 (2020). https://api. semanticscholar.org/CorpusID:221516475 - · arXiv
- [26][26] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ArXiv abs/2311.05232 (2023). https: - · arXiv
- [25][25] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2021. Unsolved Problems in ML Safety. ArXiv abs/2109.13916 (2021). https://api. semanticscholar.org/CorpusID:238198240 [//api.semanticscholar.org/CorpusID:265067168](https://api.semanticscholar.org/CorpusID:265067168) - · arXiv
- [27][27] Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2021. Accurate, yet inconsistent? Consistency Analysis on Language Understanding Models. ArXiv abs/2108.06665 (2021). https://api.semanticscholar.org/CorpusID:237091303 - · arXiv
- [29][29] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ArXiv abs/1705.03551 (2017). https://api.semanticscholar.org/CorpusID: 26501419 - · arXiv
- [33][33] Lars Krupp, Steffen Steinert, Maximilian Kiefer-Emmanouilidis, Karina E. Avila, Paul Lukowicz, Jochen Kuhn, Stefan Küchemann, and Jakob Karolus. 2023. Challenges and Opportunities of Moderating Usage of Large Language Models in Education. ArXiv abs/2312.14969 (2023). https://api.semanticscholar.org/CorpusID: 266550959 - · arXiv
- [42][42] Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024. The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models. https://api.semanticscholar. org/CorpusID:266844012 -
- [44][44] Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Annual Meeting of the Association for Computational Linguistics . https://api.semanticscholar.org/CorpusID:237532606 -
- [45][45] Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. Trans. Mach. Learn. Res. 2022 (2022). https: //api.semanticscholar.org/CorpusID:249191391 -
- [47][47] Erwin Marsi and Ferdi Van Rooden. 2007. Expressing uncertainty with a talking head in a multimodal question-answering system. https://api.semanticscholar. org/CorpusID:2482651 -
- [46][46] Diane M Mackie. 1987. Systematic and nonsystematic processing of majority and minority persuasive communications. Journal of Personality and Social Psychology 53, 1 (1987), 41. -
- [50][50] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. ArXiv abs/2305.14251 (2023). https://api.semanticscholar.org/CorpusID: 258841470 - · arXiv
- [56][56] Peter S. Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. 2023. AI Deception: A Survey of Examples, Risks, and Potential Solutions. arXiv:cs.CY/2308.14752 -
- [59][59] Abhilasha Ravichander, Eduard H. Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT. In STARSEM . https: //api.semanticscholar.org/CorpusID:227230677 -
- [60][60] Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are Red Roses Red? Evaluating Consistency of Question-Answering Models. In Annual Meeting of the Association for Computational Linguistics . https://api.semanticscholar.org/ CorpusID:196182403 -
- [62][62] Kyoungwon Seo, Joice Tang, Ido Roll, Sidney Fels, and Dongwook Yoon. 2021. The impact of artificial intelligence on learner-instructor interaction in online learning. International journal of educational technology in higher education 18, 1 (2021), 1-23. -
- [61][61] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: HighPrecision Model-Agnostic Explanations. In AAAI Conference on Artificial Intelligence . https://api.semanticscholar.org/CorpusID:3366554 -
- [63][63] Sofia Eleni Spatharioti, David M. Rothschild, Daniel G. Goldstein, and Jake M. Hofman. 2023. Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment. ArXiv abs/2307.03744 (2023). https://api. semanticscholar.org/CorpusID:259375527 - · arXiv
- [66][66] Meiqi Sun, Wilson Yan, P. Abbeel, and Igor Mordatch. 2022. Quantifying Uncertainty in Foundation Models via Ensembles. https://api.semanticscholar.org/ CorpusID:254183614 -
- [68][68] Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. ArXiv abs/2004.04228 (2020). https://api.semanticscholar.org/CorpusID:215548661 - · arXiv
- [67][67] Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023. MedHALT: Medical Domain Hallucination Test for Large Language Models. In Conference on Computational Natural Language Learning . https://api.semanticscholar. org/CorpusID:260316324 -
- [69][69] Lu Wang, Max Song, Rezvaneh Rezapour, Bum Chul Kwon, and Jina Huh-Yoo. 2023. People's Perceptions Toward Bias and Related Concepts in Large Language Models: A Systematic Review. ArXiv abs/2309.14504 (2023). https: //api.semanticscholar.org/CorpusID:262825989 - · arXiv
- [71][71] Justin D. Weisz, Michael J. Muller, Jessica He, and Stephanie Houde. 2023. Toward General Design Principles for Generative AI Applications 130-144. In IUI Workshops . https://api.semanticscholar.org/CorpusID:255825625 -
- [70][70] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ArXiv abs/2203.11171 (2022). https://api.semanticscholar.org/ CorpusID:247595263 - · arXiv
- [72][72] Ikedinachi Ayodele Power Wogu, Sanjay Misra, Esther Fadeke Olu-Owolabi, Patrick A Assibong, Oluwakemei D Udoh, SO Ogiri, and R Damasevicius. 2018. Artificial intelligence, artificial teachers and the fate of learners in the 21st century education sector: Implications for theory and practice. International Journal of Pure and Applied Mathematics 119, 16 (2018), 2245-2259. -
- [77][77] Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the Grey Area: Expressions of Overconfidence and Uncertainty in Language Models. ArXiv abs/2302.13439 (2023). https://api.semanticscholar.org/CorpusID:257220189 - · arXiv
- [76][76] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A Survey of Large Language Models. arXiv:cs.CL/2303.18223
Cited by
Sorted by citations
Buele, J., & Llerena-Aguirre, L. A. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1603763 · Cited by 64 · see citing works
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58 · see citing works
“These findings, along with prior findings on other unreliability cues (e.g., inconsistencies between multiple responses [71, 109]), suggest that guiding people’s attention to these cues can be an effective approach to reducing overreliance.”
InfluentialRapp, A., Lodovico, C. D., & Di, L. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103471 · Cited by 49 · see citing works
The bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15 · see citing works
A Review of Prompt Engineering Techniques for Large Language Models
Naser, M. Z. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2607553 · Cited by 17 · see citing works
LLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 10 · see citing works
“Generally available generative technology is new, and prior work has identified the randomness in LLM-generated text as a point of confusion for users [55, 81, 77, 43].”
Leveraging Approximate Caching for Faster Retrieval-Augmented Generation
Bergman, S. A., Zhang, J., Kermarrec, A.-M., Petrescu, D., Pires, R. P., Randl, M., & de Vos, M. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721941 · Cited by 10 · see citing works
Efficient Federated Search for Retrieval-Augmented Generation
Guerraoui, R., Kermarrec, A.-M., Petrescu, D., Pires, R. P., Randl, M., & de Vos, M. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721942 · Cited by 10 · see citing works
Galindez-Acosta, J. S., & Giraldo-Huertas, J. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101024 · Cited by 10 · see citing works
Smith, A. J., Liebrenz, M., Bhugra, D., Graña, J., Schleifer, R., & Buadze, A. (2025) · International Journal of Social Psychiatry · doi:10.1177/00207640251358071 · Cited by 6 · see citing works
Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions
Chen, M., Iyer, A., & Pavel, A. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746393 · Cited by 6 · see citing works
Li, T. W., Song, Y., Sundaram, H., & Karahalios, K. G. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729550 · Cited by 5 · see citing works
Kim, D. H., Jeong, D., Yadgarova, S., Shin, H., Son, J., Subramonyam, H., & Kim, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714044 · Cited by 4 · see citing works
Self-citationSQUIRE: Interactive UI Authoring via Slot QUery Intermediate REpresentations
Leung, A., Cheng, R., Wu, J., Nichols, J., & Barik, T. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747672 · Cited by 4 · see citing works
Shin, D.-H., Chen, T.-Y., Hsieh, G., & Wang, L. L. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735686 · Cited by 5 · see citing works
Dong, Y., Zhang, T., & Tian, X. (2026) · Journal of Hospitality and Tourism Technology · doi:10.1108/jhtt-11-2025-1080 · Cited by 7 · see citing works
Human–AI Co-Agency in Peer-Based Collaborative Pedagogies for Education 5.0
Ijiri, A., & Marette, C. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1195-1.ch007 · Cited by 3 · see citing works
Generative AI for Research Data Processing: Lessons Learnt From Three Use Cases
Mitra, M., de Vos, M. G., Cortinovis, N., & Ometto, D. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678704 · Cited by 3 · see citing works
Pal, O., Duddu, V., Goyal, A., Goel, D., & Saha, K. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799386 · Cited by 2 · see citing works
Palmieri, J., Lippi, M., & Marino, A. (2026) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2026.105480 · Cited by 2 · see citing works
Raees, M., Iwamoto, Y., Papangelis, K., Heard, J., & Tigwell, G. W. (2026) · arXiv · doi:10.48550/arxiv.2602.08937 · Cited by 1 · see citing works
“Where HAI-Text occasionally missed factual details, participants noted that it was not clearly read/evaluated by human authors.”
Iwamoto, Y., Raees, M., Heard, J., & Tigwell, G. W. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3759763 · Cited by 1 · see citing works
Performance of Large Language Models on the Radiation and Cancer Biology Practice Exam
Bertschmann, J., Xu, Y., Bayley, C., Abdellatif, A., & Lee, S. L. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.07.1347 · Cited by 1 · see citing works
Perceptions of the Fairness Impacts of Multiplicity in Machine Learning
Meyer, A. P., Kim, Y.-S., D'Antoni, L., & Albarghouthi, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713524 · Cited by 1 · see citing works
Prasad, A., Shalmani, S. A., He, L., Wang, Y., & McRoy, S. (2025) · BioMedInformatics · doi:10.3390/biomedinformatics5030040 · Cited by 1 · see citing works
Stark, C., Stanzilis, K., Plauche, S., & Palacios, A. F. (2025) · Families in Society The Journal of Contemporary Social Services · doi:10.1177/10443894251378612 · Cited by 1 · see citing works
Beyond Anthropomorphism: a Spectrum of Interface Metaphors for LLMs
So, J., Cheng, C., & Murthy, S. K. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798591 · Cited by 1 · see citing works
Dhanorkar, S., Passi, S., & Vorvoreanu, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812402 · Cited by 3 · see citing works
The Death of Cognitive Conflict? AI, Learning, and the Case for Disruptive Pedagogy
Loreto, I. D. (2025) · Interaction design & architecture(s)/ID&A Interaction design & architecture(s) · doi:10.55612/s-5002-067-001 · Cited by 1 · see citing works
HUMAID: A THEORY OF HUMAN MULTI-AI AGENT INTERACTION DYNAMICSPreprint
Jia, S., Shan, G., & Leidner, D. E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6650059 · Cited by 1 · see citing works
Interacting with AI by Manipulating Intents
Kim, T. S. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3707611
Self-citationTransforming Digital Access in Parliament: Topic Modeling Meets Retrieval Augmented Generation
Matoshi, V., Singh, S. S., Kucera, J., Meyer, P., & Gygli, M. (2025) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg65568.2025.11081627
Evaluating Structural Preprocessing in RAG for Academic Curriculum Applications
İntepe, G., Liang, Y., & Park, L. A. F. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-6786-7_33
Inconsistency Matters: How LLMs Struggle with Identifying Online Hate Speech and Its Consequences
Ye, H., & Du, W. (2025) · International Conference on Computers and Artificial Intelligence Technology (CAIT) · doi:10.1109/cait68620.2025.11424850
Sensemaking in Multi-Agent LLM Interfaces: How Users Interpret Transparency and Trustworthiness Cues
Pareek, S., Govers, J., Kollerup, N. K., Wong, E., Velloso, E., & Gonçalves, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791157
Auditing LLM Responses in a Complex Policy Landscape: Abortion Law in the United States
Encarnación, R., Jones, C., & Metaxa, D. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806451
Personal Data Accuracy is a Blind Spot in LLM Privacy Research - Opinion Paper
Magnussen, K., & Böhme, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35899-8_2
Bhojan, A., & Li, T. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39106-3_15
- Authors
- Lee, Y., Son, K., Kim, T. S., Kim, J., Chung, J. J. Y., Adar, E., & Kim, J.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Georgia Institute of Technology (US)
- Midjourney, United States of America
- University of Michigan (US)
- Venue
- The 2024 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2024-06-03
- Pages
- 2518-2531
- OpenAlex
- W4396822175
- Cited by
- 36
- References
- 36
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 2
- FWCI
- 13.71
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Ministry of Science and ICT, Republic of Korea (2021-0-01347)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- large language models, hallucinations, user understanding, information seeking, AI transparency, critical use of AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
