CoChat
Sign Up
PreprintCited by 2Information Retrieval and Search BehaviorTopic ModelingRecommender Systems and Techniques

Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses

Open access
Venkit, Pranav Narayanan, Laban, Philippe, Zhou, Yilun, Mao, Yixin, Wu, Chien-Sheng·2024·arXiv
DOI 10.48550/arxiv.2410.22349arXiv 2410.22349
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Model (LLM)-based applications are graduating from research prototypes to products serving millions of users, influencing how people write and consume information. A prominent example is the appearance of Answer Engines: LLM-based generative search engines supplanting traditional search engines. Answer engines not only retrieve relevant sources to a user query but synthesize answer summaries that cite the sources. To understand these systems' limitations, we first conducted a study with 21 participants, evaluating interactions with answer vs. traditional search engines and identifying 16 answer engine limitations. From these insights, we propose 16 answer engine design recommendations, linked to 8 metrics. An automated evaluation implementing our metrics on three popular engines (You.com, Perplexity.ai, BingChat) quantifies common limitations (e.g., frequent hallucination, inaccurate citation) and unique features (e.g., variation in answer confidence), with results mirroring user study insights. We release our Answer Engine Evaluation benchmark (AEE) to facilitate transparent evaluation of LLM-based applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.22349

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 68

  1. Sabira Arefin. 2024. AI Revolutionizing Healthcare: Innovations, Challenges, and Ethical Considerations. MZ Journal of Artificial Intelligence 1, 2 (2024), 1-17.
  2. Carl Auerbach and Louise B Silverstein. 2003. Qualitative data: An introduction to coding and analysis. Vol. 21. NYU press.
  3. Emily M Bender. 2024. Resisting Dehumanization in the Age of "AI". Current Directions in Psychological Science 33, 2 (2024), 114-120.
  4. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610-623.
  5. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Lan- guage (Technology) is Power: A Critical Survey of "Bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 5454- 5476.
  6. Valerio Capraro, Austin Lentsch, Daron Acemoglu, Selin Akgun, Aisel Akhme- dova, Ennio Bilancini, Jean-François Bonnefon, Pablo Brañas-Garza, Luigi Butera, Karen M Douglas, et al. 2024. The impact of generative artificial intelligence on socioeconomic inequalities and policy making. PNAS nexus 3, 6 (2024).
  7. Kathy Charmaz. 2017. Constructivist grounded theory. The Journal of Positive Psychology 12, 3 (2017), 299-300.
  8. Pratyush Chauhan, Rahul Kumar Sahani, Soham Datta, Ali Qadir, Manish Raj, and Mohd Mohsin Ali. 2024. Evaluating Top-k RAG-based approach for Game Review Generation. In 2024 IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT), Vol. 5. IEEE, 258-263.
  9. Robert Cooper and Michael Foster. 1971. Sociotechnical systems. American Psychologist 26, 5 (1971), 467.
  10. Dipto Das. 2023. Studying Multi-dimensional Marginalization of Identity from Decolonial and Postcolonial Perspectives. In Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing. 437- 440.
  11. Dipto Das, Shion Guha, Jed R Brubaker, and Bryan Semaan. 2024. The"Colonial Impulse" of Natural Language Processing: An Audit of Bengali Sentiment Analysis Tools and Their Identity-based Biases. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-18. 11 https://github.com/SalesforceAIResearch/answer-engine-eval
  12. Dipto Das and Bryan Semaan. 2022. Decolonial and Postcolonial Computing Research: A Scientometric Exploration. In Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing. 168- 174.
  13. Sarah Dean, Thomas Krendl Gilbert, Nathan Lambert, and Tom Zick. 2021. Axes for sociotechnical inquiry in AI research. IEEE Transactions on Technology and Society 2, 2 (2021), 62-70.
  14. Yujuan Ding, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meets llms: Towards retrieval- augmented large language models. arXiv preprint arXiv:2405.06211 (2024). · arXiv
  15. Upol Ehsan, Samir Passi, Q Vera Liao, Larry Chan, I-Hsiang Lee, Michael Muller, and Mark O Riedl. 2024. The Who in XAI: How AI Background Shapes Perceptions of AI Explanations. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-32.
  16. Upol Ehsan and Mark O Riedl. 2020. Human-centered explainable ai: Towards a reflective sociotechnical approach. In HCI International 2020-Late Breaking Papers: Multimodality and Intelligence: 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19-24, 2020, Proceedings 22. Springer, 449-466.
  17. Upol Ehsan, Elizabeth A Watkins, Philipp Wintersberger, Carina Manger, Sun- nie SY Kim, Niels Van Berkel, Andreas Riener, and Mark O Riedl. 2024. Human- Centered Explainable AI (HCXAI): Reloading Explainability in the Era of Large Language Models (LLMs). In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 1-6.
  18. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 150-158.
  19. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ra- gas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023). · arXiv
  20. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6491-6501.
  21. Emilio Ferrara. 2024. GenAI against humanity: Nefarious applications of genera- tive artificial intelligence and large language models. Journal of Computational Social Science (2024), 1-21.
  22. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023). · arXiv
  23. Sanjana Gautam, Pranav Narayanan Venkit, and Sourojit Ghosh. 2024. From melting pots to misrepresentations: Exploring harms in generative ai. arXiv preprint arXiv:2403.10776 (2024). · arXiv
  24. Sourojit Ghosh, Pranav Narayanan Venkit, Sanjana Gautam, Shomir Wilson, and Aylin Caliskan. 2024. Do Generative AI Models Output Harm while Representing Non-Western Cultures: Evidence from A Community-Centered Approach. arXiv preprint arXiv:2407.14779 (2024). · arXiv
  25. Barney Glaser and Anselm Strauss. 1967. Discovery of grounded theory: Strategies for qualitative research. Routledge.
  26. Eric Goldman. 2005. Search engine bias and the demise of search engine utopi- anism. Yale JL & Tech. 8 (2005), 188.
  27. David Grant. 2025. Populism, Artificial Intelligence and Law: A New Understanding of the Dynamics of the Present. Taylor & Francis.
  28. Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. 2024. RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. arXiv preprint arXiv:2401.08406 (2024). · arXiv
  29. Vipul Gupta, Pranav Narayanan Venkit, Shomir Wilson, and Rebecca J Passon- neau. 2024. Sociodemographic bias in language models: A survey and forward path. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP). 295-322.
  30. Jutta Haider and Olof Sundin. 2019. Invisible search and online search engines: The ubiquity of search in everyday life. Taylor & Francis.
  31. Donna Haraway. 2013. Situated knowledges: The science question in feminism and the privilege of partial perspective 1. In Women, science, and technology. Routledge, 455-472.
  32. Carol Mullins Hayes. 2023. Generative artificial intelligence and copyright: Both sides of the Black Box. Available at SSRN 4517799 (2023).
  33. Wayne Holmes and Ilkka Tuomi. 2022. State of the art and practice in AI in education. European Journal of Education 57, 4 (2022), 542-570.
  34. John E Hopcroft and Richard M Karp. 1973. An nˆ5/2 algorithm for maximum matchings in bipartite graphs. SIAM Journal on computing 2, 4 (1973), 225-231.
  35. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, chal- lenges, and open questions. arXiv preprint arXiv:2311.05232 (2023). · arXiv
  36. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 7029-7043.
  37. Navreet Kaur, Monojit Choudhury, and Danish Pruthi. 2024. Evaluating Large Language Models for Health-related Queries with Presuppositions. In Findings of the Association for Computational Linguistics ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 14308-14331. https://aclanthology.org/ 2024.findings-acl.850
  38. Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Man- junatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. FABLES: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261 (2024). · arXiv
  39. Philippe Laban, Alexander R Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems. arXiv preprint arXiv:2407.01370 (2024). · arXiv
  40. Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander R Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. Llms as factual reasoners: Insights from existing benchmarks and beyond. arXiv preprint arXiv:2305.14540 (2023). · arXiv
  41. Philippe Laban, Lidiya Murakhovs' ka, Caiming Xiong, and Chien-Sheng Wu. 2023. Are you sure? challenging llms leads to performance drops in the flipflop experiment. arXiv preprint arXiv:2311.08596 (2023). · arXiv
  42. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. Sum- maC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics 10 (2022), 163-177.
  43. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via Paraphrasing. Advances in Neural Information Processing Systems 33 (2020), 18470-18481.
  44. Alice Li and Luanne Sinnamon. 2024. Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority. arXiv preprint arXiv:2405.14034 (2024). · arXiv
  45. Sebastian Lins, Konstantin D Pandl, Heiner Teigeler, Scott Thiebes, Calvin Bayer, and Ali Sunyaev. 2021. Artificial intelligence as a service: classification and research directions. Business & Information Systems Engineering 63 (2021), 441- 456.
  46. Sharon McDonald, Helen M Edwards, and Tingting Zhao. 2012. Exploring think- alouds in usability testing: An international survey. IEEE Transactions on Profes- sional Communication 55, 1 (2012), 2-19.
  47. Shahan Ali Memon and Jevin D West. 2024. Search engines post-ChatGPT: How generative artificial intelligence could make search less reliable. arXiv preprint arXiv:2402.11707 (2024). · arXiv
  48. Abbe Mowshowitz and Akira Kawaguchi. 2005. Measuring search engine bias. Information processing & management 41, 5 (2005), 1193-1205.
  49. Rainer Mühlhoff. 2018. Digitale Entmündigung und User Experience Design. Leviathan 46, 4 (2018), 551-574.
  50. Tarek Naous, Michael Joseph Ryan, and Wei Xu. 2023. Having Beer after Prayer? Measuring Cultural Bias in Large Language Models. In Annual Meeting of the Asso- ciation for Computational Linguistics. https://api.semanticscholar.org/CorpusID: 258865272
  51. Pranav Narayanan Venkit. 2023. Towards a holistic approach: Understanding sociodemographic biases in nlp models using an interdisciplinary lens. In Pro- ceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 1004-1005.
  52. Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. 2023. Unmasking nationality bias: A study of human perception of nationalities in ai-generated articles. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 554-565.
  53. Mie Nørgaard and Kasper Hornbæk. 2006. What do usability evaluators do in practice? An explorative study of think-aloud testing. In Proceedings of the 6th conference on Designing Interactive systems. 209-218.
  54. Sanjeev Pulapaka, Srinath Godavarthi, and Dr Sherry Ding. 2024. GenAI and the Public Sector. In Empowering the Public Sector with Generative AI: From Strategy and Design to Real-World Applications. Springer, 31-43.
  55. Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922 (2023). · arXiv
  56. Sujoy Roychowdhury, Sumit Soman, HG Ranjani, Neeraj Gunda, Vansh Chhabra, and Sai Krishna Bala. 2024. Evaluation of RAG Metrics for Question Answering in the Telecom Domain. arXiv preprint arXiv:2407.12873 (2024). · arXiv
  57. Eryk Salvaggio. 2024. Challenging the myths of Generative AI. https://www. techpolicy.press/challenging-the-myths-of-generative-ai/
  58. Chirag Shah and Emily M Bender. 2024. Envisioning information access systems: What makes for good tools and a healthy Web? ACM Transactions on the Web 18, 3 (2024), 1-24.
  59. Nikhil Sharma, Q Vera Liao, and Ziang Xiao. 2024. Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17.
  60. Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kalu- arachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering. Transactions of the Association for Computational Linguistics 11 (2023), 1-17.
  61. Liyan Tang, Philippe Laban, and Greg Durrett. 2024. MiniCheck: Efficient Fact- Checking of LLMs on Grounding Documents. arXiv preprint arXiv:2404.10774 (2024). · arXiv
  62. Pranav Venkit, Mukund Srinath, Sanjana Gautam, Saranya Venkatraman, Vipul Gupta, Rebecca J Passonneau, and Shomir Wilson. 2023. The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 13743-13763.
  63. Pranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs, Mukund Srinath, Koustava Goswami, Sarah Rajtmajer, and Shomir Wilson. 2024. " Confidently Nonsensical?": A Critical Survey on the Perspectives and Challenges of'Hallucinations' in NLP. arXiv preprint arXiv:2404.07461 (2024). · arXiv
  64. Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. 2023. Nationality Bias in Text Generation. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 116-122.
  65. Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. 2023. Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Models. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023). 26-34.
  66. Kevin Wu, Eric Wu, and James Zou. 2024. How faithful are RAG models? Quan- tifying the tug-of-war between RAG and LLMs' internal prior. arXiv preprint arXiv:2404.10198 (2024). · arXiv
  67. Elvin Wyly. 2014. Automated (post) positivism. Urban Geography 35, 5 (2014), 669-690.
  68. Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, et al. 2024. RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework. arXiv preprint arXiv:2408.01262 (2024). · arXiv

Cited by

Sorted by citations

Authors
Venkit, P. N., Laban, P., Zhou, Y., Mao, Y., & Wu, C.-S.
Affiliations
  • Pennsylvania State University (US)
  • Salesforce (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.22349
arXiv
2410.22349
OpenAlex
W4404342140
Cited by
2
References
1
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
referencing, hallucinations, factual accuracy, answer confidence, evaluation metrics, evaluation benchmark, large language models, generative search, user study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review