The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority
Abstract
This paper reports on an audit study of generative AI systems (ChatGPT, Bing Chat, and Perplexity), which investigates how these new search engines construct responses and establish authority for topics of public importance. We collected system responses using a set of 48 authentic queries for 4 topics over 7 days, and analyzed the data using sentiment analysis, inductive coding, and source classification. Results provide an overview of the nature of system responses across these systems, and provide evidence of sentiment bias based on the queries and topics, and commercial and geographic biases in sources. The quality of sources used to support claims is uneven, relying heavily on News and Media, Business and Digital Media websites. Implications for system users emphasize the need to critically examine Generative AI system outputs when making decisions related to public interest and personal well‐being.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Similar behaviors have been documented across other retrieval-enhanced models, which consistently return content with clean metadata structure and high indexability (Li & Sinnamon, 2024).”
From Understanding Generative AI-mediated User Engagement with Academic Library Resources (2026) · cites this paper as Li & Sinnamon, 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 5×
“Such techniques pose the danger that bad actors may take advantage of these outputs as LLMs are not subject to truth or ethics (Bender et al., 2021).”
InfluentialSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“LLMs are extremely powerful but prone to errors: generation of convincing misinformation (hallucination) (Bubeck et al., 2023; Ji et al., 2023; White, 2024), perpetuation and amplification of biases acquired through the model training process (Bender et al., 2021; White, 2024), dissemination of…”
Kincaid, J. P., Fishburne, Jr., Robert P., R., Richard L., C., & Brad, S. (1975) · doi:10.21236/ada006655 · Cited by 3,497
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Friedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203 · cited 2×
“Bias in SEs is any type of systematic skew that causes harm (Friedman & Nissenbaum, 1996).”
Hilligoss, B., & Rieh, S. Y. (2008) · Information Processing & Management · doi:10.1016/j.ipm.2007.10.001 · Cited by 611
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · cited 2×
“Many types and sources of bias influence SEs, including those that arise from system design and user interactions (Baeza-Yates, 2018; Friedman & Nissenbaum, 1996; Kay et al., 2015).”
Baeza-Yates, R. (2018) · Communications of the ACM · doi:10.1145/3209581 · Cited by 432 · cited 2×
“Many types and sources of bias influence SEs, including those that arise from system design and user interactions (Baeza-Yates, 2018; Friedman & Nissenbaum, 1996; Kay et al., 2015).”
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143
Research works printed but not linked: 41
- 10.17705/1jais.00458 · doi:10.17705/1jais.00458
- 10.1145/3406522.3446023 · DOI · Paper page
- 10.1016/j.gloenvcha.2003.10.001 · DOI · Paper page
- Bubeck S. Chandrasekaran V. Eldan R. Gehrke J. Horvitz E. Kamar E. …Zhang Y.(2023).Sparks of artificial general intelligence: Early experiments with gpt‐4. arXiv preprint.https://doi.org/10.48550/arXiv.2303.12712 · DOI · Paper page
- Capra R. &Arguello J.(2023).How does AI chat change search behaviors?arXiv preprint.https://doi.org/10.48550/arXiv.2307.03826 · DOI
- 10.1145/1863879.1863883 · DOI
- 10.3389/fmed.2020.00400 · DOI
- 10.4324/9780429448546 · DOI
- 10.1038/s41586-020-2649-2 · DOI · Paper page
- Hovland C., 1953, Communication and persuasion
- 10.1145/3392854 · DOI · Paper page
- 10.1145/3331184.3331314 · DOI
- 10.1002/pra2.927 · DOI
- 10.4135/9781071878781.n13 · doi:10.4135/9781071878781.n13
- 10.1007/978-3-031-09316-6_5 · DOI
- Li A. &Sinnamon L.(2023 August 28–September 2).Examining query sentiment bias effects on search results in large language models.The Symposium on Future Directions in Information Access (FDIA) co‐located with the 2023 European Summer School on Information Retrieval (ESSIR).https://2023.essir.eu/FDIA/papers/FDIA_2023_paper_2.pdf
- Li D. Sun Z. Hu X. Liu Z. Chen Z. Hu B. …Zhang M.(2023).A survey of large language Models attribution. arXiv preprint.https://doi.org/10.48550/arXiv.2311.03731 · DOI
- 10.1177/20539517231191528 · DOI
- McKinney W.(2010).Data structures for statistical computing in Python.Proceedings of the Python in Science Conferences.https://doi.org/10.25080/majora-92bf1922-00a · DOI · Paper page
- Mehdi Y., 2023, Confirmed: the new Bing runs on OpenAI's GPT‐4
- 10.1145/3449100 · DOI
- 10.1561/1100000083 · DOI · Paper page
- 10.18574/nyu/9781479833641.001.0001 · DOI · Paper page
- 10.1177/160940691989922 · doi:10.1177/160940691989922
- 10.1002/asi.10017 · DOI · Paper page
- 10.1145/3627508.3638345 · DOI
- 10.1108/JD-12-2018-0209 · DOI · Paper page
- Sandvig C., 2014, Auditing algorithms: Research methods for detecting discrimination on internet platforms, Data and discrimination: converting critical concerns into productive inquiry, 22, 4349
- 10.1145/3498366.3505816 · DOI
- Shi X. Liu J. Liu Y. Cheng Q. &Lu W.(2023).Know where to go: Make LLM a relevant responsible and trustworthy searcher. arXiv preprint.https://doi.org/10.48550/arXiv.2310.12443 · DOI
- Smith M. S., 2024, Perplexity.ai revamps Google SEO model for LLM era > AI search leader mixes Meta‐built smarts with scrappy startup fervor, IEEE Spectrum
- 10.1002/asi.20652 · DOI · Paper page
- 10.1108/00220411111164709 · DOI
- Taraborelli D., 2008, Proceeding of the 2008 Conference on Current Issues in Computing and Philosophy, 194
- 10.1177/1098214005283748 · DOI · Paper page
- 10.1609/aaai.v37i11.26599 · DOI
- Urman A. &Makhortykh M.(2023).The silence of the LLMs: Cross‐lingual analysis of political bias and false information prevalence in ChatGPT Google Bard and Bing Chat. OSF Preprints.https://doi.org/10.31219/osf.io/q9v8f · DOI
- 10.1145/3627508.3638305 · DOI
- 10.1145/2484028.2484053 · DOI · Paper page
- Wilson P., 1983, Second‐hand knowledge: An inquiry into cognitive authority
- 10.1145/3627508.3638316 · DOI
Cited by
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2025) · Proceedings of the 17th ACM Web Science Conference 2025 · doi:10.1145/3717867.3717903 · Cited by 81
The impact of generative AI search result design on user perceptions and information adoption
Li, A. (2026) · Proceedings of the 2026 Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787937
Self-citationUnderstanding Generative AI-mediated User Engagement with Academic Library Resources
Kim, H. M., & Stanislaw, S. (2026) · arXiv · doi:10.48550/arxiv.2607.20328
Metadata
- Authors
- Li, A., & Sinnamon, L.
- Affiliations
- The University of British Columbia Canada
- Venue
- Proceedings of the Association for Information Science and Technology
- Publisher
- Wiley
- Published
- 2024-10
- Volume
- 61
- Issue
- 1
- Pages
- 205-217
- ISSN
- 2373-9231
- Cited by
- 15
- References
- 15
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority. https://app.cochat.ai/papers/10.1002/pra2.1021
- The paper
- Li, A., & Sinnamon, L. (2024). Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority. Proceedings of the Association for Information Science and Technology. https://doi.org/10.1002/pra2.1021
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.