Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 15

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority

Li, Alice, Sinnamon, Luanne · 2024 · Proceedings of the Association for Information Science and Technology

Abstract

This paper reports on an audit study of generative AI systems (ChatGPT, Bing Chat, and Perplexity), which investigates how these new search engines construct responses and establish authority for topics of public importance. We collected system responses using a set of 48 authentic queries for 4 topics over 7 days, and analyzed the data using sentiment analysis, inductive coding, and source classification. Results provide an overview of the nature of system responses across these systems, and provide evidence of sentiment bias based on the queries and topics, and commercial and geographic biases in sources. The quality of sources used to support claims is uneven, relying heavily on News and Media, Business and Digital Media websites. Implications for system users emphasize the need to critically examine Generative AI system outputs when making decisions related to public interest and personal well‐being.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/pra2.1021

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 15 references

Research works printed but not linked: 41

  1. 10.17705/1jais.00458 · doi:10.17705/1jais.00458
  2. 10.1145/3406522.3446023 · DOI · Paper page
  3. 10.1016/j.gloenvcha.2003.10.001 · DOI · Paper page
  4. Bubeck S. Chandrasekaran V. Eldan R. Gehrke J. Horvitz E. Kamar E. …Zhang Y.(2023).Sparks of artificial general intelligence: Early experiments with gpt‐4. arXiv preprint.https://doi.org/10.48550/arXiv.2303.12712 · DOI · Paper page
  5. Capra R. &Arguello J.(2023).How does AI chat change search behaviors?arXiv preprint.https://doi.org/10.48550/arXiv.2307.03826 · DOI
  6. 10.1145/1863879.1863883 · DOI
  7. 10.3389/fmed.2020.00400 · DOI
  8. 10.4324/9780429448546 · DOI
  9. 10.1038/s41586-020-2649-2 · DOI · Paper page
  10. Hovland C., 1953, Communication and persuasion
  11. 10.1145/3392854 · DOI · Paper page
  12. 10.1145/3331184.3331314 · DOI
  13. 10.1002/pra2.927 · DOI
  14. 10.4135/9781071878781.n13 · doi:10.4135/9781071878781.n13
  15. 10.1007/978-3-031-09316-6_5 · DOI
  16. Li A. &Sinnamon L.(2023 August 28–September 2).Examining query sentiment bias effects on search results in large language models.The Symposium on Future Directions in Information Access (FDIA) co‐located with the 2023 European Summer School on Information Retrieval (ESSIR).https://2023.essir.eu/FDIA/papers/FDIA_2023_paper_2.pdf
  17. Li D. Sun Z. Hu X. Liu Z. Chen Z. Hu B. …Zhang M.(2023).A survey of large language Models attribution. arXiv preprint.https://doi.org/10.48550/arXiv.2311.03731 · DOI
  18. 10.1177/20539517231191528 · DOI
  19. McKinney W.(2010).Data structures for statistical computing in Python.Proceedings of the Python in Science Conferences.https://doi.org/10.25080/majora-92bf1922-00a · DOI · Paper page
  20. Mehdi Y., 2023, Confirmed: the new Bing runs on OpenAI's GPT‐4
  21. 10.1145/3449100 · DOI
  22. 10.1561/1100000083 · DOI · Paper page
  23. 10.18574/nyu/9781479833641.001.0001 · DOI · Paper page
  24. 10.1177/160940691989922 · doi:10.1177/160940691989922
  25. 10.1002/asi.10017 · DOI · Paper page
  26. 10.1145/3627508.3638345 · DOI
  27. 10.1108/JD-12-2018-0209 · DOI · Paper page
  28. Sandvig C., 2014, Auditing algorithms: Research methods for detecting discrimination on internet platforms, Data and discrimination: converting critical concerns into productive inquiry, 22, 4349
  29. 10.1145/3498366.3505816 · DOI
  30. Shi X. Liu J. Liu Y. Cheng Q. &Lu W.(2023).Know where to go: Make LLM a relevant responsible and trustworthy searcher. arXiv preprint.https://doi.org/10.48550/arXiv.2310.12443 · DOI
  31. Smith M. S., 2024, Perplexity.ai revamps Google SEO model for LLM era > AI search leader mixes Meta‐built smarts with scrappy startup fervor, IEEE Spectrum
  32. 10.1002/asi.20652 · DOI · Paper page
  33. 10.1108/00220411111164709 · DOI
  34. Taraborelli D., 2008, Proceeding of the 2008 Conference on Current Issues in Computing and Philosophy, 194
  35. 10.1177/1098214005283748 · DOI · Paper page
  36. 10.1609/aaai.v37i11.26599 · DOI
  37. Urman A. &Makhortykh M.(2023).The silence of the LLMs: Cross‐lingual analysis of political bias and false information prevalence in ChatGPT Google Bard and Bing Chat. OSF Preprints.https://doi.org/10.31219/osf.io/q9v8f · DOI
  38. 10.1145/3627508.3638305 · DOI
  39. 10.1145/2484028.2484053 · DOI · Paper page
  40. Wilson P., 1983, Second‐hand knowledge: An inquiry into cognitive authority
  41. 10.1145/3627508.3638316 · DOI

Cited by

See all 15

Metadata

Authors
Li, A., & Sinnamon, L.
Affiliations
The University of British Columbia Canada
Venue
Proceedings of the Association for Information Science and Technology
Publisher
Wiley
Published
2024-10
Volume
61
Issue
1
Pages
205-217
ISSN
2373-9231
Cited by
15
References
15
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority. https://app.cochat.ai/papers/10.1002/pra2.1021
The paper
Li, A., & Sinnamon, L. (2024). Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority. Proceedings of the Association for Information Science and Technology. https://doi.org/10.1002/pra2.1021

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.