CoChat
Sign Up
PreprintCited by 1Information Retrieval and Search BehaviorMisinformation and Its ImpactsEthics and Social Impacts of AI

News Source Citing Patterns in AI Search Systems

Open access
Yang, Kai-Cheng·2025·arXiv
DOI 10.48550/arxiv.2507.05301arXiv 2507.05301
Open this paper in CoChat — free accountOpen PDF

Abstract

AI-powered search systems are emerging as new information gatekeepers, fundamentally transforming how users access news and information. Despite their growing influence, the citation patterns of these systems remain poorly understood. We address this gap by analyzing data from the AI Search Arena, a head-to-head evaluation platform for AI search systems. The dataset comprises over 24,000 conversations and 65,000 responses from models across three major providers: OpenAI, Perplexity, and Google. Among the over 366,000 citations embedded in these responses, 9% reference news sources. We find that while models from different providers cite distinct news sources, they exhibit shared patterns in citation behavior. News citations concentrate heavily among a small number of outlets and display a pronounced liberal bias, though low-credibility sources are rarely cited. User preference analysis reveals that neither the political leaning nor the quality of cited news sources significantly influences user satisfaction. These findings reveal significant challenges in current AI search systems and have important implications for their design and governance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.05301

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Studies of answer engines and source-cited responses document limitations such as hallucination, inaccurate citation, and mismatch between evidence and generated claims.

    “Studies of answer engines and source-cited responses document limitations such as hallucination, inaccurate citation, and mismatch between evidence and generated claims [7,8,9].”

    From From Citation Selection to Citation Absorption: A Measurement Framework for Generative Engine Optimization Across AI Search Platforms (2026) · cites this paper as [8]

    The Sources Cited by AI Search Systems section (p. 2) reports studies finding misattributed or misrepresented sources, missing citations, and mis-summarized news citations. That supports inaccurate citation and mismatch between cited evidence and generated claims, but the passages do not establish hallucination as a separate documented limitation. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 35

  1. Bakshy, E.; Messing, S.; and Adamic, L. A. 2015. Exposure to ideologically diverse news and opinion on Facebook. Science , 348(6239): 1130-1132.
  2. Barzilai-Nahon, K. 2008. Toward a theory of network gatekeeping: A framework for exploring information control. Journal of the American Society for Information Science and Technology , 59(9): 1493-1512.
  3. Brantner, C.; Karlsson, M.; and Kuai, J. 2025. Sourcing behavior and the role of news media in AI-powered search engines in the digital media ecosystem: Comparing political news retrieval across five languages. Telecommunications Policy , 49(5): 102952.
  4. Buntain, C.; Bonneau, R.; Nagler, J.; and Tucker, J. A. 2023. Measuring the Ideology of Audiences for Web Links and Domains Using Differentially Private Engagement Data. Proceedings of the International AAAI Conference on Web and Social Media , 17(1): 72-83.
  5. Chiang, W.-L.; Zheng, L.; Sheng, Y.; Angelopoulos, A. N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M. I.; Gonzalez, J. E.; and Stoica, I. 2024. Chatbot arena: an open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning , ICML'24. JMLR.org.
  6. Cronin, J.; Clemm von Hohenberg, B.; Gonc ¸alves, J. F. F.; Menchen-Trevino, E.; and Wojcieszak, M. 2023. The (null) over-time effects of exposure to local news websites: Evidence from trace data. Journal of Information Technology & Politics , 1-15.
  7. Eady, G.; Bonneau, R.; Tucker, J. A.; and Nagler, J. 2025. News Sharing on Social Media: Mapping the Ideology of News Media, Politicians, and the Mass Public. Political Analysis , 33(2): 73-90.
  8. Fischer, S.; Jaidka, K.; and Lelkes, Y. 2020. Auditing local news presence on Google News. Nature Human Behaviour , 4(12): 1236-1244.
  9. Grootendorst, M. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794. · arXiv
  10. Horne, B. D.; Gruppi, M.; Joseph, K.; Green, J.; Wihbey, J. P.; and Adalı, S. 2022. NELA-Local: A Dataset of US Local News Articles for the Study of County-Level News Ecosystems. In Proceedings of the International AAAI Conference on Web and Social Media , volume 16, 1275-1284.
  11. Kuai, J.; Brantner, C.; Karlsson, M.; Couvering, E. V.; and Romano, S. 2025. AI chatbot accountability in the age of algorithmic gatekeeping: Comparing generative search engine political information retrieval across five languages. New Media & Society , 0(0): 14614448251321162.
  12. Lasser, J.; Aroyehun, S. T.; Simchon, A.; Carrella, F.; Garcia, D.; and Lewandowsky, S. 2022. Social media sharing by political elites: An asymmetric American exceptionalism. arXiv:2207.06313. · arXiv
  13. Le Qu´ er´ e, M. A.; Chiang, T.-W.; and Naaman, M. 2022. Understanding Local News Social Coverage and Engagement at Scale during the COVID-19 Pandemic. In Proceedings of the International AAAI Conference on Web and Social Media , volume 16, 560-572.
  14. Li, A.; and Sinnamon, L. 2024. Generative AI Search Engines as Arbiters of Public Knowledge: An Audit of Bias and Authority. Proceedings of the Association for Information Science and Technology , 61(1): 205-217.
  15. Li, H.; and Aral, S. 2025. Human Trust in AI Search: A Large-Scale Experiment. arXiv:2504.06435. · arXiv
  16. Memon, S. A.; and West, J. D. 2024. Search Engines PostChatGPT: How Generative Artificial Intelligence Could Make Search Less Reliable. arXiv:2402.11707. · arXiv
  17. Miroyan, M.; Wu, T.-H.; King, L.; Li, T.; Pan, J.; Hu, X.; Chiang, W.-L.; Angelopoulos, A. N.; Darrell, T.; Norouzi, N.; and Gonzalez, J. E. 2025. Search Arena: Analyzing Search-Augmented LLMs. arXiv:2506.05334. · arXiv
  18. Monroe, B. L.; Colaresi, M. P.; and Quinn, K. M. 2008. Fightin'words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis , 16(4): 372-403.
  19. Narayanan Venkit, P.; Laban, P.; Zhou, Y.; Mao, Y.; and Wu, C.-S. 2025. Search Engines in the AI Era: A Qualitative Understanding to the False Promise of Factual and Verifiable Source-Cited Responses in LLM-based Search. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , FAccT '25, 1325-1340. New York, NY, USA: Association for Computing Machinery. ISBN 9798400714825.
  20. Pennycook, G.; and Rand, D. G. 2019. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences , 116(7): 2521-2526.
  21. Reimers, N.; and Gurevych, I. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084. · arXiv
  22. Robertson, R. E.; Green, J.; Ruck, D. J.; Ognyanova, K.; Wilson, C.; and Lazer, D. 2023. Users choose to engage with more partisan news than they are exposed to on Google Search. Nature , 618(7964): 342-348.
  23. Robertson, R. E.; Jiang, S.; Joseph, K.; Friedland, L.; Lazer, D.; and Wilson, C. 2018. Auditing Partisan Audience Bias within Google Search. Proceedings of the ACM on HumanComputer Interaction , 2(CSCW).
  24. Shah, C.; and Bender, E. M. 2022. Situating Search. In Proceedings of the 2022 Conference on Human Information Interaction and Retrieval , CHIIR '22, 221-232. New York, NY, USA: Association for Computing Machinery. ISBN 9781450391863.
  25. Shoemaker, P. J.; and Vos, T. 2009. Gatekeeping theory . Routledge.
  26. Shu, K.; Sliva, A.; Wang, S.; Tang, J.; and Liu, H. 2017. Fake News Detection on Social Media: A Data Mining Perspective. SIGKDD Explor. Newsl. , 19(1): 22-36.
  27. Spatharioti, S. E.; Rothschild, D. M.; Goldstein, D. G.; and Hofman, J. M. 2023. Comparing Traditional and LLMbased Search for Consumer Choice: A Randomized Experiment. arXiv:2307.03744. · arXiv
  28. Suri, S.; Counts, S.; Wang, L.; Chen, C.; Wan, M.; Safavi, T.; Neville, J.; Shah, C.; White, R. W.; Andersen, R.; Buscher, G.; Manivannan, S.; Rangan, N.; and Yang, L. 2024. The Use of Generative Search Engines for Knowledge Work and Complex Tasks. arXiv:2404.04268. · arXiv
  29. Tong, C. 2025. Unite or divide? Biased search queries and Google Search results in polarized politics. In Proceedings of the 17th ACM Web Science Conference 2025 , Websci '25, 158-168. New York, NY, USA: Association for Computing Machinery. ISBN 9798400714832.
  30. Trielli, D.; and Diakopoulos, N. 2019. Search as News Curator: The Role of Google in Shaping Attention to News Information. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , CHI '19, 1-15. New York, NY, USA: Association for Computing Machinery. ISBN 9781450359702.
  31. Trielli, D.; and Diakopoulos, N. 2022. Partisan search behavior and Google results in the 2018 U.S. midterm elections. Information, Communication & Society , 25(1): 145161.
  32. Wu, Z.; Sanderson, M.; Cambazoglu, B. B.; Croft, W. B.; and Scholer, F. 2020. Providing Direct Answers in Search Results: A Study of User Behavior. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , CIKM '20, 1635-1644. New York, NY, USA: Association for Computing Machinery. ISBN 9781450368599.
  33. Xiong, H.; Bian, J.; Li, Y.; Li, X.; Du, M.; Wang, S.; Yin, D.; and Helal, S. 2024. When Search Engine Services Meet Large Language Models: Visions and Challenges. IEEE Transactions on Services Computing , 17(6): 4558-4577.
  34. Yang, K.-C.; Goel, P.; Quintana-Math´ e, A.; Horgan, L.; McCabe, S. D.; Grinberg, N.; Joseph, K.; and Lazer, D. 2025. DomainDemo: a dataset of domain-sharing activities among different demographic groups on Twitter. arXiv:2501.09035. · arXiv
  35. Yang, K.-C.; and Menczer, F. 2025. Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models. In Proceedings of the 17th ACM Web Science Conference , Websci '25, 127-137. New York, NY, USA: Association for Computing Machinery.

Cited by

Sorted by citations

Authors
Yang, K.-C.
Corresponding author
Yang, Kai-Cheng
ORCID
Yang, Kai-Cheng
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.05301
arXiv
2507.05301
OpenAlex
W4416059753
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
citation behavior, information gatekeeping, liberal bias, source credibility, user satisfaction, political bias, large language models, search result quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review