Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Open accessPublished version of a preprint
Abstract
Search engines increasingly leverage large language models (LLMs) to generate direct answers, and AI chatbots now access the Internet for fresh data.As information curators for billions of users, LLMs must assess the accuracy and reliability of different sources.This paper audits nine widely used LLMs from three leading providers-OpenAI, Google, and Meta-to evaluate their ability to discern credible and high-quality information sources from low-credibility ones.We find that while LLMs can rate most tested news outlets, larger models more frequently refuse to provide ratings due to insufficient information, whereas smaller models are more prone to making errors in their ratings.For sources where ratings are provided, LLMs exhibit a high level of agreement among themselves (average Spearman's 𝜌 = 0.79), but their ratings align only moderately with human expert evaluations (average 𝜌 = 0.50).Analyzing news sources with different political leanings in the US, we observe a liberal bias in credibility ratings yielded by all LLMs in default configurations.Additionally, assigning partisan roles to LLMs consistently induces strong politically congruent bias in their ratings.These findings have important implications for the use of LLMs in curating news and political information.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 2× · see citing works
“Despite such advantages, the additional LLM components could introduce new problems (Memon and West 2024) because language models face critical technical challenges, such as hallucinations (Ji et al. 2023), and exhibit various biases (Gallegos et al. 2024).”
The echo chamber effect on social media
Cinelli, M., De Francisci Morales, G., Galeazzi, A., Quattrociocchi, W., & Starnini, M. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2023301118 · Cited by 2,092 · cited 2× · see citing works
“When handling political information, the inherent partisan biases in LLMs could exacerbate echo chambers and polarization (Cinelli et al. 2021; Conover et al. 2021).”
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364 · cited 1× · see citing works
“Our study follows a long line of scholarship aimed at understanding how online platforms affect user information diets (Rahwan et al. 2019) and their contributions to issues such as the proliferation of misinformation (Lazer et al. 2018), echo chambers (Cinelli et al. 2021), and polarization…”
Fighting misinformation on social media using crowdsourced judgments of news source quality
Pennycook, G., & Rand, D. G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806781116 · Cited by 947 · cited 6× · see citing works
“Therefore, our discussion below will focus on the results from the first approach.”
InfluentialBias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Political Polarization on Twitter
Conover, M., Ratkiewicz, J., Francisco, M., Goncalves, B., Menczer, F., & Flammini, A. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14126 · Cited by 626 · cited 2× · see citing works
“When handling political information, the inherent partisan biases in LLMs could exacerbate echo chambers and polarization (Cinelli et al. 2021; Conover et al. 2021).”
Self-citationFair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis
Budak, C., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw007 · Cited by 336 · see citing works
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., & Ren, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.923 · Cited by 270 · see citing works
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265 · cited 1× · see citing works
“At the same time, more studies on how humans interact with AI-augmented systems (Sharma, Liao, and Xiao 2024; DeVerna et al. 2024) and how information processed by AI influences humans are needed (Jakesch et al. 2023; Simchon, Edwards, and Lewandowsky 2024).”
Examining the consumption of radical content on YouTube
Hosseinmardi, H., Ghasemian, A., Clauset, A., Mobius, M., Rothschild, D. M., & Watts, D. J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2101967118 · Cited by 171 · see citing works
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143 · see citing works
Evaluating Verifiability in Generative Search Engines
Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143 · cited 2× · see citing works
“Recent audits of popular AI search engines reveal that their results often contain unsup-ported claims (Liu, Zhang, and Liang 2023) and biases depending on the queries (Li and Sinnamon 2024).”
Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking
Sharma, N., Liao, Q. V., & Xiao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642459 · Cited by 132 · cited 3× · see citing works
“Furthermore, experiments conducted by Sharma, Liao, and Xiao (2024) demonstrate that users may engage with more biased information when interacting with AI search engines, and LLMs with predefined opinions can exacerbate this bias.”
Ranking Manipulation for Conversational Search Engines
Pfrommer, S., Bai, Y., Gautam, T., & Sojoudi, S. (2024) · arXiv · doi:10.48550/arxiv.2406.03589 · Cited by 8 · cited 1× · see citing works
“…studies should explore how LLMs handle information from differ-10 ent sources in more realistic scenarios, how other information selection criteria, such as relevance and freshness (Sun et al. 2023), affect the outcomes, and how to mitigate emerging manipulation techniques (Pfrommer et al. 2024).”
Flagged first, then by citations
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“[145] / Yang and Menczer [220] (3) Natural language inference: Lee et al.”
BackgroundInfluentialCan Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369 · see citing works
- Authors
- Yang, K.-C., & Menczer, F.
- Affiliations
- Northeastern University (US)
- Indiana University Bloomington (US)
- Venue
- Proceedings of the 17th ACM Web Science Conference 2025
- Publisher
- ACM
- Published
- 2025-05-19
- Pages
- 127-137
- OpenAlex
- W4410486519
- Cited by
- 81
- References
- 14
- Citations by year
- 2026: 12 · 2025: 24 · 2024: 29 · 2023: 14 · 2022: 1 · 2021: 1
- FWCI
- 86.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Web Science Conference (WebSci) (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Volkswagen Foundation
- DARPA (HR001121C016)
- Knight Foundation
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, news source credibility, political bias, partisan bias, information quality assessment, LLM comparison, human evaluation of AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
