CoChat
Sign Up
PreprintCited by 76Ethics and Social Impacts of AIFeminist Epistemology and Gender StudiesMisinformation and Its Impacts

Epistemic Injustice in Generative AI

Open access
Kay, Jackie, Kasirzadeh, Atoosa, Mohamed, Shakir·2024·arXiv
DOI 10.48550/arxiv.2408.11441arXiv 2408.11441
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper investigates how generative AI can potentially undermine the integrity of collective knowledge and the processes we rely on to acquire, assess, and trust information, posing a significant threat to our knowledge ecosystem and democratic discourse. Grounded in social and political philosophy, we introduce the concept of \emph{generative algorithmic epistemic injustice}. We identify four key dimensions of this phenomenon: amplified and manipulative testimonial injustice, along with hermeneutical ignorance and access injustice. We illustrate each dimension with real-world examples that reveal how generative AI can produce or amplify misinformation, perpetuate representational harm, and create epistemic inequities, particularly in multilingual contexts. By highlighting these injustices, we aim to inform the development of epistemically just generative AI systems, proposing strategies for resistance, system design principles, and two approaches that leverage generative AI to foster a more equitable information ecosystem, thereby safeguarding democratic values and the integrity of knowledge production.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.11441

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A multiplicity of AI systems could lead to an increase in the quantity and quality of misinformation.

    “As described in Section 3.1, a multiplicity of AI systems could lead to an increase in the quantity and quality of misinformation (Kay et al., 2024; Zhou et al., 2023).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Kay et al., 2024

    In “Specific Harms of Generative Epistemic Injustice” (page unavailable), the paper says generative models can reproduce or synthesize misinformation and that their outputs can be highly persuasive. It also discusses information overload, but does not specifically say that a multiplicity of AI systems would increase both the quantity and quality of misinformation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 69

  1. Abulimiti, A.; Clavel, C.; and Cassell, J. 2023. How About Kind of Generating Hedges using End-to-End Neural Mod- els? In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), 877-892. Toronto, Canada: Association for Computational Linguistics.
  2. Agre, P. E. 2014. Toward a critical technical practice: Lessons learned in trying to reform AI. In Social science, technical systems, and cooperative work, 131-157. Psychol- ogy Press.
  3. Alvarado, R. 2023. AI as an Epistemic Technology. Science and Engineering Ethics, 29(5): 32.
  4. Anderau, G. 2023. Fake news and epistemic flooding. Syn- these, 202(4): 106.
  5. Augenstein, I.; Baldwin, T.; Cha, M.; Chakraborty, T.; Ciampaglia, G. L.; Corney, D.; DiResta, R.; Ferrara, E.; Hale, S.; Halevy, A.; et al. 2023. Factuality challenges in the era of large language models. arXiv preprint arXiv:2310.05189, 1-13. · arXiv
  6. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das- Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022. Training a helpful and harmless assistant with re- inforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 1-66. · arXiv
  7. Bender, E. M.; Gebru, T.; McMillan-Major, A.; and Shmitchell, S. 2021. On the dangers of stochastic par- rots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 610-623.
  8. Bender, E. M.; and Koller, A. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, 5185-5198.
  9. Berenstain, N. 2020. White feminist gaslighting. Hypatia, 35(4): 733-758.
  10. Bianchi, F.; Kalluri, P.; Durmus, E.; Ladhak, F.; Cheng, M.; Nozza, D.; Hashimoto, T.; Jurafsky, D.; Zou, J.; and Caliskan, A. 2023. Easily accessible text-to-image gener- ation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Ac- countability, and Transparency, 1493-1504.
  11. Bird, C.; Ungless, E.; and Kasirzadeh, A. 2023. Typology of risks of generative text-to-image models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 396-410.
  12. Birhane, A. 2021. Algorithmic injustice: a relational ethics approach. Patterns, 2(2): 1-9.
  13. Birhane, A.; Prabhu, V. U.; and Kahembwe, E. 2021. Mul- timodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 1-33. · arXiv
  14. Bondy, P. 2010. Argumentative injustice. Informal Logic, 30(3): 236-278.
  15. Byrnes, J.; and Spear, A. 2023. Epistemic Injustice and Algorithmic Epistemic Injustice in Healthcare. In Interna- tional Conference on Computer Ethics, volume 1, 1-4.
  16. Caliskan, A.; Bryson, J. J.; and Narayanan, A. 2017. Se- mantics derived automatically from language corpora con- tain human-like biases. Science, 356(6334): 183-186.
  17. Chan, A.; Bradley, H.; and Rajkumar, N. 2023. Reclaim- ing the Digital Commons: A Public Data Trust for Training Data. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES '23, 855-868. New York, NY, USA: Association for Computing Machinery. ISBN 9798400702310.
  18. Collins, P. H. 2000. Black feminist thought: Knowledge, con- sciousness, and the politics of empowerment. Routledge.
  19. Crenshaw, K. W. 2013. Mapping the margins: Intersection- ality, identity politics, and violence against women of color. In The public nature of private violence, 93-118. Routledge.
  20. Davis, K. M. 2021. Tainted Tap: Flint's Journey from Crisis to Recovery. UNC Press Books.
  21. De Proost, M.; and Pozzi, G. 2023. Conversational Artificial Intelligence and the Potential for Epistemic Injustice. The American Journal of Bioethics, 23(5): 51-53.
  22. Dotson, K. 2011. Tracking epistemic violence, tracking practices of silencing. Hypatia, 26(2): 236-257.
  23. Edenberg, E.; and Wood, A. 2023. An Epistemic Lens on Algorithmic Fairness. In Proceedings of the 3rd ACM Con- ference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1-10.
  24. El-Sayed, S.; Akbulut, C.; McCroskery, A.; Keeling, G.; Kenton, Z.; Jalan, Z.; Marchal, N.; Manzini, A.; Shevlane, T.; Vallor, S.; et al. 2024. A Mechanism-Based Approach to Mitigating Harms from Persuasive Generative AI. arXiv preprint arXiv:2404.15058, 1-55. · arXiv
  25. Ferrara, E. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738, 1-24. · arXiv
  26. Fredman, S.; Du Toit, D.; Graham, M.; Howson, K.; Heeks, R.; van Belle, J.-P.; Mungai, P.; and Osiki, A. 2020. Thinking out of the box: Fair work for platform workers. King's Law Journal, 31(2): 236-249.
  27. Fricker, M. 2007. Epistemic injustice: Power and the ethics of knowing. Oxford University Press.
  28. Fricker, M.; and Jenkins, K. 2017. Epistemic injustice, igno- rance, and trans experiences. In The Routledge companion to feminist philosophy, 268-278. Routledge.
  29. Ganguli, D.; Lovitt, L.; Kernion, J.; Askell, A.; Bai, Y.; Ka- davath, S.; Mann, B.; Perez, E.; Schiefer, N.; Ndousse, K.; et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 1-30. · arXiv
  30. García-Peñalvo, F.; and Vázquez-Ingelmo, A. 2023. What do we mean by GenAI? A systematic mapping of the evolu- tion, trends, and techniques involved in Generative AI. In- ternational Journal of Interactive Multimedia and Artificial Intelligence, 7-16.
  31. Glaberson, S. 2022. The Epistemic Injustice of Algorith- mic Family Policing. UC Irvine Law Review (Forthcoming), 404-456.
  32. Goldstein, J. A.; Chao, J.; Grossman, S.; Stamos, A.; and Tomz, M. 2024. How persuasive is AI-generated propa- ganda? PNAS nexus, 3(2): 1-7.
  33. Groves, L.; Peppin, A.; Strait, A.; and Brennan, J. 2023. Going public: the role of public participation approaches in commercial AI labs. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 1162-1173.
  34. Guo, Z.; Schlichtkrull, M.; and Vlachos, A. 2022. A Survey on Automated Fact-Checking. Transactions of the Associa- tion for Computational Linguistics, 10: 178-206.
  35. Hanna, A.; Denton, E.; Smart, A.; and Smith-Loud, J. 2020. Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 conference on fairness, account- ability, and transparency, 501-512.
  36. Haraway, D. 1988. Situated knowledges: The science ques- tion in feminism and the privilege of partial perspective. Feminist studies, 14(3): 575-599.
  37. Heger, A. K.; Marquis, L. B.; Vorvoreanu, M.; Wallach, H.; and Wortman Vaughan, J. 2022. Understanding Ma- chine Learning Practitioners' Data Documentation Percep- tions, Needs, Challenges, and Desiderata. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2): 1- 29.
  38. Hoffmann, A. L. 2019. Where fairness fails: data, algo- rithms, and the limits of antidiscrimination discourse. In- formation, Communication & Society, 22(7): 900-915.
  39. Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2020. The Curious Case of Neural Text Degeneration. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
  40. Huang, S.; and Siddarth, D. 2023. Generative AI and the digital commons. arXiv preprint arXiv:2303.11074, 1-18. · arXiv
  41. Hull, G. 2023. Dirty data labeled dirt cheap: Epistemic in- justice in machine learning systems. Ethics and Information Technology, 25(3): 38.
  42. Ji, Z.; Lee, N.; Frieske, R.; Yu, T.; Su, D.; Xu, Y.; Ishii, E.; Bang, Y. J.; Madotto, A.; and Fung, P. 2023. Survey of hal- lucination in natural language generation. ACM Computing Surveys, 55(12): 1-38.
  43. Kalluri, P.; et al. 2020. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature, 583(7815): 169-169.
  44. Kasirzadeh, A. 2022. Algorithmic fairness and structural in- justice: Insights from feminist political philosophy. In Pro- ceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 349-356.
  45. Kasirzadeh, A.; and Gabriel, I. 2023. In conversation with Artificial Intelligence: aligning language models with hu- man values. Philosophy & Technology, 36(2): 1-24.
  46. Kazenwadel, D.; and Steinert, C. V. 2023. How User Lan- guage Affects Conflict Fatality Estimates in ChatGPT. arXiv preprint arXiv:2308.00072, 1-24. · arXiv
  47. Kukutai, T.; and Taylor, J. 2016. Indigenous data sovereignty: Toward an agenda. ANU press.
  48. Lett, E.; Asabor, E.; Beltrán, S.; Cannon, A. M.; and Arah, O. A. 2022. Conceptualizing, contextualizing, and opera- tionalizing race in quantitative health sciences research. The Annals of Family Medicine, 20(2): 157-163.
  49. Liu, K.; Casper, S.; Hadfield-Menell, D.; and Andreas, J. 2023. Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthful- ness? In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceed- ings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing, 4791-4797. Singapore: Associ- ation for Computational Linguistics.
  50. Lu, C.; Kay, J.; and McKee, K. 2022. Subverting machines, fluctuating identities: Re-learning human categorization. In Proceedings of the 2022 ACM Conference on Fairness, Ac- countability, and Transparency, 1005-1015.
  51. McKinnon, R. 2017. Gaslighting as epistemic injustice. The Routledge handbook of epistemic injustice, 167-174.
  52. Medina, J. 2018. Misrecognition and epistemic injustice. Feminist Philosophy Quarterly, 4(4): 1-16.
  53. Monteith, S.; Glenn, T.; Geddes, J. R.; Whybrow, P. C.; Achtyes, E.; and Bauer, M. 2024. Artificial intelligence and increasing misinformation. The British Journal of Psychia- try, 224(2): 33-35.
  54. Neumann, T.; De-Arteaga, M.; and Fazelpour, S. 2022. Jus- tice in misinformation detection systems: An analysis of al- gorithms, stakeholders, and potential harms. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 1504-1515.
  55. Pohlhaus, G. 2012. Relational knowing and epistemic in- justice: Toward a theory of willful hermeneutical ignorance. Hypatia, 27(4): 715-735.
  56. Pozzi, G. 2023. Automated opioid risk scores: a case for machine learning-induced epistemic injustice in healthcare. Ethics and Information Technology, 25(1): 3.
  57. Prabhakaran, V.; Qadri, R.; and Hutchinson, B. 2022. Cul- tural incongruencies in artificial intelligence. arXiv preprint arXiv:2211.13069, 1-5. · arXiv
  58. Qadri, R.; Shelby, R.; Bennett, C. L.; and Denton, E. 2023. AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia. In Proceed- ings of the 2023 ACM Conference on Fairness, Accountabil- ity, and Transparency, 506-517.
  59. Ryan, S. 2018. Epistemic environmentalism. Journal of Philosophical Research, 43: 97-112.
  60. Schlichtkrull, M. S.; Guo, Z.; and Vlachos, A. 2023. AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web. In Thirty-seventh Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track.
  61. Swire-Thompson, B.; DeGutis, J.; and Lazer, D. 2020. Searching for the backfire effect: Measurement and design considerations. Journal of applied research in memory and cognition, 9(3): 286-299.
  62. Symons, J.; and Alvarado, R. 2022. Epistemic injustice and data science technologies. Synthese, 200(2): 87.
  63. Vaccari, C.; and Chadwick, A. 2020. Deepfakes and disin- formation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news. Social Media+ Society, 6(1): 2056305120903408.
  64. Weidinger, L.; Rauh, M.; Marchal, N.; Manzini, A.; Hen- dricks, L. A.; Mateos-Garcia, J.; Bergman, S.; Kay, J.; Grif- fin, C.; Bariach, B.; et al. 2023. Sociotechnical Safety Evaluation of Generative AI Systems. arXiv preprint arXiv:2310.11986. · arXiv
  65. Weidinger, L.; Uesato, J.; Rauh, M.; Griffin, C.; Huang, P.- S.; Mellor, J.; Glaese, A.; Cheng, M.; Balle, B.; Kasirzadeh, A.; et al. 2022. Taxonomy of risks posed by language mod- els. In Proceedings of the 2022 ACM Conference on Fair- ness, Accountability, and Transparency, 214-229.
  66. Wexler, R. 2018. The odds of justice: Code of silence: How private companies hide flaws in the software that govern- ments use to decide who goes to prison and who gets out. Chance, 31(3): 67-72.
  67. Xu, F.; Uszkoreit, H.; Du, Y.; Fan, W.; Zhao, D.; and Zhu, J. 2019. Explainable AI: A brief survey on history, research areas, approaches and challenges. In Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9- 14, 2019, Proceedings, Part II 8, 563-574. Springer.
  68. Zhang, H.; Diao, S.; Lin, Y.; Fung, Y.; Lian, Q.; Wang, X.; Chen, Y.; Ji, H.; and Zhang, T. 2024. R-Tuning: Instructing Large Language Models to Say 'I Don't Know'. In Duh, K.; Gomez, H.; and Bethard, S., eds., Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 7113-7139. Mexico City, Mexico: Association for Computational Linguistics.
  69. Ziegler, D. M.; Stiennon, N.; Wu, J.; Brown, T. B.; Radford, A.; Amodei, D.; Christiano, P.; and Irving, G. 2019. Fine- tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 1-26. · arXiv

Cited by

Sorted by citations

Authors
Kay, J., Kasirzadeh, A., & Mohamed, S.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • University of Edinburgh (GB)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
7
Pages
684-697
ISSN
3065-8365 (ISSN-L)
DOI
10.48550/arxiv.2408.11441
arXiv
2408.11441
OpenAlex
W4404518333
Cited by
76
References
0
Citations by year
2026: 44 · 2025: 31 · 2024: 1
FWCI
34.56
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Feminist Epistemology and Gender Studies
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
epistemic injustice, generative AI, testimonial injustice, misinformation, representational harm, multilingualism, democratic discourse, collective knowledge, information ecosystem

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review