The DOI matches its DataCite record.
Fairness of ChatGPT
Open accessAbstract
Understanding and addressing unfairness in LLMs are crucial for responsible AI deployment. However, there is a limited number of quantitative analyses and in-depth studies regarding fairness evaluations in LLMs, especially when applying LLMs to high-stakes fields. This work aims to fill this gap by providing a systematic evaluation of the effectiveness and fairness of LLMs using ChatGPT as a study case. We focus on assessing ChatGPT's performance in high-takes fields including education, criminology, finance and healthcare. To conduct a thorough evaluation, we consider both group fairness and individual fairness metrics. We also observe the disparities in ChatGPT's outputs under a set of biased or unbiased prompts. This work contributes to a deeper understanding of LLMs' fairness performance, facilitates bias mitigation and fosters the development of responsible AI systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Modified prompting language techniques have been shown to have limited effectiveness. Borchers et al. (2022), for example, find instructions that prompt diversity or gender equality to be unsuccessful for bias removal in outputs. Similarly, Li and Zhang (2023) find similar generated outputs when using biased and unbiased prompts.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Li and Zhang 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 26
- OpenAI, "Gpt-4 technical report," arXiv, 2023.
- Y. Ge, W. Hua, K. Mei, jianchao ji, J. Tan, S. Xu, Z. Li, and Y. Zhang, "OpenAGI: When LLM meets domain experts," in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- Y. Li, H. Chen, S. Xu, Y. Ge, J. Tan, S. Liu, and Y. Zhang, "Fairness in recommendation: Foundations, methods, and applications," ACM Transactions on Intelligent Systems and Technology, vol. 14, no. 5, pp. 1-48, 2023.
- Y. Ge, S. Liu, Z. Fu, J. Tan, Z. Li, S. Xu, Y. Li, Y. Xian, and Y. Zhang, "A survey on trustworthy recommender systems," arXiv:2207.12515, 2022. · arXiv
- T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing, "Explor- ing ai ethics of chatgpt: A diagnostic analysis," arXiv preprint arXiv:2301.12867, 2023. · arXiv
- Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu et al., "Summary of chatgpt/gpt-4 research and perspective towards the future of large lan- guage models," arXiv preprint arXiv:2304.01852, 2023. · arXiv
- W. Hariri, "Unlocking the potential of chatgpt: A com- prehensive exploration of its applications, advantages, limitations, and future directions in natural language processing," arXiv preprint arXiv:2304.02017, 2023. · arXiv
- H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, "Capabilities of gpt-4 on medical challenge problems," arXiv preprint arXiv:2303.13375, 2023. · arXiv
- M. Hardt, E. Price, and N. Srebro, "Equality of op- portunity in supervised learning," Advances in neural information processing systems, vol. 29, 2016.
- M. J. Kusner, J. Loftus, C. Russell, and R. Silva, "Coun- terfactual fairness," Advances in neural information pro- cessing systems, vol. 30, 2017.
- S. Caton and C. Haas, "Fairness in machine learning: A survey," arXiv preprint arXiv:2010.04053, 2020. · arXiv
- N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, "A survey on bias and fairness in machine learning," ACM Computing Surveys (CSUR), vol. 54, no. 6, pp. 1-35, 2021.
- T. P. Pagano, R. B. Loureiro, M. M. Araujo, F. V. N. Lisboa, R. M. Peixoto, G. A. d. S. Guimaraes, L. L. d. Santos, G. O. R. Cruz, E. L. S. de Oliveira, M. Cruz et al., "Bias and unfairness in machine learning mod- els: a systematic literature review," arXiv preprint arXiv:2202.08176, 2022. · arXiv
- M. Wan, D. Zha, N. Liu, and N. Zou, "Modeling tech- niques for machine learning fairness: A survey," arXiv preprint arXiv:2111.03015, 2021. · arXiv
- G. Lin and Y. Zhang, "Sparks of artificial general rec- ommender (agr): Experiments with chatgpt," Algorithms, vol. 16, no. 9, p. 432, 2023.
- H. Sun, Z. Zhang, J. Deng, J. Cheng, and M. Huang, "Safety assessment of chinese large language models," arXiv preprint arXiv:2304.10436, 2023. · arXiv
- W. Hua, Y. Ge, S. Xu, J. Ji, and Y. Zhang, "Up5: Unbiased foundation model for fairness-aware recom- mendation," EACL, 2024.
- T. Le Quy, T. H. Nguyen, G. Friege, and E. Ntoutsi, "Evaluation of group fairness measures in student per- formance prediction problems," in Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19-23, 2022, Pro- ceedings, Part I. Springer, 2023, pp. 119-136.
- J. Dressel and H. Farid, "The accuracy, fairness, and limits of predicting recidivism," Science advances, vol. 4, no. 1, p. eaao5580, 2018.
- D. Chen and L. Zhang, "Monotonicity for ai ethics and society: An empirical study of the monotonic neural additive model in criminology, education, health care, and finance," arXiv preprint arXiv:2301.07060, 2023. · arXiv
- M. B. Zafar, I. Valera, M. Gomez Rodriguez, and K. P. Gummadi, "Fairness beyond disparate treatment & dis- parate impact: Learning classification without disparate mistreatment," in Proceedings of the 26th international conference on world wide web, 2017, pp. 1171-1180.
- E. Krasanakis, E. Spyromitros-Xioufis, S. Papadopoulos, and Y. Kompatsiaris, "Adaptive sensitive reweighting to mitigate bias in fairness-aware classification," in Pro- ceedings of the 2018 world wide web conference, 2018, pp. 853-862.
- T. Le Quy, A. Roy, V. Iosifidis, W. Zhang, and E. Ntoutsi, "A survey on datasets for fairness-aware machine learn- ing," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 12, no. 3, p. e1452, 2022.
- C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, "Fairness through awareness," in Proceedings of the 3rd innovations in theoretical computer science conference, 2012, pp. 214-226.
- M. B. Zafar, I. Valera, M. Gomez-Rodriguez, and K. P. Gummadi, "Fairness constraints: A flexible approach for fair classification," The Journal of Machine Learning Research, vol. 20, no. 1, pp. 2737-2778, 2019.
- R. Berk, H. Heidari, S. Jabbari, M. Kearns, and A. Roth, "Fairness in criminal justice risk assessments: The state of the art," Sociological Methods & Research, vol. 50, no. 1, pp. 3-44, 2021.
Cited by
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
Introduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63
Metadata
- Authors
- Li, Y., Zhang, L., & Zhang, Y.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 1-6
- arXiv
- 2305.18569
- OpenAlex
- W4393407165
- Cited by
- 28
- References
- 0
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 6 · 2023: 7
- FWCI
- 7.65
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Conference on Information Sciences and Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, algorithmic fairness, ChatGPT, group fairness, individual fairness, bias mitigation, responsible AI, criminology, health care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fairness of ChatGPT. https://app.cochat.ai/papers/10.48550/arxiv.2305.18569
- The paper
- Li, Y., Zhang, L., & Zhang, Y. (2023). Fairness of ChatGPT. arXiv. https://doi.org/10.48550/arxiv.2305.18569
