CoChat
Sign Up
PreprintCited by 6Ethics and Social Impacts of AICultural Identity and RepresentationExplainable Artificial Intelligence (XAI)

The Case for "Thick Evaluations" of Cultural Representation in AI

Open access
Qadri, Rida, Diaz, Mark, Wang, Ding, Madaio, Michael·2025·arXiv
DOI 10.48550/arxiv.2503.19075arXiv 2503.19075

Preprint — published as 10.1609/aies.v8i3.36696

Open this paper in CoChat — free accountOpen PDF

Abstract

Generative AI model outputs have been increasingly evaluated for their (in)ability to represent non-Western cultures. We argue that these evaluations often operate through reductive ideals of representation, abstracted from how people define their own representation and neglecting the inherently interpretive and contextual nature of cultural representation. In contrast to these 'thin' evaluations, we introduce the idea of 'thick evaluations:' a more granular, situated, and discursive measurement framework for evaluating representations of social worlds in AI outputs, steeped in communities' own understandings of representation. We develop this evaluation framework through workshops in South Asia, by studying the 'thick' ways in which people interpret and assign meaning to AI-generated images of their own cultures. We introduce practices for thicker evaluations of representation that expand the understanding of representation underpinning AI evaluations and by co-constructing metrics with communities, bringing measurement in line with the experiences of communities on the ground.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.19075

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent social science work has proposed richer evaluation categories including incorrectness, missingness, specificity, coherence, and connotation.

    “Taking a different perspective, recent social science work [89] has proposed richer evaluation categories that can give more nuance to assessments of cultural images, including: incorrectness, missingness, specificity, coherence, and connotation.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [89]

  • Each community actively negotiated their own representation goals, shaped by their social, geographical and cultural contexts, and in conversation with community members and image materials.

    “Echoing recent research by Qadri et al. [89], our findings show how each community actively negotiated their own representation goals -shaped by their social, geographical and cultural contexts, and in conversation with community members and image materials.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [89]

  • Recent works argue against treatments of representation as a static concept that can be evaluated objectively.

    “This echoes recent works [10, 68, 89, 90] that argue against treatments of representation as a static concept that can be evaluated objectively.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [89]

  • Social worlds and their ideal representations are not static, but can evolve over time in response to changing contexts and aspirations.

    “Furthermore, our findings surface how social worlds and their ideal representations are not static, but can evolve over time in response to changing contexts and aspirations (cf. [89]).”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [89]

  • In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation or allow community members to self-select their participation, albeit with some limitations.

    “In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation (e.g., across geographical regions [52, 68, 91] and through local research partners [89]), or allow community members to selfselect their participation, albeit with some limitations (e.g., adults, higher-education levels or topic expertise, English-language ability”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [89]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 83

  1. Abbasi, M.; Friedler, S. A.; Scheidegger, C.; and Venkata- subramanian, S. 2019. Fairness in representation: quantify- ing stereotyping as a representational harm. In Proceedings of the 2019 SIAM International Conference on Data Mining, 801-809.
  2. Abu-Lughod, J. L. 1991. Before European hegemony: the world system AD 1250-1350. Oxford University Press, USA.
  3. Agre, P. E. 2014. Toward a critical technical practice: Lessons learned in trying to reform AI. In Social science, technical systems, and cooperative work, 131-157. Psychol- ogy Press.
  4. Arzberger, A.; Buijsman, S.; Lupetti, M. L.; Bozzon, A.; and Yang, J. 2024. Nothing Comes Without Its World-Practical Challenges of Aligning LLMs to Situated Human Values through RLHF. In Proceedings of the AAAI/ACM Confer- ence on AI, Ethics, and Society, volume 7, 61-73.
  5. Babones, S. 2016. Interpretive quantitative methods for the social sciences. Sociology, 50(3): 453-469.
  6. Barasch, M.; and Serrano, L. 1992. Icon: Studies in the His- tory of an Idea. NYU Press.
  7. Basu, A.; Babu, R. V.; and Pruthi, D. 2023. Inspecting the geographical representativeness of images from text-to- image models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 5136-5147.
  8. Bergman, S.; Marchal, N.; Mellor, J.; Mohamed, S.; Gabriel, I.; and Isaac, W. 2024. STELA: a community-centred ap- proach to norm elicitation for AI alignment. Scientific Re- ports, 14(1): 6616.
  9. Birhane, A.; Isaac, W.; Prabhakaran, V.; Diaz, M.; Elish, M. C.; Gabriel, I.; and Mohamed, S. 2022a. Power to the people? Opportunities and challenges for participatory AI. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1-8.
  10. Birhane, A.; Kalluri, P.; Card, D.; Agnew, W.; Dotan, R.; and Bao, M. 2022b. The values encoded in machine learning research. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 173-184.
  11. Blodgett, S. L.; Lopez, G.; Olteanu, A.; Sim, R.; and Wal- lach, H. 2021. Stereotyping Norwegian salmon: An inven- tory of pitfalls in fairness benchmark datasets. In Proceed- ings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Con- ference on Natural Language Processing (Volume 1: Long Papers), 1004-1015.
  12. Braun, V.; and Clarke, V. 2006. Using thematic analysis in psychology. Qualitative research in psychology, 3(2): 77- 101.
  13. Braun, V.; and Clarke, V. 2021. One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative research in psychology, 18(3): 328-352.
  14. Broussard, M. 2018. Artificial unintelligence: How comput- ers misunderstand the world. mit Press.
  15. Cabitza, F.; Campagner, A.; and Basile, V. 2023. Toward a perspectivist turn in ground truthing for predictive comput- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 6860-6868.
  16. Chandhiramowuli, S.; Taylor, A. S.; Heitlinger, S.; and Wang, D. 2024. Making Data Work Count. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1): 1-26.
  17. Chasalow, K.; and Levy, K. 2021. Representativeness in statistics, politics, and machine learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 77-89.
  18. Chien, J.; and Danks, D. 2024. Beyond Behaviorist Repre- sentational Harms: A Plan for Measurement and Mitigation. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, 933-946.
  19. Cho, J.; Zala, A.; and Bansal, M. 2022. DALL-Eval: Prob- ing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers. CoRR, abs/2202.04053. · arXiv
  20. Collins, K. M.; Kim, N.; Bitton, Y.; Rieser, V.; Omidshafiei, S.; Hu, Y.; Chen, S.; Dutta, S.; Chang, M.; Lee, K.; et al. 2024. Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, 293-303.
  21. Crabtree, A. 2024. H is for Human and How (Not) To Evaluate Qualitative Research in HCI. arXiv preprint arXiv:2409.01302. · arXiv
  22. Crawford, K. 2017. The trouble with bias. In Conference on Neural Information Processing Systems, invited speaker.
  23. Cunningham, J.; Benabdallah, G.; Rosner, D.; and Taylor, A. 2023. On the Grounds of Solutionism: Ontologies of Black- ness and HCI. ACM Trans. Comput.-Hum. Interact., 30(2).
  24. Davani, A. M.; Díaz, M.; and Prabhakaran, V. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10: 92-110.
  25. Delgado, F.; Yang, S.; Madaio, M.; and Yang, Q. 2023. The participatory turn in ai design: Theoretical foundations and the current state of practice. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mecha- nisms, and Optimization, 1-23.
  26. Denton, E.; Hanna, A.; Amironesei, R.; Smart, A.; Nicole, H.; and Scheuerman, M. K. 2020. Bringing the people back in: Contesting benchmark machine learning datasets. arXiv preprint arXiv:2007.07399. · arXiv
  27. Desai, D. 2000. Imaging difference: The politics of repre- sentation in multicultural art education. Studies in Art Edu- cation, 41(2): 114-129.
  28. Díaz, M.; Kivlichan, I.; Rosen, R.; Baker, D.; Amironesei, R.; Prabhakaran, V.; and Denton, E. 2022. Crowdwork- sheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In Proceed- ings of the 2022 ACM Conference on Fairness, Accountabil- ity, and Transparency, 2342-2351.
  29. Díaz, M.; and Smith, A. D. 2024. What Makes An Expert? Reviewing How ML Researchers Define" Expert". In Pro- ceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, 358-370.
  30. D'ignazio, C.; and Klein, L. F. 2023. Data feminism. MIT press.
  31. Ghosh, S. 2024. Interpretations, Representations, and Stereotypes of Caste within Text-to-Image Generators. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, 490-502.
  32. Ghosh, S.; Lutz, N.; and Caliskan, A. 2024. "I Don't See Myself Represented Here at All": User Experiences of Sta- ble Diffusion Outputs Containing Representational Harms across Gender Identities and Nationalities. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, vol- ume 7, 463-475.
  33. Ghosh, S.; Venkit, P. N.; Gautam, S.; Wilson, S.; and Caliskan, A. 2024. Do Generative AI Models Output Harm while Representing Non-Western Cultures: Evidence from A Community-Centered Approach. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, 476-489.
  34. Gordon, M. L.; Lam, M. S.; Park, J. S.; Patel, K.; Hancock, J.; Hashimoto, T.; and Bernstein, M. S. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Fac- tors in Computing Systems, 1-19.
  35. Hall, S. 1989. Cultural identity and cinematic representa- tion. Framework: The Journal of Cinema and Media, (36): 68-81.
  36. Hall, S. 1997. Representation: Cultural Representations and Signifying Practices. London: SAGE Publications.
  37. Hanna, A.; Denton, E.; Smart, A.; and Smith-Loud, J. 2020. Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 conference on fairness, account- ability, and transparency, 501-512.
  38. Harvey, E.; Sheng, E.; Blodgett, S. L.; Chouldechova, A.; Garcia-Gathright, J.; Olteanu, A.; and Wallach, H. 2024. Gaps Between Research and Practice When Measuring Rep- resentational Harms Caused by LLM-Based Systems. arXiv preprint arXiv:2411.15662. · arXiv
  39. Hosseini, S.; Palangi, H.; and Awadallah, A. H. 2023. An empirical study of metrics to measure representational harms in pre-trained language models. arXiv preprint arXiv:2301.09211. · arXiv
  40. Jackson, J. L. 2013. Thin description: ethnography and the African Hebrew Israelites of Jerusalem. Harvard University Press.
  41. Jacobs, A. Z.; and Wallach, H. 2021. Measurement and fair- ness. In Proceedings of the 2021 ACM conference on fair- ness, accountability, and transparency, 375-385.
  42. Jha, A.; Prabhakaran, V.; Denton, R.; Laszlo, S.; Dave, S.; Qadri, R.; Reddy, C.; and Dev, S. 2024. ViSAGe: A Global- Scale Analysis of Visual Stereotypes in Text-to-Image Gen- eration. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12333-12347.
  43. Kak, A. 2020. " The Global South is everywhere, but also always somewhere" National Policy Narratives and AI Jus- tice. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 307-312.
  44. Kannen, N.; Ahmad, A.; Andreetto, M.; Prabhakaran, V.; Prabhu, U.; Dieng, A. B.; Bhattacharyya, P.; and Dave, S. 2024. Beyond Aesthetics: Cultural Competence in Text-to- Image Models. arXiv preprint arXiv:2407.06863. · arXiv
  45. Katzman, J.; Wang, A.; Scheuerman, M.; Blodgett, S. L.; Laird, K.; Wallach, H.; and Barocas, S. 2023. Taxonomiz- ing and measuring representational harms: A look at image tagging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 14277-14285.
  46. Kay, M.; Matuszek, C.; and Munson, S. A. 2015. Unequal representation and gender stereotypes in image search re- sults for occupations. In Proceedings of the 33rd annual acm conference on human factors in computing systems, 3819- 3828.
  47. Khairullah, D. H.; and Khairullah, Z. Y. 2009. Cross-cultural analysis of gender roles: Indian and US advertisements. Asia Pacific Journal of Marketing and Logistics, 21(1): 58-75.
  48. Khanuja, S.; Bansal, D.; Mehtani, S.; Khosla, S.; Dey, A.; Gopalan, B.; Margam, D. K.; Aggarwal, P.; Nagipogu, R. T.; Dave, S.; Gupta, S.; Gali, S. C. B.; Subramanian, V.; and Talukdar, P. 2021. MuRIL: Multilingual Representations for Indian Languages. arXiv:2103.10730. · arXiv
  49. Kitchin, R. 2014. Big Data, new epistemologies and paradigm shifts. Big Data Society, 1(1): 1-12.
  50. Lahoti, P.; Blumm, N.; Ma, X.; Kotikalapudi, R.; Potluri, S.; Tan, Q.; Srinivasan, H.; Packer, B.; Beirami, A.; Beutel, A.; et al. 2023. Improving diversity of demographic represen- tation in large language models via collective-critiques and self-voting. arXiv preprint arXiv:2310.16523. · arXiv
  51. Li, H.; Jiang, L.; Dziri, N.; Ren, X.; and Choi, Y. 2024. CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting. arXiv preprint arXiv:2404.10199. · arXiv
  52. Lindtner, S.; Bardzell, S.; and Bardzell, J. 2016. Reconstitut- ing the utopian vision of making: HCI after technosolution- ism. In Proceedings of the 2016 chi conference on human factors in computing systems, 1390-1402.
  53. Love, H. 2013. Close reading and thin description. Public culture, 25(3): 401-434.
  54. Mack, K. A.; Qadri, R.; Denton, R.; Kane, S. K.; and Ben- nett, C. L. 2024. "They only care to show us the wheelchair": disability representation in text-to-image AI models. In Pro- ceedings of the CHI Conference on Human Factors in Com- puting Systems, 1-23.
  55. Madaio, M.; Kapania, S.; Qadri, R.; Wang, D.; Zaldivar, A.; Denton, R.; and Wilcox, L. 2024. Learning about Respon- sible AI On-The-Job: Learning Pathways, Orientations, and Aspirations. In The 2024 ACM Conference on Fairness, Ac- countability, and Transparency, 1544-1558.
  56. Malik, M.; and Malik, M. M. 2022. Critical technical awak- enings. Journal of Social Computing, 2(4): 365-384.
  57. Mattern, S. 2021. A City Is Not a Computer: Other Urban Intelligences. Princeton University Press.
  58. Miceli, M.; Schuessler, M.; and Yang, T. 2020. Between subjectivity and imposition: Power dynamics in data anno- tation for computer vision. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2): 1-25.
  59. Mim, N. J.; Nandi, D.; Khan, S. S.; Dey, A.; and Ahmed, S. I. 2024. In-Between Visuals and Visible: The Impacts of Text-to-Image Generative AI Tools on Digital Image- making Practices in the Global South. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '24. New York, NY, USA: Association for Computing Machinery. ISBN 9798400703300.
  60. Muller, M.; Wolf, C. T.; Andres, J.; Desmond, M.; Joshi, N. N.; Ashktorab, Z.; Sharma, A.; Brimijoin, K.; Pan, Q.; Duesterwald, E.; et al. 2021. Designing ground truth and the social life of labels. In Proceedings of the 2021 CHI conference on human factors in computing systems, 1-16.
  61. Myung, J.; Lee, N.; Zhou, Y.; Jin, J.; Putri, R. A.; Antypas, D.; Borkakoty, H.; Kim, E.; Perez-Almendros, C.; Ayele, A. A.; Gutiérrez-Basulto, V.; Ibáñez-García, Y.; Lee, H.; Muhammad, S. H.; Park, K.; Rzayev, A. S.; White, N.; Yi- mam, S. M.; Pilehvar, M. T.; Ousidhoum, N.; Camacho- Collados, J.; and Oh, A. 2025. BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages. arXiv:2406.09948. · arXiv
  62. Ortner, S. B. 1995. Resistance and the problem of ethno- graphic refusal. Comparative studies in society and history, 37(1): 173-193.
  63. Pawar, S.; Park, J.; Jin, J.; Arora, A.; Myung, J.; Yadav, S.; Haznitrama, F. G.; Song, I.; Oh, A.; and Augenstein, I. 2024. Survey of Cultural Awareness in Language Models: Text and Beyond. arXiv:2411.00860. · arXiv
  64. Qadri, R.; Davani, A. M.; Robinson, K.; and Prabhakaran, V. 2025. Risks of Cultural Erasure in Large Language Models. arXiv preprint arXiv:2501.01056. · arXiv
  65. Qadri, R.; Shelby, R.; Bennett, C. L.; and Denton, E. 2023. AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia. In Proceed- ings of the 2023 ACM Conference on Fairness, Account- ability, and Transparency, FAccT '23, 506-517. New York, NY, USA: Association for Computing Machinery. ISBN 9798400701924.
  66. Raman, P.; Harwood, J.; Weis, D.; Anderson, J. L.; and Miller, G. 2008. Portrayals of older adults in US and Indian magazine advertisements: A cross-cultural comparison. The Howard Journal of Communications, 19(3): 221-240.
  67. Riles, A. 2000. The Network Inside Out. University of Michigan Press.
  68. Sambasivan, N.; Arnesen, E.; Hutchinson, B.; Doshi, T.; and Prabhakaran, V. 2021. Re-imagining algorithmic fairness in india and beyond. In Proceedings of the 2021 ACM con- ference on fairness, accountability, and transparency, 315- 328.
  69. Scheuerman, M. K.; Wade, K.; Lustig, C.; and Brubaker, J. R. 2020. How we've taught algorithms to see identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human-computer In- teraction, 4(CSCW1): 1-35.
  70. Selbst, A. D.; Boyd, D.; Friedler, S. A.; Venkatasubrama- nian, S.; and Vertesi, J. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, accountability, and transparency, 59-68.
  71. Shelby, R.; Rismani, S.; Henne, K.; Moon, A.; Ros- tamzadeh, N.; Nicholas, P.; Yilla-Akbari, N.; Gallegos, J.; Smart, A.; Garcia, E.; et al. 2023. Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduc- tion. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 723-741.
  72. Shen, H.; Wang, L.; Deng, W. H.; Brusse, C.; Velgersdijk, R.; and Zhu, H. 2022. The model card authoring toolkit: Toward community-centered, deliberation-driven AI design. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 440-451.
  73. Singh, S.; Romanou, A.; Fourrier, C.; Adelani, D. I.; Ngui, J. G.; Vila-Suero, D.; Limkonchotiwat, P.; Marchisio, K.; Leong, W. Q.; Susanto, Y.; Ng, R.; Longpre, S.; Ko, W.- Y.; Smith, M.; Bosselut, A.; Oh, A.; Martins, A. F. T.; Choshen, L.; Ippolito, D.; Ferrante, E.; Fadaee, M.; Ermis, B.; and Hooker, S. 2024. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilin- gual Evaluation. arXiv:2412.03304. · arXiv
  74. Smith, J. J.; Satwani, A.; Burke, R.; and Fiesler, C. 2024. Recommend Me? Designing Fairness Metrics with Providers. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, 2389-2399. New York, NY, USA: Association for Comput- ing Machinery. ISBN 9798400704505.
  75. Soden, R.; Toombs, A.; and Thomas, M. 2024. Evaluating interpretive research in HCI. Interactions, 31(1): 38-42.
  76. Su, P.; Mao, W.; Zeng, D.; Li, X.; and Wang, F.-Y. 2009. Handling class imbalance problem in cultural modeling. In 2009 IEEE international conference on intelligence and se- curity informatics, 251-256. IEEE.
  77. Tsing, A. L. 2012. On nonscalability: The living world is not amenable to precision-nested scales. Common knowledge, 18(3): 505-524.
  78. Wallach, H.; Desai, M.; Cooper, A. F.; Wang, A.; Atalla, C.; Barocas, S.; Blodgett, S. L.; Chouldechova, A.; Corvi, E.; Dow, P. A.; et al. 2025. Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge. arXiv preprint arXiv:2502.00561. · arXiv
  79. Wan, Y.; Subramonian, A.; Ovalle, A.; Lin, Z.; Suvarna, A.; Chance, C.; Bansal, H.; Pattichis, R.; and Chang, K.- W. 2024. Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation. arXiv preprint arXiv:2404.01030. · arXiv
  80. Wang, A.; Bai, X.; Barocas, S.; and Blodgett, S. L. 2024a. Measuring machine learning harms from stereotypes: re- quires understanding who is being harmed by which errors in what ways. arXiv preprint arXiv:2402.04420. · arXiv
  81. Weidinger, L.; Raji, I. D.; Wallach, H.; Mitchell, M.; Wang, A.; Salaudeen, O.; Bommasani, R.; Ganguli, D.; Koyejo, S.; and Isaac, W. 2025. Toward an evaluation science for gen- erative AI systems. arXiv preprint arXiv:2503.05336. · arXiv
  82. Wolfe, R.; Dangol, A.; Howe, B.; and Hiniker, A. 2024. Rep- resentation Bias of Adolescents in AI: A Bilingual, Bicul- tural Study. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, 1621-1634.
  83. Zhang, L.; Liao, X.; Yang, Z.; Gao, B.; Wang, C.; Yang, Q.; and Li, D. 2024. Partiality and Misconception: Investigat- ing Cultural Representativeness in Text-to-Image Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 1-25.

Cited by

Sorted by citations

Authors
Qadri, R., Diaz, M., Wang, D., & Madaio, M.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
8
Issue
3
Pages
2067-2080
ISSN
3065-8365 (ISSN-L)
DOI
10.48550/arxiv.2503.19075
arXiv
2503.19075
OpenAlex
W4415195297
Cited by
6
References
0
Citations by year
2026: 4 · 2025: 2
FWCI
6.09
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Arts and Humanities › Philosophy › Cultural Identity and Representation
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
cultural representation, non-Western cultures, community co-design, South Asia, AI evaluation framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review