DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures
Open accessAbstract
Generative models are increasingly being used in various applications, such as text generation, commonsense reasoning, and question-answering. To be effective globally, these models must be aware of and account for local socio-cultural contexts, making it necessary to have benchmarks to evaluate the models for their cultural familiarity. Since the training data for LLMs is web-based and the Web is limited in its representation of information, it does not capture knowledge present within communities that are not on the Web. Thus, these models exacerbate the inequities, semantic misalignment, and stereotypes from the Web. There has been a growing call for community-centered participatory research methods in NLP. In this work, we respond to this call by using participatory research methods to introduce DOSA, the first community-generated Dataset of 615 Social Artifacts, by engaging with 260 participants from 19 different Indian geographic subcultures. We use a gamified framework that relies on collective sensemaking to collect the names and descriptions of these artifacts such that the descriptions semantically align with the shared sensibilities of the individuals from those cultures. Next, we benchmark four popular LLMs and find that they show significant variation across regional sub-cultures in their ability to infer the artifacts.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human-centered AI calls to actively involve people in AI design to ensure systems better reflect the values, preferences and needs of users and other impacted stakeholders.
“The domain of Human-centered AI HCAI calls to actively involve people in AI design to ensure systems better reflect the values, preferences and needs of users, and other impacted stakeholders (e.g., [4, 8, 16, 99, 101, 104, 110]).”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [99]
Where humans annotate data for AI image evaluation, their role commonly involves identifying objects or features in images, image comparisons, or gamified approaches.
“Where humans annotate data for AI image evaluation, their role commonly involves: identifying objects or features in images [52]; image comparisons to indicate similarities or preferences across two or more samples (e.g., [52, 85]); or gamified approaches (e.g., describing artifacts characteristics for another annotator to guess [99]).”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [99]
The inherent social nature of data production and the enjoyment from working 'together' towards a shared goal echoes findings from other community-oriented AI dataset specification and curation work.
“The inherent social nature of data production and the enjoyment from working 'together' towards a shared goal, echoes findings from other community-oriented AI dataset specification and curation work [32, 99, 107].”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [99]
In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation or allow community members to self-select their participation, albeit with some limitations.
“In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation (e.g., across geographical regions [52, 68, 91] and through local research partners [89]), or allow community members to selfselect their participation, albeit with some limitations (e.g., adults, higher-education levels or topic expertise, English-language ability”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [99]
Cites
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · see citing works
More human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329 · see citing works
Bias of AI-generated content: an examination of news produced by large language models
Fang, X., Che, S., Mao, M., Zhang, H., Zhao, M., & Zhao, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-55686-2 · Cited by 194 · see citing works
Jo, E., Epstein, D. A., Jung, H., & Kim, Y.-H. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581503 · Cited by 174 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation
Thieme, A., Faia Marques, R., Grayson, M., Balachandar, S., Tyler Cassidy, C., Choksi, M. Z., Longden, C., Huda, R. S., Ileve Kalovwe, N., Mallon, C., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790768 · Cited by 1 · see citing works
- Authors
- Seth, A., Ahuja, S., Bali, K., & Sitaram, S.
- Affiliations
- University of Michigan (US)
- Microsoft (United States) (US)
- Corresponding author
- Seth, Agrima
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2024
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4394742245
- Cited by
- 8
- References
- 4
- Citations by year
- 2026: 3 · 2025: 5
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Keywords
- social artifacts, cultural familiarity, large language models, LLM benchmarking, participatory research, collective sensemaking, cultural bias in AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
