CoChat
Sign Up
Cited by 48Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Ethics in Business and Education

STELA: a community-centred approach to norm elicitation for AI alignment

Open access
Bergman, Stevie, Marchal, Nahema, Mellor, John, Mohamed, Shakir, Gabriel, Iason, Isaac, William·2024·Scientific Reports
DOI 10.1038/s41598-024-56648-4PMC PMC10951208
Open this paper in CoChat — free accountOpen PDF

Abstract

Value alignment, the process of ensuring that artificial intelligence (AI) systems are aligned with human values and goals, is a critical issue in AI research. Existing scholarship has mainly studied how to encode moral values into agents to guide their behaviour. Less attention has been given to the normative questions of whose values and norms AI systems should be aligned with, and how these choices should be made. To tackle these questions, this paper presents the STELA process (SocioTEchnical Language agent Alignment), a methodology resting on sociotechnical traditions of participatory, inclusive, and community-centred processes. For STELA, we conduct a series of deliberative discussions with four historically underrepresented groups in the United States in order to understand their diverse priorities and concerns when interacting with AI systems. The results of our research suggest that community-centred deliberation on the outputs of large language models is a valuable tool for eliciting latent normative perspectives directly from differently situated groups. In addition to having the potential to engender an inclusive process that is robust to the needs of communities, this methodology can provide rich contextual insights for AI alignment.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-024-56648-4

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Approaches that enable communities to define and curate datasets remain underexplored.

    “Despite calls for more inclusive data-centric AI (e.g., [11, 31, 53, 96, 107]), approaches that enable communities to define and curate datasets remain underexplored.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [11]

    The Introduction calls for greater inclusion of historically marginalised communities in AI development, and the Methods say participatory approaches to AI alignment are still in their early stages. This supports the cited sentence’s general point about inclusive AI, but the paper discusses community-defined rules for AI behaviour, not communities defining or curating datasets. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Researchers have previously problematized anonymous crowdsourced data annotation approaches as being unspecific about who's worldviews and beliefs become manifest in the data.

    “Researchers have previously problematized anonymous crowdsourced data annotation approaches [31, 53, 65] as being unspecific about who's worldviews and beliefs become manifest in the data [11, 17, 51, 78, 90, 91].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. [4]Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A. & Gabriel, I. Ethical and social risks of harm from Language Models (arXiv:2112.04359). 10.48550/arXiv.2112.04359 (2021). · DOI · Paper page
  2. [5]Eubanks, V. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. (St. Martin’s Press, 2018). https://us.macmillan.com/books/9781250074317/automatinginequality.
  3. [6]Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C. & Venkatasubramanian, S. Runaway feedback loops in predictive policing. in (eds Friedler, S. A. & Wilson, C.), Proceedings of the 1st Conference on Fairness, Accountability and Transparency81, 160–171. PMLR. https://proceedings.mlr.press/v81/ensign18a.html (2018).
  4. [8]Bruhn JG The Sociology of Community Connections 2011 Springer
  5. [9]Raji, I. D. & Buolamwini, J. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. in Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 429–435. 10.1145/3306618.3314244 (2019). · DOI · Paper page
  6. [10]Field, A., Blodgett, S. L., Waseem, Z. & Tsvetkov, Y. A survey of race, racism, and anti-racism in NLP. in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 1905–1925. 10.18653/v1/2021.acl-long.149 (2021). · DOI
  7. [11]Santy, S., Liang, J., Le Bras, R., Reinecke, K. & Sap, M. NLpositionality: Characterizing design biases of datasets and models. in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9080–9102. 10.18653/v1/2023.acl-long.505 (2023). · DOI
  8. [12]Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K. & Wu, Y. Palm 2 technical report (arXiv:2305.10403). http://arxiv.org/abs/2305.10403 (2023). · arXiv
  9. [13]Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C. & Amodei, D. Language models are few-shot learners (arXiv:2005.14165). http://arxiv.org/abs/2005.14165 (2020). · arXiv
  10. [14]Lima, L., Furtado, V., Furtado, E. & Almeida, V. Empirical analysis of bias in voice-based personal assistants. in Companion Proceedings of The 2019 World Wide Web Conference, 533–538. 10.1145/3308560.3317597 (2019). · DOI
  11. [15]Mengesha Z Heldreth C Lahav M Sublewski J Tuennerman E “I don’t think these devices are very culturally sensitive.”—Impact of automated speech recognition errors on African Americans Front. Artif. Intell. 2021 10.3389/frai.2021.725911 34901836 PMC8664002 · DOI · Paper page
  12. [16]Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G. van den, Hendricks, L. A., Rauh, M., Huang, P.-S. & Irving, G. Scaling language models: Methods, analysis & insights from training gopher (arXiv:2112.11446). http://arxiv.org/abs/2112.11446 (2022). · arXiv
  13. [18]Birhane A Isaac W Prabhakaran V Diaz M Elish MC Gabriel I Mohamed S Power to the people? Opportunities and challenges for participatory AI Equity Access Algorithms, Mech., Optim. 2022 10.1145/3551624.3555290 · DOI · Paper page
  14. [19]Chambers R The origins and practice of participatory rural appraisal World Dev. 1994 22 7 953 969 10.1016/0305-750X(94)90141-4 · DOI · Paper page
  15. [20]Freire P Pedagogy of the Oppressed (50th Anniversary editon) 2018 Bloomsbury
  16. [21]Whyte W Participatory Action Research 1991 SAGE Publications Inc
  17. [22]Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A. & Procaccia, A. D. WeBuildAI: Participatory framework for algorithmic governance. in Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–35. 10.1145/3359283 (2019). · DOI
  18. [23]Katell, M., Young, M., Dailey, D., Herman, B., Guetler, V., Tam, A., Bintz, C., Raz, D. & Krafft, P. M. Toward situated interventions for algorithmic equity: Lessons from the field. in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 45–55. 10.1145/3351095.3372874 (2020). · DOI · Paper page
  19. [24]Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C. & Irving, G. Improving alignment of dialogue agents via targeted human judgements (arXiv:2209.14375). http://arxiv.org/abs/2209.14375 (2022). · arXiv
  20. [25]Gabriel I Ghazavi V Véliz C The challenge of value alignment: From fairer algorithms to AI safety The Oxford Handbook of Digital Ethics (1st edition) 2022 Oxford University Press
  21. [28]Bicchieri C Mercier H Xenitidou M Edmonds B Norms and Beliefs: How change occurs The Complexity of Social Norms 2014 Springer 37 54
  22. [29]Haidt J The Righteous Mind: Why Good People are Divided by Politics and Religion 2013 Penguin Books
  23. [30]Kroes P Verbeek P-P The Moral Status of Technical Artefacts 2014 Springer
  24. [35]Arun, C. AI and the global south: Designing for other worlds. Forthcoming in M. D. Dubber, F. Pasquale, and S. Das (Eds.), The Oxford Handbook of Ethics of AI. (Oxford University Press, 2019).
  25. [36]Prabhakaran, V., Qadri, R. & Hutchinson, B. Cultural incongruencies in artificial intelligence (arXiv:2211.13069). 10.48550/arXiv.2211.13069 (2022). · DOI · Paper page
  26. [37]Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J. & Kaplan, J. A general language assistant as a laboratory for alignment (arXiv:2112.00861). 10.48550/arXiv.2112.00861 (2021). · DOI · Paper page
  27. [43]Solaiman, I. & Dennison, C. Process for adapting language models to society (PALMS) with values-targeted datasets (arXiv:2106.10328). 10.48550/arXiv.2106.10328 (2021). · DOI · Paper page
  28. [45]Fletcher-Watson S Adams J Brook K Charman T Crane L Cusack J Leekam S Milton D Parr JR Pellicano E Making the future together: Shaping autism research through meaningful participation Autism 2019 23 4 943 953 10.1177/1362361318786721 30095277 PMC6512245 · DOI · Paper page
  29. [46]Scott J A Dictionary of Sociology 2014 4 Oxford University Press
  30. [47]Tönnies F Community and Society 2017 Routledge
  31. [50]van der Veer SN Riste L Cheraghi-Sohi S Phipps DL Tully MP Bozentko K Atwood S Hubbard A Wiper C Oswald M Peek N Trading off accuracy and explainability in AI decision-making: Findings from 2 citizens’ juries J. Am. Med. Inf. Assoc. 2021 28 10 2128 2138 10.1093/jamia/ocab127 PMC8522832 34333646 · DOI
  32. [51]Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S. & Amodei, D. Deep reinforcement learning from human preferences (arXiv:1706.03741). 10.48550/arXiv.1706.03741 (2023). · DOI · Paper page
  33. [52]Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D. & Christiano, P. Learning to summarize from human feedback (arXiv:2009.01325). http://arxiv.org/abs/2009.01325 (2022). · arXiv
  34. [54]van de Poel I Michelfelder DP McCarthy N Goldberg DE Translating Values into Design Requirements Philosophy and Engineering: Reflections on Practice, Principles and Process 2013 Springer 253 266
  35. [55]Martin Jr., D., Prabhakaran, V., Kuhlberg, J., Smart, A. & Isaac, W. S. Participatory problem formulation for fairer machine learning through community-based system dynamics (arXiv:2005.07572). 10.48550/arXiv.2005.07572 (2020). PMC7762892 33390696 · DOI
  36. [57]Prabhakaran V Martin D Jr Participatory machine learning using community-based system dynamics Health Hum. Rights 2020 22 2 71 33390696 PMC7762892
  37. [59]Weidinger L McKee KR Everett R Huang S Zhu TO Chadwick MJ Summerfield C Gabriel I Using the veil of ignorance to align AI systems with principles of justice Proc. Natl. Acad. Sci. 2023 120 18 e2213709120 10.1073/pnas.2213709120 37094137 PMC10160973 · DOI
  38. [62]Bakker, M. A., Chadwick, M. J., Sheahan, H. R., Tessler, M. H., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M. M. & Summerfield, C. Fine-tuning language models to find agreement among humans with diverse preferences (arXiv:2211.15006). http://arxiv.org/abs/2211.15006 (2022). · arXiv
  39. [63]Chambers S Reasonable Democracy: Jürgen Habermas and the Politics of Discourse 1996 Cornell University Press
  40. [64]Fishkin JS Luskin RC Experimenting with a democratic ideal: Deliberative polling and public opinion Acta Politica 2005 40 3 284 298 10.1057/palgrave.ap.5500121 · DOI
  41. [65]Rothwell E Anderson R Botkin JR Deliberative discussion focus groups Qual. Health Res. 2016 26 6 734 740 10.1177/1049732315591150 26078330 PMC5847289 · DOI
  42. [66]Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. Red teaming language models with language models (arXiv:2202.03286). 10.48550/arXiv.2202.03286 (2022). · DOI
  43. [69]Rauh M Mellor J Uesato J Huang P-S Welbl J Weidinger L Dathathri S Glaese A Irving G Gabriel I Isaac W Hendricks LA Characteristics of harmful text: Towards rigorous benchmarking of language models Adv. Neural Inf. Proc. Syst. 2022 35 24720 24739
  44. [70]Shen, T., Jin, R., Huang, Y., Liu, C., Dong, W., Guo, Z., Wu, X., Liu, Y. & Xiong, D. Large language model alignment: A survey (arXiv:2309.15025). http://arxiv.org/abs/2309.15025 (2023). · arXiv
  45. [71]Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B. & Huang, P. Challenges in detoxifying language models (arXiv:2109.07445v1). https://arxiv.org/abs/2109.07445 (2021). · arXiv
  46. [72]Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., DasSarma, N., Drain, D., Elhage, N., El-Showk, S., Fort, S. & Clark, J. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned (arXiv:2209.07858v2). https://arxiv.org/abs/2209.07858 (2022). · arXiv
  47. [73]Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. de L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. van den, Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E. & Sifre, L. Training compute-optimal large language models (arXiv:2203.15556). 10.48550/arXiv.2203.15556 (2022). · DOI · Paper page
  48. [75]The British Psychological Society. Ethics guidelines for internet-mediated research. British Psychological Society. 10.53841/bpsrep.2021.rep155 (2021). · DOI
  49. [76]Morgan DL Focus Groups as Qualitative Research 1996 SAGE Publications
  50. [77]Onwuegbuzie AJ Dickinson WB Leech NL Zoran AG A qualitative framework for collecting and analyzing data in focus group research Int. J. Qual. Methods 2009 8 3 1 21 10.1177/160940690900800301 · DOI · Paper page
  51. [78]Hughes DL DuMont K Revenson TA D’Augelli AR French SE Hughes DL Livert D Seidman E Shinn M Yoshikawa H Using Focus Groups to Facilitate Culturally Anchored Research Ecological Research to Promote Social Change: Methodological Advances from Community Psychology 2002 Springer 257 289
  52. [79]Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S. & Bikel, D. Llama 2: Open foundation and fine-tuned chat models. (arXiv:2307.09288). https://arxiv.org/abs/2307.09288 (2023). · arXiv
  53. [81]Miles MB Huberman AM Qualitative Data Analysis: An Expanded Sourcebook 1994 2 Sage Publications Inc
  54. [82]Ferri, G. & Gloerich, I. Risk and harm: Unpacking ideologies in the AI discourse. Proceedings of the 5th International Conference on Conversational User Interfaces, 1–6. 10.1145/3571884.3603751 (2023). · DOI
  55. [83]Jakesch, M., Buçinca, Z., Amershi, S. & Olteanu, A. How different groups prioritize ethical values for responsible AI. in 2022 ACM Conference on Fairness, Accountability, and Transparency, 310–323. 10.1145/3531146.3533097 (2022). · DOI · Paper page
  56. [84]Tronto JC Moral Boundaries: A Political Argument for an Ethic of Care 2020 1 Routledge
  57. [85]Collins PH The social construction of black feminist thought Signs 1989 14 4 745 773 10.1086/494543 · DOI
  58. [86]Crenshaw K On Intersectionality: Essential Writings 2019 New Press
  59. [87]Spivak, G. C. Can the Subaltern Speak? Reflections on the History of an Idea (eds Morris, R. C.). (Columbia University Press, 2010). (Original Work Published 1988).
  60. [88]Rottger, P., Vidgen, B., Hovy, D. & Pierrehumbert, J. Two contrasting data annotation paradigms for subjective NLP tasks. in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 175–190. 10.18653/v1/2022.naacl-main.13 (2022). · DOI
  61. [89]Uma A Almanea D Poesio M Scaling and disagreements: Bias, noise, and ambiguity Front. Artif. Intell. 2022 10.3389/frai.2022.818451 35434607 PMC9012579 · DOI
  62. [92]Gadiraju, V., Kane, S., Dev, S., Taylor, A., Wang, D., Denton, E. & Brewer, R. “I wouldn’t say offensive but...”: Disability-centered perspectives on large language models. in 2023 ACM Conference on Fairness, Accountability, and Transparency, 205–216. 10.1145/3593013.3593989 (2023). · DOI · Paper page

Cited by

Sorted by citations

  • Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation

    Thieme, A., Faia Marques, R., Grayson, M., Balachandar, S., Tyler Cassidy, C., Choksi, M. Z., Longden, C., Huda, R. S., Ileve Kalovwe, N., Mallon, C., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790768 · Cited by 1 · see citing works

    “This has led to calls for more inclusive, community-centric AI evaluation approaches [11, 68] that leverage qualitative methods like focus groups [89] and online workshops [68] to capture nuanced, socially situated and dynamic aspects of concepts like cultural or disability representation; as well…”
    Influential

Authors
Bergman, S., Marchal, N., Mellor, J., Mohamed, S., Gabriel, I., & Isaac, W.
Affiliations
Google DeepMind (United Kingdom) (GB)
Corresponding author
Marchal, Nahema
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2024-03-19
Volume
14
Issue
1
Pages
6616-6616
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-024-56648-4
PMC
PMC10951208
OpenAlex
W4392939433
Cited by
48
References
14
Citations by year
2026: 17 · 2025: 22 · 2024: 9
FWCI
19.89
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Decision Sciences › Information Systems and Management › Ethics in Business and Education
Keywords
AI alignment, value alignment, participatory design, socio-technical approach, underrepresented groups, large language models, human values, inclusive design
MeSH terms
Artificial Intelligence, Humans, Language, Morals, Rest

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review