The Values Encoded in Machine Learning Research
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Machine learning currently exerts an outsized influence on the world, increasingly affecting institutional practices and impacted communities. It is therefore critical that we question vague conceptions of the field as value-neutral or universally beneficial, and investigate what specific values the field is advancing. In this paper, we first introduce a method and annotation scheme for studying the values encoded in documents such as research papers. Applying the scheme, we analyze 100 highly cited machine learning papers published at premier machine learning conferences, ICML and NeurIPS. We annotate key features of papers which reveal their values: their justification for their choice of project, which attributes of their project they uplift, their consideration of potential negative consequences, and their institutional affiliations and funding sources. We find that few of the papers justify how their project connects to a societal need (15%) and far fewer discuss negative potential (1%). Through line-by-line content analysis, we identify 59 values that are uplifted in ML research, and, of these, we find that the papers most frequently justify and assess themselves based on Performance, Generalization, Quantitative evidence, Efficiency, Building on past work, and Novelty. We present extensive textual evidence and identify key themes in the definitions and operationalization of these values. Notably, we find systematic textual evidence that these top values are being defined and applied with assumptions and implications generally supporting the centralization of power. Finally, we find increasingly close ties between these highly cited papers and tech companies and elite universities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a study on the values encoded in the ML research literature, Birhane et al. identified a number of ethical values [45], from which the papers would draw motivation.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [45]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we start from the premise that these AI writing tools are both flawed and fallible, due to their algorithmic biases and technical shortcomings [9], but that they also have powerful affordances, including the ability to assist the large numbers of scientists and engineers around the world who need to publish and present their work in English but are not native speakers of the language.”
From Incorporating generative AI into a writing-intensive undergraduate course without off-loading learning (2025) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Benchmarks set the agenda and orient progress: we should aspire for holistic, pluralistic, and democratic benchmarks.3”
From Holistic Evaluation of Language Models (2023) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“In a total of 100 highly cited papers, the study found that performance was clearly the most frequent value and pointed out that performance is not a neutral term but comes with ethical implications.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [45]
The cited source says what the claim attributes to it.
In the citing paper
“They note that most of these focus on endogenous notions of technical performance and novelty, and leave out considerations of broader context and impact that are necessary to shaping a governance effort.”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [18]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
Qualitative evaluation and research methods
Ellis, D. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,943
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
(2005) · PubMed · Cited by 7,011
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Sorting Things Out: Classification and Its Consequences
(2000) · The Journal of Academic Librarianship · doi:10.1016/s0099-1333(00)00155-5 · Cited by 5,210
How to plan and perform a qualitative study using content analysis
Bengtsson, M. (2016) · NursingPlus Open · doi:10.1016/j.npls.2016.01.001 · Cited by 4,616 · cited 3×
“, LWPR [8], standard GPR [1], sparse online Gaussian process regression (OGP) [5] and υ-support vector regression (υ-SVR) [11],”
BackgroundMethodRigor in Qualitative Research: The Assessment of Trustworthiness
Krefting, L. (1991) · American Journal of Occupational Therapy · doi:10.5014/ajot.45.3.214 · Cited by 4,174 · cited 1×
“We adhere to a number of best practices to establish reliability: We practice prolonged engagement, conducting long-term orientation to and analysis of data over more than a year (in lieu of short-term analysis that is dominated by preconceptions) [40]; We triangulate across researchers (six researchers) and points in…”
Method
Research works printed but not linked: 10
- 7. Samy Bengio and Deborah Raji. 2021. A Retrospective on the NeurIPS 2021 Ethics Review Process. https://blog.neurips.cc/2021/12/03/a-retrospective-on-the- neurips-2021-ethics-review-process/
- 10. Bruce L. Berg and Howard Lune. 2017. Qualitative research methods for the social sciences (ninth edition ed.). Pearson.
- 16. Kate Crawford. 2017. The Trouble with Bias. (2017). NeurIPS Keynote.
- 18. Catherine D'Ignazio and Lauren F Klein. 2020. Data Feminism. MIT Press.
- 23. Barney G. Glaser and Anselm L. Strauss. 1999. The discovery of grounded theory: strategies for grounded research. Aldine de Gruyter.
- 24. Ben Green. 2019. 'Good' isn't Good Enough. In NeurIPS Joint Workshop on AI for Social Good.
- 33. Maurice G Kendall and B Babington Smith. 1939. The problem of m rankings. The annals of mathematical statistics 10, 3 (1939), 275-287.
- 40. Yvonna S. Lincoln and Egon G. Guba. 2006. Naturalistic inquiry. Sage Publ.
- 51. Frank Pasquale. 2015. The black box society. Harvard University Press.
- 57. Anna Rogers. 2019. Peer review in NLP: reject-if-not-SOTA. Hacking Semantics blog (2019). https://hackingsemantics.xyz/2020/reviewing-models/#everything- wrong-with-reject-if-not-sota
Cited by
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 633
British Journal of Social Psychology
(2010) · British Journal of Social Psychology · doi:10.1111/(issn)2044-8309 · Cited by 558
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
“The Test and Evaluation stage is continuous throughout the entire AI Development Lifecycle.”
MethodInfluentialHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
The future of AI and education: Some cautionary notes
Selwyn, N. (2022) · European Journal of Education · doi:10.1111/ejed.12532 · Cited by 401
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
BloombergGPT: A Large Language Model for Finance
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., & Mann, G. (2023) · arXiv · Cited by 357
Eight Things to Know about Large Language Models
Bowman, S. R. (2024) · Critical AI · doi:10.1215/2834703x-11556011 · Cited by 122
Power to the People? Opportunities and Challenges for Participatory AI
Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022) · Equity and Access in Algorithms Mechanisms and Optimization · doi:10.1145/3551624.3555290 · Cited by 330
“The core motivation and objectives of this form of participation are somewhat incompatible with the core values of AI research, which include efficiency, performance, accuracy, novelty, and beating state-of-the-art [13].”
BackgroundSelf-citationOECD (2025) · OECD eBooks · doi:10.1787/795de142-en · Cited by 329
Gkintoni, E., Antonopoulou, H., Sortwell, A., & Halkiopoulos, C. (2025) · Brain Sciences · doi:10.3390/brainsci15020203 · Cited by 309
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“This work requires thinking critically about the distribution of benefits and harms of algorithmic systems [33, 169] and existing social hierarchies [35].”
BackgroundVQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., & Raff, E. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19836-6_6 · Cited by 279
Measuring the Carbon Intensity of AI in Cloud Instances
Dodge, J., Prewitt, T., Tachet des Combes, R., Odmark, E., Schwartz, R., Strubell, E., Luccioni, A. S., Smith, N. A., DeCario, N., & Buchanan, W. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533234 · Cited by 257
“When considering the net impacts of an AI application, it is imperative to determine the extent to which AI is incentivizing practices that have a negative impact on the environment, or the extent to which applications are directly reducing emissions or otherwise incentivizing practices that are beneficial to the clim…”
BackgroundCollective Constitutional AI: Aligning a Language Model with Public Input
Huang, S., Siddarth, D., Lovitt, L., Liao, T. I., Durmus, E., Tamkin, A., & Ganguli, D. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658979 · Cited by 76
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Putting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“reflects a deep value of the field [14], which prioritizes optimizing decision outcomes rather than the experience of human decision makers.”
BackgroundGhosh, S., & Caliskan, A. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604672 · Cited by 112
Measuring the environmental impacts of artificial intelligence compute and applications
OECD (2022) · OECD digital economy papers · doi:10.1787/7babf571-en · Cited by 189
Ethics and diversity in artificial intelligence policies, strategies and initiatives
Roche, C., Wall, P. J., & Lewis, D. (2023) · AI and Ethics · doi:10.1007/s43681-022-00218-9 · Cited by 179
“…to be inherently value-laden, moreover, it is considered to be “socially and politically loaded, frequently neglecting societal needs and harms, while prioritizing and promoting the concentration of resources, tools, knowledge, and power in the hands of already powerful actors ” [28, p. 182].”
BackgroundHagendorff, T. (2022) · AI and Ethics · doi:10.1007/s43681-021-00122-8 · Cited by 103
Artificial intelligence in government: Concepts, standards, and a unified framework
Straub, V. J., Morgan, D., Bright, J., & Margetts, H. Z. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101881 · Cited by 176
Rohde, F., Wagner, J., Meyer, A., Reinhard, P., Voß, M., Petschow, U., & Mollen, A. (2024) · Current Opinion in Environmental Sustainability · doi:10.1016/j.cosust.2023.101411 · Cited by 171
International AI Safety Report 2026Preprint
Bengio, Y., Clare, S., Prunkl, C., Andriushchenko, M., Bucknall, B., Murray, M., Bommasani, R., Casper, S., Davidson, T., Douglas, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.21012 · Cited by 177
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. J. L., Reinecke, K., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.505 · Cited by 166
Counting Carbon: A Survey of Factors Influencing the Emissions of Machine LearningPreprint
Luccioni, A. S., & Hernández-García, Á. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.08476 · Cited by 162
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160
Self-citationPythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160
Reclaiming AI as a Theoretical Tool for Cognitive Science
van Rooij, I., Guest, O., Adolfi, F., de Haan, R., Kolokolova, A., & Rich, P. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00217-5 · Cited by 97
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
“As there are several serious, although gradual, and pernicious harms emanating from AI – for instance, the chilling effect of surveillance, or corporate power concentration [182] – it is clear that not all pressing issues regarding the technology can be addressed with audits.”
BackgroundSelf-citationLost in Translation: Large Language Models in Non-English Content AnalysisPreprint
Nicholas, G., & Bhatia, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07377 · Cited by 159
Evaluation Gaps in Machine Learning Practice
Hutchinson, B., Rostamzadeh, N., Greer, C., Heller, K., & Prabhakaran, V. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533233 · Cited by 62
A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl
Baack, S. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659033 · Cited by 153
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
“…AI policy guidelines (Jobin et al., 2019), sociotechnical harms of algorithmic systems (Shelby et al., 2023), values in machine learning research (Birhane et al., 2021), risks of specific applications like language models (Weidinger et al., 2022), occurrences of harmful machine behavior (Park et…”
A Research Ethics Framework for the Clinical Translation of Healthcare Machine Learning
McCradden, M. D., Anderson, J. A., A. Stephenson, E., Drysdale, E., Erdman, L., Goldenberg, A., & Zlotnik Shaul, R. (2022) · The American Journal of Bioethics · doi:10.1080/15265161.2021.2013977 · Cited by 114
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137
“…arise with the development of deployment of large-scale AI systems, including issues around environmental impacts, access, mis-use, privacy, safety, and the concentration of power (Amodei et al., 2016; Bender et al., 2021; Bommasani et al., 2021; Birhane et al., 2022; Weidinger et al., 2022, i.a.).”
Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122
How Different Groups Prioritize Ethical Values for Responsible AI
Jakesch, M., Buçinca, Z., Amershi, S., & Olteanu, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533097 · Cited by 116
Collaborative Intelligence: A Scoping Review Of Current Applications
Schleiger, E., Mason, C., Naughtin, C. K., Reeson, A. F., & Paris, C. L. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2327890 · Cited by 117
Robots Enact Malignant Stereotypes
Hundt, A., Agnew, W. S., Zeng, V., Kacianka, S., & Gombolay, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533138 · Cited by 104
Self-citationDeng, W. H., Yildirim, N., Chang, M., Eslami, M., Holstein, K., & Madaio, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594037 · Cited by 57
Large models of what? Mistaking engineering achievements for human linguistic agency
Birhane, A., & McGann, M. (2024) · Language Sciences · doi:10.1016/j.langsci.2024.101672 · Cited by 99
Self-citationUnsolved Problems in ML Safety
Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2109.13916 · Cited by 95
“Beyond technical work, policy and governance work will be integral to safe deployment [43, 12, 16, 210, 25].”
BackgroundRahm, L., & Rahm‐Skågeby, J. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13319 · Cited by 57
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 93
“, 2021) and value-driven (Birhane et al., 2021) endeavor.”
BackgroundMethodInfluentialThe Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Fleisig, E., Blodgett, S. L., Klein, D., & Talat, Z. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.126 · Cited by 93
Black-Box Access is Insufficient for Rigorous AI Audits
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659037 · Cited by 86
Model Multiplicity: Opportunities, Concerns, and Solutions
Black, E., Raghavan, M., & Barocas, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533149 · Cited by 85
“This focus is pervasive throughout machine learning scholarship and practice [8], perhaps best evidenced by the Common Task Framework [27], through which independent researchers compare predictive performance on common datasets.”
BackgroundStereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models
Ungless, E. L., Roß, B., & Lauscher, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.502 · Cited by 84
Harvey, E., Koenecke, A., & Kizilcec, R. F. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713210 · Cited by 86
Collaborative Diffusion: Boosting Designerly Co-Creation with Generative AI
Verheijden, M. P., & Funk, M. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3585680 · Cited by 83
“Related work on machine learning often favors benefts for research and industry, thereby overlooking the potential negative societal implications [2].”
BackgroundExplainable Artificial Intelligence (XAI): Adoption and Advocacy
Ridley, M. (2022) · Information Technology and Libraries · doi:10.6017/ital.v41i2.14683 · Cited by 82
Public procurement of artificial intelligence systems: new risks and future proofing
Hickok, M. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01572-2 · Cited by 81
Rakova, B., & Dobbe, R. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594014 · Cited by 39
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 75
“[18] review recent literature in ML to identify values that are typically put forward to motivate work in that field.”
BackgroundTowards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75
LLMs differ from human cognition because they are not embodied
Chemero, A. P. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01723-5 · Cited by 75
Multimodal Objects Categorization by Fusing GMM and Multi-layer Perceptron
Naseer, A., & Jalal, A. (2024) · International Conference on Advancements in Computational Sciences (ICACS) · doi:10.1109/icacs60934.2024.10473242 · Cited by 74
Diaz, F., & Madaio, M. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31641 · Cited by 72
Earth for AI: A Political Ecology of Data-Driven Climate Initiatives
Nost, E., & Colven, E. (2022) · Geoforum · doi:10.1016/j.geoforum.2022.01.016 · Cited by 71
Reimagining the machine learning life cycle to improve educational outcomes of students
Liu, L. T., Wang, S., Britton, T., & Abebe, R. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2204781120 · Cited by 70
The Impact and Opportunities of Generative AI in Fact-Checking
Wolfe, R., & Mitra, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658987 · Cited by 40
Warmth and competence in human-agent cooperation
McKee, K. R., Bai, X., & Fiske, S. T. (2024) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-024-09649-6 · Cited by 69
On Hate Scaling Laws For Data-SwampsPreprint
Birhane, A., Prabhu, V. U., Han, S., & Boddeti, V. N. (2023) · arXiv.org · doi:10.48550/arxiv.2306.13141 · Cited by 48
Self-citationComputer-vision research powers surveillance technology
Kalluri, P., Agnew, W. S., Cheng, M., Owens, K., Soldaini, L., & Birhane, A. (2025) · Nature · doi:10.1038/s41586-025-08972-6 · Cited by 69
Self-citationWhat do academics say about artificial intelligence ethics? An overview of the scholarship
Bakıner, O. (2022) · AI and Ethics · doi:10.1007/s43681-022-00182-4 · Cited by 68
Explainable machine learning practices: opening another black box for reliable medical AI
Ratti, E., & Graves, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00141-z · Cited by 61
“[3], that provided a list of 67 values which influence to various degrees ML research, including technical values, and van de Poel [35] who formalizes technical norms within sociotechnical AI systems.”
BackgroundSquare One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold
Ruder, S., Vulić, I., & Søgaard, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.184 · Cited by 62
Towards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533118 · Cited by 39
Saxena, D., Moon, E. S.-Y., Chaurasia, A., Guan, Y., & Guha, S. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581308 · Cited by 34
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Talat, Z., Névéol, A., Biderman, S., Clinciu, M., Dey, M., Longpre, S., Luccioni, S., Masoud, M., Mitchell, M., Radev, D., et al. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.3 · Cited by 59
The Illusion of Artificial Inclusion
Agnew, W., Bergman, A. S., Chien, J., Díaz, M., El-Sayed, S., Pittman, J., Mohamed, S., & McKee, K. R. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642703 · Cited by 58
Self-citationInsolvent: How to Reorient Computing for Just Sustainability
Becker, C. (2023) · doi:10.7551/mitpress/14668.001.0001 · Cited by 56
Social Inclusion in Curated Contexts: Insights from Museum Practices
Huang, H.-Y., & Liem, C. C. S. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533095 · Cited by 56
Learning about Responsible AI On-The-Job: Learning Pathways, Orientations, and Aspirations
Madaio, M., Kapania, S., Qadri, R., Wang, D., Zaldivar, A., Denton, E., & Wilcox, L. G. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658988 · Cited by 56
Automating Ambiguity: Challenges and Pitfalls of Artificial IntelligencePreprint
Birhane, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.04179 · Cited by 57
Self-citationGhosh, S., & Caliskan, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10510 · Cited by 55
ANALISIS DATA KUALITATIF: MANUAL DATA ANALISIS PROSEDUR
Rofiah, C., & Bungin, B. (2024) · Develop · doi:10.25139/dev.v8i1.7319 · Cited by 54
Time for a Pause: Without Effective Public Oversight, AI in Schools Will Do More Harm Than Good.
Molnar, A., Boninger, F., & Williamson, B. (2024) · CU Scholar (University of Colorado Boulder) · Cited by 56
Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates
Davani, A. M., Díaz, M., Baker, D., & Prabhakaran, V. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659021 · Cited by 53
Varanasi, R. A., & Goyal, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580903 · Cited by 52
Public Opinion toward Artificial Intelligence
Zhang, B. (2023) · The Oxford Handbook of AI Governance · doi:10.1093/oxfordhb/9780197579329.013.36 · Cited by 34
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Smith, J. J., Amershi, S., Barocas, S., Wallach, H., & Vaughan, J. W. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533122 · Cited by 52
Sociotechnical Safety Evaluation of Generative AI Systems
Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11986 · Cited by 49
Disambiguating Algorithmic Bias: From Neutrality to Justice
Edenberg, E., & Wood, A. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604695 · Cited by 46
Moral Foundations of Large Language Models
Abdulhai, M., Serapio-García, G., Crepy, C., Valter, D., Canny, J., & Jaques, N. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.982 · Cited by 45
Going public: the role of public participation approaches in commercial AI labs
Groves, L., Peppin, A., Strait, A., & Brennan, J. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594071 · Cited by 45
Topics, Authors, and Institutions in Large Language Model Research: Trends from 17K arXiv Papers
Movva, R., Balachandar, S., Peng, K., Agostini, G., Garg, N., & Pierson, E. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.67 · Cited by 45
Stop! In the Name of Flaws: Disentangling Personal Names and Sociodemographic Attributes in NLP
Gautam, V., Subramonian, A., Lauscher, A., & Keyes, O. (2024) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2024.gebnlp-1.20 · Cited by 45
Geographies of artificial intelligence: Labor, surveillance, and activism
Walker, M. A., & Winders, J. (2023) · Human Geography · doi:10.1177/19427786231208458 · Cited by 45
Ethical considerations and statistical analysis of industry involvement in machine learning research
Hagendorff, T., & Meding, K. (2021) · AI & Society · doi:10.1007/s00146-021-01284-z · Cited by 44
Sachdeva, P., Barreto, R., von Vacano, C., & Kennedy, C. J. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533216 · Cited by 44
Black, E., Naidu, R. C. A., Ghani, R., Rodolfa, K. T., Ho, D. E., & Heidari, H. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623259 · Cited by 44
London, A. J., & Heidari, H. (2024) · Minds and Machines · doi:10.1007/s11023-024-09696-8 · Cited by 41
Gururaja, S., Bertsch, A., Na, C., Widder, D. G., & Strubell, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.822 · Cited by 41
The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who
Schlichtkrull, M., Ousidhoum, N. D., & Vlachos, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.577 · Cited by 41
Metadata
- Authors
- Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M.
- Affiliations
- University College Dublin (IE)
- Stanford University (US)
- University of Michigan (US)
- University of Washington (US)
- University of Pittsburgh (US)
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 173-184
- OpenAlex
- W3174220540
- Cited by
- 317
- References
- 69
- Citations by year
- 2026: 72 · 2025: 66 · 2024: 63 · 2023: 70 · 2022: 36
- FWCI
- 58.38
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Science Foundation Ireland (13/RC/2094-2)
- National Defense Science and Engineering Graduate
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- value neutrality, research ethics, research funding sources, institutional affiliation, societal impact of AI, novelty, centralization of power, NeurIPS
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.