CoChat
Sign Up
PreprintCited by 317Ethics and Social Impacts of AIComputational and Text Analysis Methodsscientometrics and bibliometrics research

The Values Encoded in Machine Learning Research

Open access
Birhane, Abeba, Kalluri, Pratyusha, Card, Dallas, Agnew, William, Dotan, Ravit, Bao, Michelle·2021·arXiv
DOI 10.48550/arxiv.2106.15590arXiv 2106.15590
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning currently exerts an outsized influence on the world, increasingly affecting institutional practices and impacted communities. It is therefore critical that we question vague conceptions of the field as value-neutral or universally beneficial, and investigate what specific values the field is advancing. In this paper, we first introduce a method and annotation scheme for studying the values encoded in documents such as research papers. Applying the scheme, we analyze 100 highly cited machine learning papers published at premier machine learning conferences, ICML and NeurIPS. We annotate key features of papers which reveal their values: their justification for their choice of project, which attributes of their project they uplift, their consideration of potential negative consequences, and their institutional affiliations and funding sources. We find that few of the papers justify how their project connects to a societal need (15\%) and far fewer discuss negative potential (1\%). Through line-by-line content analysis, we identify 59 values that are uplifted in ML research, and, of these, we find that the papers most frequently justify and assess themselves based on Performance, Generalization, Quantitative evidence, Efficiency, Building on past work, and Novelty. We present extensive textual evidence and identify key themes in the definitions and operationalization of these values. Notably, we find systematic textual evidence that these top values are being defined and applied with assumptions and implications generally supporting the centralization of power.Finally, we find increasingly close ties between these highly cited papers and tech companies and elite universities.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.15590

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent work on the values encoded in current approaches to research in NLP and ML more broadly finds that narrow definitions of performance and efficiency were often prioritized over considerations of social impact in research and development.

    “Motivated by recent work on the values encoded in current approaches to research in NLP and ML more broadly (Leahy and Biderman, 2021; Birhane et al., 2021), which finds that narrow definitions of performance and efficiency were often prioritized over considerations of social impact in research and development.”

    From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Birhane et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 50

  1. Mohsen Abbasi, Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkata- subramanian. 2019. Fairness in Representation: Quantifying Stereotyping as a Representational Harm. In Proceedings of the 2019 SIAM International Conference on Data Mining.
  2. Mohamed Abdalla and Moustafa Abdalla. 2021. The Grey Hoodie Project: Big Tobacco, Big Tech, and the Threat on Academic Integrity. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/ 3461702.3462563 · DOI
  3. Grace Abuhamad and Claudel Rheault. 2020. Like a Researcher Stating Broader Impact For the Very First Time. arXiv preprint arXiv:2011.13032 (2020). · arXiv
  4. Nur Ahmed and Muntasir Wahed. 2020. The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research. arXiv preprint arXiv:2010.15581 (2020). · arXiv
  5. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of FAccT (2021).
  6. Samy Bengio and Deborah Raji. 2021. A Retrospective on the NeurIPS 2021 Ethics Review Process. https://blog.neurips.cc/2021/12/03/a-retrospective-on-the- neurips-2021-ethics-review-process/
  7. Mariette Bengtsson. 2016. How to plan and perform a qualitative study using content analysis. NursingPlus Open 2 (2016), 8-14.
  8. Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Wiley.
  9. Bruce L. Berg and Howard Lune. 2017. Qualitative research methods for the social sciences (ninth edition ed.). Pearson.
  10. Abeba Birhane. 2020. Algorithmic Colonization of Africa. SCRIPTed 17, 2 (2020).
  11. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Lan- guage (Technology) is Power: A Critical Survey of "Bias" in NLP. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.485 · DOI · Paper page
  12. Geoffrey C. Bowker and Susan Leigh Star. 2000. Sorting things out: Classification and its consequences. MIT press.
  13. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the Conference on Fairness, Accountability and Transparency.
  14. Sasha Costanza-Chock. 2018. Design Justice, AI, and Escape from the Matrix of Domination. Journal of Design and Science (2018).
  15. Kate Crawford. 2017. The Trouble with Bias. (2017). NeurIPS Keynote.
  16. Norman K Denzin. 2017. Sociological methods: a sourcebook. McGraw-Hill.
  17. Catherine D'Ignazio and Lauren F Klein. 2020. Data Feminism. MIT Press.
  18. Ravit Dotan and Smitha Milli. 2019. Value-Laden Disciplinary Shifts in Machine Learning. arXiv preprint arXiv:1912.01172 (2019). · arXiv
  19. Louise Doyle, Catherine McCabe, Brian Keogh, Annemarie Brady, and Margaret McCann. 2020. An overview of the qualitative descriptive design within nursing research. Journal of Research in Nursing 25, 5 (Aug 2020), 443-455. https: //doi.org/10.1177/1744987119880234 · DOI
  20. Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the Eye of the User: A Critique of NLP Leaderboards. In Proceedings of EMNLP.
  21. Luciano Floridi and Josh Cowls. 2019. A Unified Framework of Five Principles for AI in Society. Harvard Data Science Review 1, 1 (2019).
  22. Ben Green. 2019. 'Good' isn't Good Enough. In NeurIPS Joint Workshop on AI for Social Good.
  23. Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M. Branham. 2018. Gender Recognition or Gender Reductionism? The Social Implications of Embedded Gender Recognition Systems. In Proceedings CHI.
  24. Alex Hanna and Tina M. Park. 2020. Against Scale: Provocations and Resistances to Scale Thinking. arXiv preprint arXiv:2010.08850 (2020). · arXiv
  25. Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In Proceedings of CHI.
  26. Hsiu-Fang Hsieh and Sarah E. Shannon. 2005. Three Approaches to Qualitative Content Analysis. Qualitative Health Research 15, 9 (2005), 1277-1288. https: //doi.org/10.1177/1049732305276687 · DOI · Paper page
  27. John P. A. Ioannidis. 2012. Why Science Is Not Necessarily Self-Correcting. Perspectives on Psychological Science 7, 6 (2012), 645-654.
  28. Pratyusha Kalluri. 2020. Don't ask if Artificial Intelligence is Good or Fair, ask how it Shifts Power. Nature 583, 7815 (2020), 169-169.
  29. Maurice G Kendall and B Babington Smith. 1939. The problem of m rankings. The annals of mathematical statistics 10, 3 (1939), 275-287.
  30. Andrei P Kirilenko and Svetlana Stepchenkova. 2016. Inter-coder agreement in one-to-many classification: fuzzy kappa. PloS one 11, 3 (2016), e0149787.
  31. Laura Krefting. 1991. Rigor in Qualitative Research: The Assessment of Trust- worthiness. American Journal of Occupational Therapy 45, 3 (03 1991), 214-222. https://doi.org/10.5014/ajot.45.3.214 · DOI · Paper page
  32. Klaus Krippendorff. 2018. Content Analysis: An Introduction to its Methodology. Sage Publications.
  33. Thomas S. Kuhn. 1977. Objectivity, Value Judgment, and Theory Choice. In The Essential Tension: Selected Studies in Scientific Tradition and Change. University of Chicago Press, 320-39.
  34. Zachary C. Lipton and Jacob Steinhardt. 2019. Troubling Trends in Machine Learning Scholarship: Some ML Papers Suffer from Flaws That Could Mislead the Public and Stymie Future Research. Queue 17, 1 (2019), 45-77. https: //doi.org/10.1145/3317287.3328534 · DOI · Paper page
  35. Helen E. Longino. 1996. Cognitive and Non-Cognitive Values in Science: Re- thinking the Dichotomy. In Feminism, Science, and the Philosophy of Science, Lynn Hankinson Nelson and Jack Nelson (Eds.). Springer Netherlands, 39-58.
  36. Ernan McMullin. 1982. Values in science. In Proceedings of the Biennial Meeting of the Philosophy of Science Association.
  37. Robert K. Merton. 1973. The Sociology of Science: Theoretical and Empirical Investigations. University of Chicago press.
  38. Shakir Mohamed, Marie-Therese Png, and William Isaac. 2020. Decolonial AI: De- colonial Rheory as Sociotechnical Foresight in Artificial Intelligence. Philosophy & Technology 33 (2020), 659--684.
  39. Priyanka Nanayakkara, Jessica Hullman, and Nicholas Diakopoulos. 2021. Un- packing the Expressed Consequences of AI Research in Broader Impact State- ments. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3461702.3462608 · DOI
  40. Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. NYU Press.
  41. Frank Pasquale. 2015. The black box society. Harvard University Press.
  42. Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. 2020. Performative Prediction. In Proceedings of ICML.
  43. Vinay Uday Prabhu and Abeba Birhane. 2020. Large Image Datasets: A Pyrrhic Win for Computer Vision? arXiv preprint arXiv:2006.16923 (2020). https://arxiv. org/abs/2006.16923 · arXiv
  44. Anna Rogers. 2019. Peer review in NLP: reject-if-not-SOTA. Hacking Semantics blog (2019). https://hackingsemantics.xyz/2020/reviewing-models/#everything- wrong-with-reject-if-not-sota
  45. Harini Suresh and John V. Guttag. 2019. A Framework for Understanding Un- intended Consequences of Machine Learning. arXiv preprint arXiv:1901.10002 (2019). http://arxiv.org/abs/1901.10002 · arXiv
  46. Denis Trapido. 2015. How Novelty in Knowledge Earns Recognition: The Role of Consistent Identities. Research Policy 44, 8 (2015), 1488-1500.
  47. Kiri Wagstaff. 2012. Machine Learning that Matters. In Proceedings of ICML.
  48. Joseph Weizenbaum. 1972. On the Impact of the Computer on Society. Science 176, 4035 (1972), 609-614.
  49. Langdon Winner. 1977. Autonomous Technology: Technics-out-of-Control as a Theme in Political Thought. MIT Press.
  50. Langdon Winner. 1980. Do Artifacts Have Politics? Daedalus 109, 1 (1980), 121-136.

Cited by

Sorted by citations

Authors
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M.
Affiliations
  • University College Dublin (IE)
  • Stanford University (US)
  • University of Michigan (US)
  • University of Washington (US)
  • University of Pittsburgh (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
173-184
DOI
10.48550/arxiv.2106.15590
arXiv
2106.15590
OpenAlex
W3174220540
Cited by
317
References
0
Citations by year
2026: 72 · 2025: 66 · 2024: 63 · 2023: 70 · 2022: 36 · 2021: 10
FWCI
58.38
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Science Foundation Ireland (13/RC/2094-2)
  • National Defense Science and Engineering Graduate
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Keywords
value neutrality, research ethics, research funding sources, institutional affiliation, societal impact of AI, novelty, centralization of power, NeurIPS

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review