CoChat
Sign Up
PreprintCited by 82Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Safety Systems Engineering in Autonomy

The DOI matches its DataCite record.

A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle

Open access
Suresh, Harini, Guttag, John V.·2019·arXiv
DOI 10.48550/arxiv.1901.10002arXiv 1901.10002
Open this paper in CoChat — free accountOpen PDF

Abstract

As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this paper, we provide a framework that identifies seven distinct potential sources of downstream harm in machine learning, spanning data collection, development, and deployment. In doing so, we aim to facilitate more productive and precise communication around these issues, as well as more direct, application-grounded ways to mitigate them.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1901.10002

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 42

  1. James Atwood, Yoni Halpern, Pallavi Baljekar, Eric Breck, D Sculley, Pavel Ostyakov, Sergey I Nikolenko, Igor Ivanov, Roman Solovyev, Weimin Wang, et al. 2020. The Inclusive Images Competition. In The NeurIPS'18 Competition. Springer, 155-186.
  2. Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. 2019. Differential privacy has disparate impact on model accuracy. Advances in Neural Information Processing Systems 32 (2019), 15479-15488.
  3. Solon Barocas and Andrew D Selbst. 2016. Big data's disparate impact. Cal. L. Rev. 104 (2016), 671.
  4. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accu- racy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency. 77-91.
  5. Karen L Calderone. 1990. The influence of gender on the frequency of pain and sedative medication administered to postoperative patients. Sex Roles 23, 11-12 (1990), 713-725.
  6. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321-357.
  7. Irene Chen, Fredrik D Johansson, and David Sontag. 2018. Why Is My Classifier Discriminatory?. In Advances in Neural Information Processing Systems. 3539- 3550.
  8. Erin Collins. 2018. Punishing Risk. Geo. LJ 107 (2018), 57.
  9. Sam Corbett-Davies, Sharad Goel, Jamie Morgenstern, and Rachel Cummings. 2018. Defining and Designing Fair Algorithms. In Proceedings of the 2018 ACM Conference on Economics and Computation (Ithaca, NY, USA) (EC '18). Association for Computing Machinery, New York, NY, USA, 705. https://doi.org/10.1145/ 3219166.3277556 · DOI
  10. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Chris- tian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representa- tion bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency. 120-128.
  11. Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, Hilary Nicole, and Morgan Klaus Scheuerman. 2020. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. arXiv:2007.07399 [cs.CY] · arXiv
  12. Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580.
  13. Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018. Decoupled Classifiers for Group-Fair and Efficient Machine Learning. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 119-133. http://proceedings.mlr.press/ v81/dwork18a.html
  14. Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway Feedback Loops in Predictive Policing. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 160-171. http://proceedings.mlr.press/ v81/ensign18a.html
  15. Jessie Finocchiaro, Roland Maio, Faidra Monachou, Gourab K Patro, Manish Raghavan, Ana-Andreea Stoica, and Stratis Tsirtsis. 2021. Bridging Machine Learning and Mechanism Design towards Algorithmic Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 489-503.
  16. William R. Frey, Desmond U. Patton, Michael B. Gaskell, and Kyle A. McGre- gor. 2020. Artificial Intelligence and Inclusion: Formerly Gang-Involved Youth as Domain Experts for Analyzing Unstructured Twitter Data. Social Science Computer Review 38, 1 (2020), 42-56. https://doi.org/10.1177/0894439318788314 arXiv:https://doi.org/10.1177/0894439318788314 · DOI
  17. Friedler, Scheidegger, Venkatasubramanian, Choudhary, Hamilton, and Roth. 2019. A comparative study of fairness-enhancing interventions in machine learning. In ACM Conference on Fairness, Accountability and Transparency (FAT*). ACM. http://arxiv.org/abs/1802.04422 · arXiv
  18. Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems. ACM Transactions on Information Systems (TOIS) 14, 3 (1996), 330-347.
  19. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 115, 16 (2018), E3635-E3644.
  20. Kelly M Hoffman, Sophie Trawalter, Jordan R Axt, and M Norman Oliver. 2016. Racial bias in pain assessment and treatment recommendations, and false be- liefs about biological differences between blacks and whites. Proceedings of the National Academy of Sciences 113, 16 (2016), 4296-4301.
  21. Diane E Hoffmann and Anita J Tarzian. 2001. The girl who cried pain: a bias against women in the treatment of pain. The Journal of Law, Medicine & Ethics 28 (2001), 13-27.
  22. Sara Hooker. 2021. Moving beyond "algorithmic bias is a data problem". Patterns 2, 4 (2021), 100241.
  23. Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. Characterising Bias in Compressed Models. https://arxiv.org/abs/2010. 03058
  24. Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 624-635.
  25. Kimmo Karkkainen and Jungseock Joo. 2021. FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1548-1558.
  26. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. 2018. Algorithmic fairness. In AEA Papers and Proceedings, Vol. 108. 22-27.
  27. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent Trade- Offs in the Fair Determination of Risk Scores. In 8th Innovations in Theoretical Computer Science Conference (ITCS 2017) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 67), Christos H. Papadimitriou (Ed.). Schloss Dagstuhl- Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 43:1-43:23. https://doi. org/10.4230/LIPIcs.ITCS.2017.43 · DOI · Paper page
  28. Min Li, Amy Mickel, and Stanley Taylor. 2018. "Should This Loan be Approved or Denied?": A Large Dataset with Class Assignment Guidelines. Journal of Statistics Education 26, 1 (2018), 55-66. https://doi.org/10.1080/10691898.2018.1434342 arXiv:https://doi.org/10.1080/10691898.2018.1434342 · DOI
  29. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Com- puting Surveys (CSUR) 54, 6 (2021), 1-35.
  30. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* '19). Association for Computing Machinery, New York, NY, USA, 220-229. https://doi.org/10.1145/ 3287560.3287596 · DOI · Paper page
  31. Jana M Mossey. 2011. Defining racial and ethnic disparities in pain management. Clinical Orthopaedics and Related Research® 469, 7 (2011), 1859-1870.
  32. Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria-Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. 2020. Bias in data-driven artificial intelligence systems-An introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10, 3 (2020), e1356.
  33. Desmond U. Patton, William R. Frey, Kyle A. McGregor, Fei-Tzin Lee, Kathleen McKeown, and Emanuel Moss. 2020. Contextual Analysis of Social Media: The Promise and Challenge of Eliciting Context in Social Media Posts with Natural Language Processing. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA) (AIES '20). Association for Computing Machin- ery, New York, NY, USA, 337-342. https://doi.org/10.1145/3375627.3375841 · DOI
  34. Sean M Phelan, Diane J Burgess, Mark W Yeazel, Wendy L Hellerstedt, Joan M Griffin, and Michelle van Ryn. 2015. Impact of weight bias and stigma on quality of care and outcomes for patients with obesity. obesity reviews 16, 4 (2015), 319-326.
  35. Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. 2020. Mitigating bias in algorithmic hiring: Evaluating claims and practices. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 469-481.
  36. Hee Jung Ryu, Hartwig Adam, and Margaret Mitchell. 2018. Inclusivefacenet: Improving face attribute detection with race and gender diversity. In Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML).
  37. Steven L Salzberg. 1997. On comparing classifiers: Pitfalls to avoid and a recom- mended approach. Data mining and knowledge discovery 1, 3 (1997), 317-328.
  38. Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 59-68.
  39. Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. 2017. No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World. In NIPS 2017 workshop: Machine Learning for the Developing World.
  40. Megan Stevenson. 2018. Assessing risk assessment in action. Minn. L. Rev. 103 (2018), 303.
  41. Harini Suresh, Jen J. Gong, and John V. Guttag. 2018. Learning Tasks for Multitask Learning: Heterogenous Patient Populations in the ICU. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM.
  42. Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning. 325-333.

Cited by

Sorted by citations

Metadata

Authors
Suresh, H., & Guttag, J. V.
Affiliations
Massachusetts Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
1-9
DOI
10.48550/arxiv.1901.10002
arXiv
1901.10002
OpenAlex
W3172100757
Cited by
82
References
0
Citations by year
2026: 1 · 2025: 8 · 2024: 12 · 2023: 15 · 2022: 8 · 2021: 23 · 2020: 13 · 2019: 2
FWCI
13.13
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
machine learning lifecycle, harm mitigation, data collection, model deployment, societal impact of AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review