This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology
Open accessAbstract
The explosion in the use of software in important sociotechnical systems has renewed focus on the study of the way technical constructs reflect policies, norms, and human values. This effort requires the engagement of scholars and practitioners from many disciplines. And yet, these disciplines often conceptualize the operative values very differently while referring to them using the same vocabulary. The resulting conflation of ideas confuses discussions about values in technology at disciplinary boundaries. In the service of improving this situation, this paper examines the value of shared vocabularies, analytics, and other tools that facilitate conversations about values in light of these disciplinary specific conceptualizations, the role such tools play in furthering research and practice, outlines different conceptions of "fairness" deployed in discussions about computer systems, and provides an analytic tool for interdisciplinary discussions and collaborations around the concept of fairness. We use a case study of risk assessments in criminal justice applications to both motivate our effort--describing how conflation of different concepts under the banner of "fairness" led to unproductive confusion--and illustrate the value of the fairness analytic by demonstrating how the rigorous analysis it enables can assist in identifying key areas of theoretical, political, and practical misunderstanding or disagreement, and where desired support alignment or collaboration in the absence of consensus.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
'Fair' can mean different things in different contexts to different people.
“But 'fair' can mean diferent things in diferent contexts to diferent people [109].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [109]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color
Crenshaw, K. (1991) · Stanford Law Review · doi:10.2307/1229039 · Cited by 29,585 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Justice as Fairness: A Restatement
Metz, T., Rawls, J., & Kelly, E. I. (2002) · The Philosophical Review · doi:10.2307/1556442 · Cited by 3,487 · see citing works
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2018) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,318 · see citing works
Steps toward an ecology of infrastructure: Design and access for large information spaces
Star, S. L., & Ruhleder, K. (1996) · Information Systems Research · doi:10.1287/isre.7.1.111 · Cited by 3,238 · see citing works
What Is the Point of Equality?
Anderson, E. S. (1999) · Ethics · doi:10.1086/233897 · Cited by 3,112 · see citing works
Scanlon, T. (2000) · Harvard University Press eBooks · doi:10.2307/j.ctv134vmrn · Cited by 3,053 · see citing works
Satterthwaite, M. A. (1975) · Journal of Economic Theory · doi:10.1016/0022-0531(75)90050-2 · Cited by 2,880 · see citing works
The Fragility of Goodness: Luck and Ethics in Greek Tragedy and Philosophy
Gagarin, M., & Nussbaum, M. C. (1987) · The Classical World · doi:10.2307/4350105 · Cited by 2,460 · see citing works
An Introduction to the Principles of Morals and Legislation
Bentham, J. (1789) · Oxford University Press eBooks · doi:10.1093/oseo/instance.00077240 · Cited by 2,201 · see citing works
A Difficulty in the Concept of Social Welfare
Arrow, K. J. (1950) · Journal of Political Economy · doi:10.1086/256963 · Cited by 2,079 · see citing works
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · see citing works
Friedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203 · cited 1× · see citing works
“terrogatingthevaluesoftechnicalandsociotechnicalsystems havedeeprootsfrommultipledisciplines,includingscience&technologystudies, law,philosophy, HCI, and computer science & engineering (e.g., [22, 27, 50, 89, 90, 96, 126, 151]). In addition to the critical analysis of understanding values in technical artifact…”
BackgroundAlgorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149 · see citing works
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · see citing works
Algorithms as culture: Some tactics for the ethnography of algorithmic systems
Seaver, N. (2017) · Big Data & Society · doi:10.1177/2053951717738104 · Cited by 1,010 · see citing works
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · see citing works
Friedman, B. (1996) · Interactions · doi:10.1145/242485.242493 · Cited by 966 · cited 2× · see citing works
“esign and “values in design”, include forms of both analysis to identify and critique values associated with systems, and methods for incorporating values into the processes of engineering and design [48, 49, 78, 137]. Some of these approachestovalueshavebeencritiquedasviewingvaluesasoverlystableanduniversal, without…”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941 · see citing works
On strategy-proofness and single peakedness
Moulin, H. J. (1980) · Public Choice · doi:10.1007/bf00128122 · Cited by 890 · see citing works
The Color of Supremacy: Beyond the discourse of ‘white privilege’
Leonardo, Z. (2004) · Educational Philosophy and Theory · doi:10.1111/j.1469-5812.2004.00057.x · Cited by 830 · see citing works
Calafiore, G. C., & Ghaoui, L. E. (2018) · Cambridge University Press eBooks · doi:10.1017/cbo9781107279667.013 · Cited by 822 · see citing works
Network Neutrality, Broadband DiscriminationPreprint
Wu, T. (2003) · SSRN Electronic Journal · doi:10.2139/ssrn.388863 · Cited by 791 · see citing works
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
McIntosh, P. (2019) · Routledge eBooks · doi:10.4324/9781351133791-3 · Cited by 773 · see citing works
Algorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733 · see citing works
Toward Information Infrastructure Studies: Ways of Knowing in a Networked Environment
Bowker, G. C., Baker, K. S., Millerand, F., & Ribes, D. (2009) · Springer eBooks · doi:10.1007/978-1-4020-9789-8_5 · Cited by 693 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 2× · see citing works
“Research by social scientists and humanities scholars has similarly advanced the state of theory and application [5, 11, 80, 136, 149].”
BackgroundPrediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2018) · arXiv · doi:10.48550/arxiv.1811.07867 · Cited by 687 · see citing works
How to analyze political attention with minimal assumptions and costs
Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., & Radev, D. R. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00427.x · Cited by 671 · see citing works
International Handbook of Internet Research
Hunsinger, J., Allen, M., & Klastrup, L. (2010) · Springer eBooks · doi:10.1007/978-1-4020-9789-8 · Cited by 559 · see citing works
Winner, L. (2020) · Routledge eBooks · doi:10.4324/9781003074960-3 · Cited by 556 · see citing works
The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition
Keyes, O. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274357 · Cited by 505 · see citing works
Lex Informatica: The Formulation of Information Policy Rules through Technology
Reidenberg, J. R. (1997) · FLASH - Fordham Law Archive of Scholarship & History (Fordham University) · Cited by 429 · see citing works
Star, S. L., & Ruhleder, K. (1994) · Proceedings of the 1994 ACM conference on Computer supported cooperative work - CSCW '94 · doi:10.1145/192844.193021 · Cited by 412 · see citing works
Shaping Technology/Building Society: Studies in Sociotechnical Change
Gieryn, T. F., Bijker, W. E., & Law, J. (1994) · Technology and Culture · doi:10.2307/3106331 · Cited by 390 · see citing works
Goodhart, C. (1984) · doi:10.1007/978-1-349-17295-5 · Cited by 350 · see citing works
Does Unconscious Racial Bias Affect Trial Judges
Rachlinski, J. J., Johnson, S. L., Wistrich, A. J., & Guthrie, C. (2009) · Scholarship @ Cornell Law (Cornell University) · Cited by 311 · see citing works
A survey on provenance: What for? What form? What from?
Herschel, M., Diestelkämper, R., & Ben Lahmar, H. (2017) · The VLDB Journal · doi:10.1007/s00778-017-0486-1 · Cited by 300 · see citing works
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · see citing works
Virtue Ethics: A Misleading Category?
Nussbaum, M. C. (1999) · The Journal of Ethics · doi:10.1023/a:1009877217694 · Cited by 248 · see citing works
Flores, A. W., Bechtel, K., & Lowenkamp, C. T. (2016) · Federal probation · Cited by 244 · see citing works
Discrimination Comes in Many Forms
Pincus, F. L. (1996) · American Behavioral Scientist · doi:10.1177/0002764296040002009 · Cited by 239 · see citing works
Hardt, M., Megiddo, N., Papadimitriou, C. H., & Wootters, M. K. (2016) · Conference on Innovations in Theoretical Computer Science · doi:10.1145/2840728.2840730 · Cited by 238 · see citing works
Houston, L., Jackson, S. J., Rosner, D. K., Ahmed, S. I., Young, M., & Kang, L. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858470 · Cited by 235 · see citing works
How computer systems embody values
Nissenbaum, H. (2001) · Computer · doi:10.1109/2.910905 · Cited by 232 · see citing works
Passi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225 · see citing works
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220 · see citing works
Rosner, D. K., & Ames, M. G. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531692 · Cited by 212 · see citing works
Jeffrey, P. J. (2003) · Social Studies of Science · doi:10.1177/0306312703334003 · Cited by 212 · see citing works
Assessing Risk Assessment in Action
Stevenson, M. T. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3016088 · Cited by 195 · see citing works
Values and Ethics in Human-Computer Interaction
Shilton, K. (2018) · Found. Trends Hum. Comput. Interact. · doi:10.1561/1100000073 · Cited by 187 · see citing works
Lee, M. K., & Baykal, S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998230 · Cited by 182 · see citing works
Values as Hypotheses: Design, Inquiry, and the Service of Values
JafariNaimi, N., Nathan, L. P., & Hargraves, I. G. (2015) · Design Issues · doi:10.1162/desi_a_00354 · Cited by 160 · see citing works
Knobel, C. P., & Bowker, G. C. (2011) · Communications of the ACM · doi:10.1145/1965724.1965735 · Cited by 158 · see citing works
Flanagan, M., Howe, D. C., & Nissenbaum, H. F. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055076 · Cited by 149 · see citing works
Privacy is an essentially contested concept: a multi-dimensional analytic for mapping privacy
Mulligan, D. K., Koopman, C., & Doty, N. (2016) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2016.0118 · Cited by 149 · cited 5× · see citing works
“Together the definitional overview and analytic also help clarify different contexts of usage, so that these ongoing moments of debate can lead to “meaningful contestation” [107] rather than talking past one another.”
BackgroundMethodInfluentialSelf-citationThe policy knot: re-integrating policy, practice and design in cscw studies of social computing
Jackson, S. J., Gillespie, T., & Payette, S. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531674 · Cited by 143 · see citing works
Does mitigating ML's impact disparity require treatment disparity?Preprint
Lipton, Z. C., Chouldechova, A., & McAuley, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07076 · Cited by 125 · see citing works
Procaccia, A. D. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781107446984.014 · Cited by 117 · see citing works
Passi, S., & Jackson, S. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998331 · Cited by 113 · see citing works
Encyclopedia of Science, Technology, and Ethics
Daly, E. M. (2006) · Science & Technology Studies · doi:10.23987/sts.55205 · Cited by 94 · see citing works
Privacy Considerations for Internet Protocols
Cooper, A. F., Tschofenig, H., Aboba, B., et al. (2013) · doi:10.17487/rfc6973 · Cited by 87 · see citing works
A not quite random walk: Experimenting with the ethnomethods of the algorithm
Ziewitz, M. (2017) · Big Data & Society · doi:10.1177/2053951717738105 · Cited by 85 · see citing works
Machine Learning Risk Assessments in Criminal Justice Settings
Berk, R. (2018) · Cambridge International Law Journal · doi:10.1007/978-3-030-02272-3 · Cited by 84 · see citing works
Proceedings of the Aristotelian Society
(1922) · Nature · doi:10.1038/109371a0 · Cited by 74 · see citing works
How to see values in social computing
Shilton, K., Koepfler, J. A., & Fleischmann, K. R. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531625 · Cited by 68 · see citing works
From critical design to critical infrastructure
Irani, L. C., & Silberman, M. ‘. (2014) · interactions · doi:10.1145/2627392 · Cited by 60 · see citing works
Collaboration in Metagenomics: Sequence Databases and the Organization of Scientific Work
Bietz, M. J., & Lee, C. P. (2009) · Springer eBooks · doi:10.1007/978-1-84882-854-4_15 · Cited by 55 · see citing works
Between Chaos and Routine: Boundary Negotiating Artifacts in Collaboration
Lee, C. P. (2006) · Springer eBooks · doi:10.1007/1-4020-4023-7_20 · Cited by 55 · see citing works
Clark, D. D., Wroclawski, J. T., Sollins, K. R., et al. (2002) · ACM SIGCOMM Computer Communication Review · doi:10.1145/964725.633059 · Cited by 51 · see citing works
The Accuracy, Equity, and Jurisprudence of Criminal Risk Assessment
Goel, S., Shroff, R., Skeem, J. L., & Slobogin, C. (2018) · Edward Elgar Publishing eBooks · doi:10.4337/9781788972826.00007 · Cited by 50 · see citing works
Steinhardt, S. B., & Jackson, S. J. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531736 · Cited by 41 · see citing works
Mulligan, D. K. (2018) · eScholarship (California Digital Library) · doi:10.15779/z38qn5zb5h · Cited by 39 · see citing works
Self-citationEaglin, J. M. (2017) · University of North Carolina School of Law Scholarship Repository (University of North Carolina Hospitals) · Cited by 37 · see citing works
Breaking Down While Building Up
Steinhardt, S. B. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858420 · Cited by 32 · see citing works
The Changing Contours of "Participation" in Data-driven, Algorithmic Ecosystems
Wolf, C. T., Zhu, H., Bullard, J., Lee, M. K., & Brubaker, J. R. (2018) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3272973.3273005 · Cited by 32 · see citing works
ASSESSMENT IN THE SCHOOLS: PURPOSES AND CONSEQUENCES
Messick, S. (1987) · ETS Research Report Series · doi:10.1002/j.2330-8516.1987.tb00255.x · Cited by 29 · see citing works
International Practices of Criminal Justice
(2017) · Routledge eBooks · doi:10.4324/9781315144788 · Cited by 29 · see citing works
How Computer Systems Embody Values
NissenbaumHelen (2001) · Computer · doi:10.5555/619061.621683 · Cited by 20 · see citing works
Can I be a luck egalitarian and a Rawlsian
Tomlin, P. (2012) · Ethical Perspectives · doi:10.2143/ep.19.3.2172296 · Cited by 15 · see citing works
Vicente, L. (2010) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 13 · see citing works
Algorithms at Work: Empirical Diversity, Analytic Vocabularies, Design Implications
Wagenknecht, S., Lee, M., Lustig, C., O’Neill, J., & Zade, H. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818052.2855519 · Cited by 12 · see citing works
Political Equality: An Essay in Democratic Theory.
Lichtenberg, J., & Beitz, C. R. (1992) · The Philosophical Review · doi:10.2307/2186084 · Cited by 11 · see citing works
Fairness as a Program PropertyPreprint
Albarghouthi, A., D’Antoni, L., Drews, S., & Nori, A. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.06067 · Cited by 5 · see citing works
The Social Cost of Strategic ClassificationPreprint
Milli, S., Miller, J. P., Dragan, A. D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08460 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 46
- Chelsea Barabas, Karthik Dinakar, Joichi Ito Virza, Jonathan Zittrain, et al. 2018. Interventions over predictions: Reframing the ethical debate for actuarial risk assessment. Conference on Fairness, Accountability, and Transparency (2018).
- Geoffrey C Bowker, C Geoffrey, W Bernard Carlson, et al. 1994. Science on the run: Information management and industrial geophysics at Schlumberger, 1920-1940. MIT press.
- Geoffrey C Bowker and Susan Leigh Star. 2000. Sorting things out: Classification and its consequences. MIT press.
- John Broome. 1990. Fairness. In Proceedings of the Aristotelian Society, Vol. 91. JSTOR, 87-101.
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency. 77-91.
- Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153-163.
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134-148.
- Angèle Christin. 2017. Algorithms in practice: Comparing web journalism and criminal justice. Big Data & Society 4, 2 (2017).
- Computing Community Consortium. 2015. Privacy by Design-Engineering Privacy: Workshop 3 Report. https: //cra.org/ccc/wp-content/uploads/sites/2/2015/12/PbD3-Workshop-Report-v2.pdf.
- Computing Community Consortium. 2015. Privacy by Design-Privacy Enabling Design: Workshop 2 Report. https://cra.org/ccc/wp-content/uploads/sites/2/2015/05/PbD2-Report-v5.pdf.
- Computing Community Consortium. 2015. Privacy by Design-State of Research and Practice: Workshop 1 Report. https://cra.org/ccc/wp-content/uploads/sites/2/2015/02/PbD-Workshop-1-Report-.pdf.
- Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023 (2018). · arXiv
- Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580.
- fat Since 2018. The ACM Conference on Fairness, Accountability, and Transparency (FAT * ). https://fatconference.org.
- fatml Since 2014. The Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML). https://fatml.org.
- Batya Friedman, Peter H. Kahn, and Alan Borning. 2008. Value Sensitive Design and Information Systems. In The Handbook of Information and Computer Ethics, Kenneth Einar Himma and Herman T. Tavani (Eds.). John Wiley & Sons, Inc., Chapter 4, 69-101.
- Allan Gibbard. 1973. Manipulation of voting schemes: a general result. Econometrica: journal of the Econometric Society (1973), 587-601.
- Moritz Hardt and Solon Barocas. 2017. Fairness in Machine Learning. Neural Information Processing Symposium, Tutorials Track.
- Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. Neural Information Processing Symposium (2016).
- Herbert Lionel Adolphus Hart. 1961. The concept of law. Oxford University Press.
- Deborah Hellman. 2008. When is discrimination wrong? Harvard University Press.
- Anna Lauren Hoffmann. 2017. Data, technology, and gender: Thinking about (and from) trans lives. In Spaces for the Future. Routledge, 3-13. https://doi.org/10.4324/9780203735657-1 · DOI
- Lily Hu, Nicole Immorlica, and Jennifer Wortman Vaughan. 2018. The Disparate Effects of Strategic Manipulation. arXiv preprint arXiv:1808.08646 (2018). · arXiv
- Daniel Kluttz, Jenna Burrell, Deirdre Mulligan, Joshua A. Kroll, Amit Elazari, and Andrew Smart. 2018. Report from the First AFOG Summer Workshop. https://afog.berkeley.edu/2018/08/13/report-from-the-first-afog-summer-workshop/.
- Donald Ervin Knuth. 1968. The art of computer programming: fundamental algorithms. Vol. 1. Pearson Education.
- Issa Kohler-Hausmann. 2018. Eddie Murphy and the Dangers of Counterfactual Causal Thinking About Detecting Racial Discrimination. Available at SSRN, https:// ssrn.com/ abstract=3050650 (2018).
- Nitin Kohli and Paul Laskowski. 2018. Epsilon Voting: Mechanism Design for Parameter Selection in Differential Privacy. In 2018 IEEE Symposium on Privacy-Aware Computing (PAC). IEEE, 19-30.
- Christine M Korsgaard. 1996. Creating the kingdom of ends. Cambridge University Press.
- Christine M Korsgaard, Christine Marion Korsgaard, Gerald Allan Cohen, Raymond Geuss, Thomas Nagel, and Bernard Williams. 1996. The sources of normativity. Cambridge University Press.
- Joshua A. Kroll. 2015. Accountable Algorithms. Ph.D. Dissertation. Princeton University.
- Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsylvania Law Review 165, 3 (2017).
- Joshua A. Kroll, Nitin Kohli, and Deirdre Mulligan. 2018. Common Pitfalls for Studying the Human Side of Machine Learning. Tutorial at the Neural Information Processing Symposium (NeurIPS).
- Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems. 4066-4076.
- Bruno Latour. 1989. The moral dilemmas of a safety belt. Alliage 1 (1989), 21-27. Originally published as "La ceinture de sécurité". Unpublished English translation by Lydia Davis.
- Law Review Editors. 2017. State v. Loomis. 130 Harvard Law Review 1530, https://harvardlawreview.org/2017/03/ state-v-loomis/.
- Christopher A. Le Dantec, Erika Shehan Poole, and Susan P. Wyche. 2009. Values as lived experience: Evolving value sensitive design in support of value discovery. In Proceedings of the 27th international conference on Human factors in computing systems -CHI 09. ACM Press, New York, New York, USA, 1141. https://doi.org/10.1145/1518701.1518875 · DOI
- Kenneth O May. 1952. A set of independent necessary and sufficient conditions for simple majority decision. Econometrica: Journal of the Econometric Society (1952), 680-684.
- Samuel Messick. 1987. Validity. ETS Research Report Series 1987, 2 (1987), i-208.
- Arvind Narayanan. 2018. 21 Definitions of Fairness and Their Politics. Tutorial Presented at the First Conference on Fairness, Accountability, and Transparency (FAT * ).
- Beatriz Pérez, Julio Rubio, and Carlos Sáenz-Adán. 2018. A systematic review of provenance systems. Knowledge and Information Systems (2018), 1-49.
- Zachary Piso. 2016. Integration, Values, and Well-Ordered Interdisciplinary Science. the pluralist 11, 1 (2016), 49-57.
- John Rawls. 2009. A theory of justice. Harvard university press.
- Christian Sandvig. 2015. Seeing the sort: The aesthetic and industrial defense of "the algorithm". Journal of the New Media Caucus (2015).
- Alicia Solow-Niederman, YooJung Choi, and Guy Van den Broeck. 2019. The Institutional Life of Algorithmic Risk Assessment. Berkeley Technology Law Journal (2019).
- Manuel Velasquez, Claire Andre, T Shanks, and Michael J Meyer. 2014. Justice and fairness. Issues in Ethics 3, 2 (2014).
- Sandra Wachter and Brent Mittelstadt. 2019. A right to reasonable inferences: re-thinking data protection law in the age of big data and AI. Columbia Business Law Review (2019).
Cited by
Sorted by citations
Timmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221134490 · Cited by 263 · see citing works
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229 · see citing works
Good proctor or “big brother”? Ethics of online exam supervision technologies
Coghlan, S., Miller, T., & Paterson, J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00476-1 · Cited by 219 · see citing works
Re-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · see citing works
Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445888 · Cited by 111 · see citing works
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99 · see citing works
Schoeffer, J., Kuehl, N., & Machowski, Y. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533218 · Cited by 67 · see citing works
Lewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30 · see citing works
A Human-Centric Perspective on Fairness and Transparency in Algorithmic Decision-Making
Schoeffer, J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503811 · Cited by 9 · see citing works
Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems
Kroll, J. A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445937 · Cited by 2 · see citing works
Self-citation
- Authors
- Mulligan, D. K., Kroll, J. A., Kohli, N., & Wong, R. Y.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2019-11-07
- Volume
- 3
- Issue
- CSCW
- Pages
- 1-36
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4288086173
- Cited by
- 111
- References
- 86
- Citations by year
- 2026: 23 · 2025: 17 · 2024: 17 · 2023: 20 · 2022: 20 · 2021: 13 · 2020: 1
- FWCI
- 14.86
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Funders
- NSF (DGE 1752814)
- National Science Foundation (INSPIRE SES1537324)
- Berkeley Center for Law and Technology
- Microsoft (Gift)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- fairness, technology and values, shared vocabulary, interdisciplinary collaboration, risk assessment, socio-technical systems, human values
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
