Bridging Machine Learning and Mechanism Design towards Algorithmic Fairness
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Decision-making systems increasingly orchestrate our world: how to intervene on the algorithmic components to build fair and equitable systems is therefore a question of utmost importance; one that is substantially complicated by the context-dependent nature of fairness and discrimination. Modern decision-making systems that involve allocating resources or information to people (e.g., school choice, advertising) incorporate machine-learned predictions in their pipelines, raising concerns about potential strategic behavior or constrained allocation, concerns usually tackled in the context of mechanism design. Although both machine learning and mechanism design have developed frameworks for addressing issues of fairness and equity, in some complex decision-making systems, neither framework is individually sufficient. In this paper, we develop the position that building fair decision-making systems requires overcoming these limitations which, we argue, are inherent to each field. Our ultimate objective is to build an encompassing framework that cohesively bridges the individual frameworks of mechanism design and machine learning. We begin to lay the ground work towards this goal by comparing the perspective each discipline takes on fair decision-making, teasing out the lessons each field has taught and can teach the other, and highlighting application domains that require a strong collaboration between these disciplines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“For example, prior work studied how machine learning formulations may underpredict black patients’ health care needs [158].”
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Machine learning has been increasingly used to supplement human decisions, drawing attention to biases rooted in learning from historically prejudiced data [13, 21, 44].”
BackgroundThe EU General Data Protection Regulation (GDPR)
Voigt, P., & von dem Bussche, A. (2017) · doi:10.1007/978-3-319-57959-7 · Cited by 3,001
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 3×
“Numerous definitions have been proposed for group fairness [150, 202], suggesting to impose either simple statistical parity conditions between groups [61] or more complex classification constraints; some aim to equalize each group’s opportunity to positive outcomes [105], balance the misclassification”
BackgroundFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 4×
“om machine learning can be useful. Because machine learning treats uncertaintyasafirstclassconcept,manyconceptionsoffairnessfrom themachinelearning literatureexplicitlyconsider errorsand their impacts [55, 100]. Uncertaintycanalsomanifestwithrespecttooutcomes,notjust preferences. Many application domains consider inher…”
BackgroundInfluentialAwad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018) · Nature · doi:10.1038/s41586-018-0637-6 · Cited by 1,859
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Socio-technical systems: From design methods to systems engineering
Baxter, G., & Sommerville, I. (2011) · Interacting with Computers · doi:10.1016/j.intcom.2010.07.003 · Cited by 1,642
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 1×
“assumption: it is assumed that the issue at hand is reducible to a standard domain problem, and that if the solution to this problem is fair and equitable, then so too will be the overall sociotechnical system [183].”
BackgroundLee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 2×
“5, some first approaches have been made [96, 97, 142, 177, 187, 207] towards studying human perceptions of fairness but questions regarding who are the relevant stakeholders in criminal justice, what notions are more appropriate in that field and how to aggregate preferences still need to be answered.”
BackgroundThe bias blind spot: perceptions of bias in self versus others
Pronin, E., Lin, D. Y., & Ross, L. (2002) · Personality and Social Psychology Bulletin · doi:10.1177/0146167202286008 · Cited by 1,289
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · cited 1×
“Numerous definitions have been proposed for group fairness [150, 202], suggesting to impose either simple statistical parity conditions between groups [61] or more complex classification constraints; some aim to equalize each group’s opportunity to positive outcomes [105], balance the misclassification”
BackgroundRacial Discrimination in the Sharing Economy: Evidence from a Field Experiment
Edelman, B. G., Luca, M., & Svirsky, D. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2701902 · Cited by 999
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · cited 1×
“conceptually and practically: from early computer science works that defines fairness through observations [70, 100] or representations [79, 201] to understanding causal relationships between features [122, 128]. Whilesatisfyingmultipledefinitionsmaynotalwaysbepossible[125], the different definitions of fairness in machin…”
BackgroundFairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943
Self-citationInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Self-citationMeta-analysis of field experiments shows no change in racial discrimination in hiring over time
Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1706255114 · Cited by 887
Lambrecht, A., & Tucker, C. (2019) · Management Science · doi:10.1287/mnsc.2018.3093 · Cited by 854
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Self-citationDiscrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“h as gender [130], age [13], or race [11], can endupreceiving verydifferent typesofads.Forexample,searches done with black-sounding names are highly likely to be shown ads suggestive of arrest records [183]. Another study showed that women were shown relatively fewer advertisements for high paying jobs than men with si…”
Background'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
Fairness in Machine Learning: Lessons from Political Philosophy
Binns, R. (2017) · arXiv · doi:10.48550/arxiv.1712.03586 · Cited by 676
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · cited 2×
“rates among groups [34, 209] or provide similar classifications under counterfactual group memberships [138].”
BackgroundRacial Bias in Motor Vehicle Searches: Theory and Evidence
Knowles, J., Persico, N., & Todd, P. (2001) · Journal of Political Economy · doi:10.1086/318603 · Cited by 608
Sweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · cited 1×
“[138] emphasize that if one is to strive for quantitative fairness, the notion one optimizes for should be context-dependent and developed in partnership with stakeholders.”
BackgroundExplanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388 · cited 1×
“[37, 69, 169], transparency [169, 204], and context-specific feature selection [98, 199] in improving societal fairness perceptions, which mechanism design has traditionally considered as out of scope or assumed to be known, leading to a recent surge in explainability and transparency studies in machine learning.”
BackgroundAvoiding Discrimination through Causal Reasoning
Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., & Schölkopf, B. (2017) · arXiv · doi:10.48550/arxiv.1706.02744 · Cited by 358
Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr
Hannák, A., Wagner, C., Garcia, D., Mislove, A., Strohmaier, M., & Wilson, C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998327 · Cited by 286 · cited 1×
“[97] document the existence of linguistic and other biases in employers’ reviews for gig workers on two online labor platforms, TaskRabbit”
BackgroundExplaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · cited 2×
“5, some first approaches have been made [96, 97, 142, 177, 187, 207] towards studying human perceptions of fairness but questions regarding who are the relevant stakeholders in criminal justice, what notions are more appropriate in that field and how to aggregate preferences still need to be answered.”
BackgroundDiscrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268 · cited 3×
“selection of the source audience in the Lookalike audience tool by Facebook. Following a lawsuit [177], Facebook removed targeting options for housing, credit, and employment ads [67]. Other studies [9,171] again reveal thatad delivery mechanisms could still result in skewed audience distribution based on sensitive at…”
BackgroundInfluentialA Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 256
Stackelberg games for adversarial prediction problems
Brückner, M., & Scheffer, T. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020495 · Cited by 229 · cited 1×
“cation [42, 43, 55, 65, 70, 104, 112] which focuses on incentive-aware machine learning algorithms that try to reduce misclassification caused by transparency-induced strategic behavior.”
BackgroundWang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207
Fairness in Learning: Classic and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., & Roth, A. (2016) · arXiv · doi:10.48550/arxiv.1605.07139 · Cited by 199 · cited 1×
“Resulting from the historical imbalance of loan acceptance [144, 172], Kallus and Zhou [124] observe that algorithms might still yield “bias in, bias out” phenomena, even with fairness constraints, and online machine learning approaches aim to face this issue by incentivizing exploration [47, 120, 121, 131, 170, 182].”
BackgroundMachine learning for combinatorial optimization: A methodological tour d'horizon
Bengio, Y., Lodi, A., & Prouvost, A. (2021) · European Journal of Operational Research · doi:10.1016/j.ejor.2020.07.063 · Cited by 195
Assessing Risk Assessment in Action
Stevenson, M. (2018) · Minnesota Law Review · doi:10.24926/265535.958 · Cited by 193
Srivastava, M., Heidari, H., & Krause, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330664 · Cited by 179 · cited 3×
“Second, what options and information should be made available to the participants? Some studies [18, 106, 157, 181] directly asked participants to choose the model with the best fairness notion or the best outcomes, whereas others [97, 187, 208] asked indirect questions to infer the acceptable fairness principles (e.”
BackgroundSajjadiani, S., Sojourner, A. J., Kammeyer-Mueller, J. D., & Mykerezi, E. (2019) · Journal of Applied Psychology · doi:10.1037/apl0000405 · Cited by 173 · cited 2×
“Emerging data-driven techniques for employee selection have begun to employ techniques from machine learning to evaluate and sort candidates [40, 163, 170].”
BackgroundMethodRunaway Feedback Loops in Predictive Policing
Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., & Venkatasubramanian, S. (2017) · arXiv · doi:10.48550/arxiv.1706.09847 · Cited by 161 · cited 1×
“ely this lack ofinformation that machine learning techniques are designed to handle. Many of the challenges that arise during learning, including data scarcity for certain groups [44], feedback loops [75], preference elicitation [39, 85, 89, 205], and explore-exploit trade-offs [38, 110, 162], implicate serious fairnes…”
BackgroundDwork, C., & Ilvento, C. (2019) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2019.33 · Cited by 91 · cited 1×
“On the auctioneer’s side, the allocation mechanism can be redesigned to ensure fair audience distribution [48, 53, 73, 113].”
BackgroundHomophily and the Glass Ceiling Effect in Social Networks
Avin, C., Keller, B., Lotker, Z., Mathieu, C., Peleg, D., & Pignolet, Y.-A. (2015) · Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science · doi:10.1145/2688073.2688097 · Cited by 90 · cited 1×
“Often, such inequality arises from the individual preference for establishing new connections as well as from pre-defined communities [16].”
BackgroundResidual unfairness in fair machine learning from prejudiced data
Kallus, N., & Zhou, A. (2018) · arXiv · doi:10.48550/arxiv.1806.02887 · Cited by 83
Grgić-Hlača, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · arXiv · doi:10.48550/arxiv.1802.09548 · Cited by 75
(2017) · doi:10.4230/lipics.forc.2020.6 · Cited by 27
Facebook's Advertising Platform: New Attack Vectors and the Need for Interventions
Faizullabhoy, I., & Korolova, A. (2018) · arXiv · doi:10.48550/arxiv.1803.10099 · Cited by 21 · cited 1×
“One of the first reasons behind such unjustifiably skewed ad delivery can be the explicit targeting of users based on sensitive attributes [11, 77] which can be stopped by examining and disallowing ad targeting based on sensitive attributes especially for housing, credit, and employment ads.”
Background
Cited by
Inherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256
Proactive and reactive engagement of artificial intelligence methods for education: a review
Mallik, S., & Gangopadhyay, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1151391 · Cited by 139
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2019) · arXiv · doi:10.48550/arxiv.1901.10002 · Cited by 82
Metadata
- Authors
- Finocchiaro, J., Maio, R., Monachou, F., Patro, G. K., Raghavan, M., Stoica, A.-A., & Tsirtsis, S.
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 489-503
- OpenAlex
- W3092998473
- Cited by
- 78
- References
- 56
- FWCI
- 16.23
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.