Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data
Open accessPreprint — published as 10.1177/2053951717743530 · Paper page
The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
Abstract
Cite as: Veale, Michael and Binns, Reuben (2017) Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society 4(2). doi:10.1177/2053951717743530Decisions based on algorithmic, machine learning models can be unfair, reproducing biases in historical data used to train them. While computational techniques are emerging to address aspects of these concerns through communities such as discrimination-aware data mining (DADM) and fair, accountable and transparent machine learning (FATML), their practical implementation faces real-world challenges. For legal, institutional or commercial reasons, organisations might not hold the data on sensitive attributes such as gender, ethnicity, sexuality or disability needed to diagnose and mitigate emergent indirect discrimination-by-proxy, such as redlining. Such organisations might also lack the knowledge and capacity to identify and manage fairness issues that are emergent properties of complex sociotechnical systems.This paper presents and discusses three potential approaches to deal with such knowledge and information deficits in the context of fairer machine learning. Trusted third parties could selectively store data necessary for performing discrimination discovery and incorporating fairness constraints into model-building in a privacy-preserving manner. Collaborative online platforms would allow diverse organisations to record, share and access contextual and experiential knowledge to promote fairness in machine learning systems. Finally, unsupervised learning and pedagogically interpretable algorithms might allow fairness hypotheses to be built for further selective testing and exploration.Real-world fairness challenges in machine learning are not abstract, constrained optimisation problems, but are institutionally and contextually grounded. Computational fairness tools are useful, but must be researched and developed in and with the messy contexts that will shape their deployment, rather than just for imagined situations. Not doing so risks real, near-term algorithmic harm.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
DiMaggio, P. J., & Powell, W. W. (1983) · American Sociological Review · doi:10.2307/2095101 · Cited by 38,435
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
The Logic of Collective Action: Public Goods and the Theory of Groups.
Sufrin, S. C., & Olson, M. (1966) · Industrial and Labor Relations Review · doi:10.2307/2521228 · Cited by 11,518
Sorting Things Out: Classification and Its Consequences
Bowker, G. C., & Star, S. L. (1999) · doi:10.7551/mitpress/6352.001.0001 · Cited by 6,188
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
Sorting Things Out: Classification and Its Consequences
(2000) · The Journal of Academic Librarianship · doi:10.1016/s0099-1333(00)00155-5 · Cited by 5,210
Pandora's Hope: Essays on the Reality of Science Studies
Casper, M. J., & Latour, B. (2000) · Contemporary Sociology · doi:10.2307/2655272 · Cited by 5,049
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Wikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Privacy-preserving data mining
Agrawal, R., & Srikant, R. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335438 · Cited by 2,961
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Dataset Shift in Machine Learning
(2008) · doi:10.7551/mitpress/9780262170055.001.0001 · Cited by 1,553
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
Code/Space: Software and Everyday Life
Kitchin, R., & Dodge, M. (2011) · doi:10.7551/mitpress/9780262042482.001.0001 · Cited by 964
The looping effects of human kinds
Hacking, I. (1996) · Causal Cognition · doi:10.1093/acprof:oso/9780198524021.003.0012 · Cited by 954
Midgley, L., & Olson, M. (1969) · The Western Political Quarterly · doi:10.1177/106591296902200144 · Cited by 935
Evaluating Learning Algorithms: A Classification Perspective
Japkowicz, N., & Shah, M. (2011) · doi:10.1017/cbo9780511921803 · Cited by 767
Model-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694
Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations
Perry, W., McInnis, B., Price, C., Smith, S., & Hollywood, J. (2013) · doi:10.7249/rr233 · Cited by 693
Discrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676
(2011) · Cited by 675
Evidence-Based Policy: A Practical Guide to Doing It Better
Cartwright, N., & Hardie, J. (2012) · doi:10.1093/acprof:osobl/9780199841608.001.0001 · Cited by 672
Elements of Statistical Disclosure Control
Willenborg, L., & de Waal, T. (2001) · Lecture Notes in Statistics · doi:10.1007/978-1-4613-0121-9 · Cited by 649
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632
Against Prediction: Profiling, Policing, and Punishing in an Actuarial Age
Harcourt, B. E. (2007) · bepress Legal Repository · Cited by 599
We Need Both Exploratory and Confirmatory
Tukey, J. W. (1980) · The American Statistician · doi:10.1080/00031305.1980.10482706 · Cited by 571
McKeown, B., & Thomas, D. (2013) · Sage eBooks · doi:10.4135/9781483384412 · Cited by 544
Evidence-based policy: a practical guide to doing it better
(2013) · Choice Reviews Online · doi:10.5860/choice.50-5831 · Cited by 535
(2014) · Digital Commons at University of Maryland Carey Law (University of Maryland Francis King Carey School of Law) · Cited by 524
The threat of algocracy: reality, resistance and accommodation
Danaher, J. (2016) · Philosophy & Technology · doi:10.1007/s13347-015-0211-1 · Cited by 519
Principles and procedures of exploratory data analysis.
Behrens, J. T. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.2.131 · Cited by 465
Tickle, A. B., Andrews, R., Golea, M., & Diederich, J. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.728352 · Cited by 438
Bizer, C., Heath, T., & Berners-Lee, T. (2011) · Semantic Services, Interoperability and Web Applications · doi:10.4018/978-1-60960-593-3.ch008 · Cited by 426
Statistical Disclosure Control
Hundepool, A., Domingo‐Ferrer, J., Franconi, L., Giessing, S., Nordholt, E. S., Spicer, K., & de Wolf, P. (2012) · Wiley eBooks · doi:10.1002/9781118348239 · Cited by 385
Code/Space: Software and Everyday Life
Zook, M. (2012) · Regional Studies · doi:10.1080/00343404.2012.696477 · Cited by 381
An Overview of Concept Drift Applications
Žliobaitė, I., Pechenizkiy, M., & Gama, J. (2016) · Studies in Big Data · doi:10.1007/978-3-319-26989-4_4 · Cited by 379
Smart Technologies and the End(s) of Law: Novel Entanglements of Law and Technology
Hildebrandt, M. (2015) · doi:10.4337/9781849808774 · Cited by 353
Accountability in a computerized society
Nissenbaum, H. (1996) · Science and Engineering Ethics · doi:10.1007/bf02639315 · Cited by 346
A Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312
Inherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
(2008) · Choice Reviews Online · doi:10.5860/choice.46-2349 · Cited by 267
Obfuscation: A User's Guide for Privacy and Protest
Brunton, F., & Nissenbaum, H. (2015) · The MIT Press eBooks · doi:10.7551/mitpress/9780262029735.001.0001 · Cited by 256
Slave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244
Self-citationProfiling the European Citizen
(2008) · Springer eBooks · doi:10.1007/978-1-4020-6914-7 · Cited by 215
From Freebase to Wikidata: The Great Migration
Pellissier Tanon, T., Vrandečić, D., Schaffert, S., Steiner, T., & Pintscher, L. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2874809 · Cited by 195
Against Prediction: Profiling, Policing, and Punishing in an Actuarial Age
Harcourt, B. E. (2006) · doi:10.7208/chicago/9780226315997.001.0001 · Cited by 193
(2012) · Law, Governance and Technology Series · doi:10.1007/978-94-007-2543-0 · Cited by 183
A Review of Financial Accounting Fraud Detection based on Data Mining Techniques
Sharma, A., & Kumar Panigrahi, P. (2012) · International Journal of Computer Applications · doi:10.5120/4787-7016 · Cited by 179
We Need Both Exploratory and Confirmatory
Tukey, J. W. (1980) · The American Statistician · doi:10.2307/2682991 · Cited by 169
Mantelero, A. (2016) · Computer Law & Security Review · doi:10.1016/j.clsr.2016.01.014 · Cited by 147
Data protection impact assessments: a meta-regulatory approach
Binns, R. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipw027 · Cited by 130
Self-citationHandling Conditional Discrimination
liobaite, I., Kamiran, F., & Calders, T. (2011) · 2011 IEEE 11th International Conference on Data Mining · doi:10.1109/icdm.2011.72 · Cited by 125
Žliobaitė, I., & Custers, B. (2016) · Artificial Intelligence and Law · doi:10.1007/s10506-016-9182-5 · Cited by 125
Children in the Public Benefit System at Risk of Maltreatment
Vaithianathan, R., Maloney, T., Putnam-Hornstein, E., & Jiang, N. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.04.022 · Cited by 122
Machine Learning in Adversarial Settings
McDaniel, P., Papernot, N., & Celik, Z. B. (2016) · IEEE Security & Privacy · doi:10.1109/msp.2016.51 · Cited by 120
Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation
Binns, R., Veale, M., Kleek, M. G. V., & Shadbolt, N. (2017) · UCL Discovery (University College London) · Cited by 113
Self-citationCollaborative ontology engineering: a survey
Simperl, E. P. B., & Luczak–Roesch, M. (2013) · The Knowledge Engineering Review · doi:10.1017/s0269888913000192 · Cited by 110
KDD: The challenge to individualism
Vedder, A. (1999) · Ethics and Information Technology · doi:10.1023/a:1010016102284 · Cited by 102
Discrimination and Privacy in the Information Society: Data Mining and Profiling in Large Databases
(2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3 · Cited by 100
Obfuscation: A User's Guide for Privacy and Protest
Cramer, B. W. (2016) · Journal of Information Policy · doi:10.5325/jinfopoli.6.2016.0475 · Cited by 100
Law, Human Agency and Autonomic Computing
(2011) · Routledge eBooks · doi:10.4324/9780203828342 · Cited by 81
Hajian, S., Domingo‐Ferrer, J., & Farràs, O. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0346-1 · Cited by 73
A framework for enhancing the social good in computing education: a values approach
Goldweber, M., Barr, J., Clear, T., Davoli, R., Mann, S., Patitsas, E., & Portnoff, S. (2013) · ACM Inroads · doi:10.1145/2432596.2432616 · Cited by 65
Enhancing the social issues components in our computing curriculum: computing for the social good
Goldweber, M., Davoli, R., Little, J. C., Riedesel, C., Walker, H., Cross, G., & Von Konsky, B. R. (2011) · ACM Inroads · doi:10.1145/1929887.1929907 · Cited by 57
Techniques for Discrimination-Free Predictive Models
Kamiran, F., Calders, T., & Pechenizkiy, M. (2012) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-642-30487-3_12 · Cited by 51
Berendt, B., & Preibusch, S. (2014) · Artificial Intelligence and Law · doi:10.1007/s10506-013-9152-0 · Cited by 44
Finerman, A. (1969) · ACM Computing Surveys · doi:10.1145/356540.356541 · Cited by 44
A Comparative Analysis of Anti-Discrimination and Data Protection Legislations
Gellert, R. M., de Vries, K., Hert, P. D., & Gutwirth, S. (2012) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-642-30487-3_4 · Cited by 42
Khwaja, M. S., Awasthi, R., & Loeprick, J. (2011) · The World Bank Open Knowledge Repository (World Bank) · doi:10.1596/978-0-8213-8754-2 · Cited by 40
Beyond Innovation and Competition: The Need for Qualified Transparency in Internet IntermediariesPreprint
Pasquale, F. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1686043 · Cited by 39
From Data Minimization to Data Minimummization
van der Sloot, B. (2012) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-642-30487-3_15 · Cited by 38
Linking multiple workflow provenance traces for interoperable collaborative science
Missier, P., Ludäscher, B., Bowers, S., et al. (2010) · doi:10.1109/works.2010.5671861 · Cited by 38
Data utility and privacy protection trade-off in k-anonymisation
Loukides, G., & Shao, J. (2008) · doi:10.1145/1379287.1379296 · Cited by 35
Simultaneous Generation of Accurate and Interpretable Neural Network Classifiers
Jin, Y., Sendhoff, B., & Körner, E. (2006) · Springer eBooks · doi:10.1007/3-540-33019-4_13 · Cited by 29
Synchronous digital circuits as functional programs
Gammie, P. (2013) · ACM Computing Surveys · doi:10.1145/2543581.2543588 · Cited by 28
A Study on the Impact of Data Anonymization on Anti-discrimination
Hajian, S., & Domingo‐Ferrer, J. (2012) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2012.19 · Cited by 20
Veale, M. (2017) · doi:10.31235/osf.io/6cdhe · Cited by 10
Self-citationEthics training and decision-making
Spradling, C. L., Soh, L., & Ansorge, C. J. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1352322.1352188 · Cited by 10
Visual data mining for higher-level patterns: Discrimination-aware data mining and beyond
Bo, G., & Berendt, B. (2011) · Lirias · Cited by 9
Technology, Virtuality and Utopia.
Rouvroy, A. (2009) · Cited by 7
Introducing Positive Discrimination in Predictive Models
Verwer, S., & Calders, T. (2012) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-642-30487-3_14 · Cited by 7
Blank, A., Cram, F., Dare, T., et al. (2015) · ResearchSpace (University of Auckland) · Cited by 5
Cited by
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
The Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
Fairness in Machine Learning: Lessons from Political Philosophy
Binns, R. (2017) · arXiv · doi:10.48550/arxiv.1712.03586 · Cited by 676
Self-citation
Metadata
- Authors
- Veale, M., & Binns, R.
- Affiliations
- University College London (GB)
- University of Oxford (GB)
- Corresponding author
- Veale, Michael
- Venue
- Big Data & Society
- Publisher
- California Digital Library (CDL)
- Published
- 2017-10-27
- Volume
- 4
- Issue
- 2
- Pages
- 205395171774353-205395171774353
- ISSN
- 2053-9517 (ISSN-L)
- OpenAlex
- W2765146466
- Cited by
- 512
- References
- 48
- Citations by year
- 2026: 84 · 2025: 97 · 2024: 59 · 2023: 74 · 2022: 78 · 2021: 50 · 2020: 35 · 2019: 20 · 2018: 14 · 2017: 1
- FWCI
- 43.94
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Locations
- Big Data & Society (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/J017728/2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- algorithmic fairness, indirect discrimination, sensitive attributes, trusted third party, collaborative platforms, unsupervised learning, interpretable algorithms, algorithmic harms, redlining, socio-technical systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.