Toward Accountable Discrimination-Aware Data Mining: The Importance of Keeping the Human in the Loop—and Under the Looking Glass
Open accessAbstract
"Big Data" and data-mined inferences are affecting more and more of our lives, and concerns about their possible discriminatory effects are growing. Methods for discrimination-aware data mining and fairness-aware data mining aim at keeping decision processes supported by information technology free from unjust grounds. However, these formal approaches alone are not sufficient to solve the problem. In the present article, we describe reasons why discrimination with data can and typically does arise through the combined effects of human and machine-based reasoning, and argue that this requires a deeper understanding of the human side of decision-making with data mining. We describe results from a large-scale human-subjects experiment that investigated such decision-making, analyzing the reasoning that participants reported during their task to assess whether a loan request should or would be granted. We derive data protection by design strategies for making decision-making discrimination-aware in an accountable way, grounding these requirements in the accountability principle of the European Union General Data Protection Regulation, and outline how their implementations can integrate algorithmic, behavioral, and user interface factors.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deployment of bigdata analyses will itself create data that become input to further data analyses, which can create vicious-cycle phenomena that perpetuate or aggravate bias and discrimination.
“In general, since the deployment of bigdata analyses will itself create data that then become input to further data analyses, this can easily create vicious-cycle phenomena that have been observed by sociologists for long, dynamics that can perpetuate or even aggravate bias and discrimination [64].”
From AI for the Common Good?! Pitfalls, challenges, and ethics pen-testing (2019) · cites this paper as [64]
Cites
Analysing and Assessing Accountability: A Conceptual Framework <sup>1</sup>
Bovens, M. (2007) · European Law Journal · doi:10.1111/j.1468-0386.2007.00378.x · Cited by 2,667 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
An American Dilemma: The Negro Problem and Modern Democracy.
Young, K., Myrdal, G., Sterner, R., & Rose, A. M. (1944) · American Sociological Review · doi:10.2307/2086087 · Cited by 2,017 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · see citing works
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · see citing works
Lum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611 · see citing works
Hoepman, J.-H. (2014) · IFIP advances in information and communication technology · doi:10.1007/978-3-642-55415-5_38 · Cited by 318 · see citing works
A Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312 · see citing works
A multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288 · see citing works
Measuring discrimination in algorithmic decision making
Žliobaitė, I. (2017) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-017-0506-1 · Cited by 272 · see citing works
Why Muslim Integration Fails in Christian-Heritage Societies
Adida, C. L., Laitin, D. D., & Valfort, M. (2016) · Harvard University Press eBooks · doi:10.4159/9780674088962 · Cited by 253 · see citing works
McNamara, G. M., & Bromiley, P. (1997) · Academy of Management Journal · doi:10.2307/256927 · Cited by 193 · see citing works
Increasing cheat robustness of crowdsourcing tasks
Eickhoff, C., & de Vries, A. P. (2012) · Information Retrieval · doi:10.1007/s10791-011-9181-9 · Cited by 171 · see citing works
Kamiran, F., Žliobaitė, I., & Calders, T. (2013) · Knowledge and Information Systems · doi:10.1007/s10115-012-0584-8 · Cited by 99 · see citing works
Lowry, S., & Macpherson, G. (1988) · BMJ · doi:10.1136/bmj.296.6623.657 · Cited by 72 · see citing works
Bygrave, L. (2001) · Computer Law and Security Review · doi:10.1016/s0267-3649(01)00104-2 · Cited by 65 · see citing works
Choice Architecture for Human-Computer Interaction
Jameson, A., Berendt, B., Gabrielli, S., Cena, F., Cristina Gena, A., Vernero, F., & Reinecke, K. (2014) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000028 · Cited by 64 · see citing works
Self-citationBerendt, B., & Preibusch, S. (2014) · Artificial Intelligence and Law · doi:10.1007/s10506-013-9152-0 · Cited by 44 · see citing works
Self-citationThe Statistical Physics of Learning (in a nutshell):News from the stoneage of neural networks
Biehl, M. L. (2018) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 40 · see citing works
The Meaning of ‘Accountability’ in the Information Privacy Context
Raab, C. D. (2012) · Palgrave Macmillan UK eBooks · doi:10.1057/9781137032225_2 · Cited by 35 · see citing works
Exploring Discrimination: A User-centric Evaluation of Discrimination-Aware Data Mining
Berendt, B., & Preibusch, S. (2012) · 2012 IEEE 12th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2012.109 · Cited by 24 · see citing works
Self-citationStrong Accountability: Beyond Vague Promises
Butin, D., Chicote, M., & Métayer, D. L. (2013) · Springer eBooks · doi:10.1007/978-94-007-7540-4_16 · Cited by 21 · see citing works
Considerations on Fairness-Aware Data Mining
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2012.101 · Cited by 15 · see citing works
Discrimination discovery in scientific project evaluation: A case study
Romei, A., Ruggieri, S., & Turini, F. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.05.016 · Cited by 13 · see citing works
Generalized discrimination discovery on semi-structured data supported by ontology
Thanh, B. L. (2011) · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · see citing works
Legal and human rights issues of AI: Gaps, challenges and vulnerabilities
Rodrigues, R. (2020) · Journal of Responsible Technology · doi:10.1016/j.jrt.2020.100005 · Cited by 494 · see citing works
Big Data and discrimination: perils, promises and solutions. A systematic review
Favaretto, M., Clercq, E. D., & Elger, B. S. (2019) · Journal Of Big Data · doi:10.1186/s40537-019-0177-4 · Cited by 223 · see citing works
Peeters, R. (2020) · Information Polity · doi:10.3233/ip-200253 · Cited by 78 · see citing works
Interpretability of Machine Learning Solutions in Public Healthcare: The CRISP-ML Approach
Kolyshkina, I., & Simoff, S. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.660206 · Cited by 60 · see citing works
Leonelli, S., Lovell, R., Wheeler, B. W., Fleming, L. E., & Williams, H. T. P. (2021) · Big Data & Society · doi:10.1177/20539517211010310 · Cited by 52 · see citing works
Explainable outlier detection: What, for Whom and Why?
Sejr, J. H., & Schneider-Kamp, A. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100172 · Cited by 32 · see citing works
Interpretability and Transparency in Artificial Intelligence
Mittelstadt, B. (2022) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198857815.013.20 · Cited by 30 · see citing works
Adomavicius, G., & Yang, M. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3519420 · Cited by 20 · see citing works
Bias in artificial intelligence systems
Rejmaniak, R. (2021) · Białostockie Studia Prawnicze · doi:10.15290/bsp.2021.26.03.02 · Cited by 22 · see citing works
Excitement and Concerns about Machine Learning-Based Chatbots and Talkbots: A Survey
Rivas, P., Holzmayer, K., Hernandez, C. G., & Grippaldi, C. (2018) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas.2018.8638280 · Cited by 14 · see citing works
Inteligencia artificial y derechos de las personas con discapacidad. El poder de los algoritmos
Escolano, R. V. (2023) · Revista Española de Discapacidad · doi:10.5569/2340-5104.11.01.01 · Cited by 12 · see citing works
AI for the Common Good?! Pitfalls, challenges, and ethics pen-testing
Berendt, B. (2019) · Paladyn, Journal of Behavioral Robotics · doi:10.1515/pjbr-2019-0004 · Cited by 8 · see citing works
“In general, since the deployment of bigdata analyses will itself create data that then become input to further data analyses, this can easily create viciouscycle phenomena that have been observed by sociologists for long, dynamics that can perpetuate or even aggravate bias and discrimination [64].”
BackgroundSelf-citationHeidary, K., van der Rest, J.-P., & Custers, B. (2024) · Internet Policy Review · doi:10.14763/2024.4.1809 · Cited by 7 · see citing works
Artificial intelligence and rights of people with disabilities: The power of algorithms
Escolano, R. V. (2023) · Revista Española de Discapacidad · doi:10.5569/2340-5104.11.01.02 · Cited by 5 · see citing works
Adomavicius, G., & Yang, M. (2019) · Social Science Research Network · doi:10.2139/ssrn.3446944 · Cited by 3 · see citing works
Fairness and Artificial Intelligence
Naudts, L., & Vedder, A. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009367783.006 · Cited by 4 · see citing works
Al-Zanbouri, Z., Sharma, G., & Raza, S. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00105 · Cited by 4 · see citing works
Schwerpunkt: Digitale Transformation und Gender Bias
Bröcher, L., Gredel, E., & Schelenz, L. (2023) · GENDER – Zeitschrift für Geschlecht Kultur und Gesellschaft · doi:10.3224/gender.v15i1 · Cited by 3 · see citing works
Lütz, F. (2023) · GENDER – Zeitschrift für Geschlecht Kultur und Gesellschaft · doi:10.3224/gender.v15i1.03 · Cited by 2 · see citing works
The Terminator of Social Welfare? The Economic Consequences of Algorithmic DiscriminationPreprint
Bauer, K., Pfeuffer, N., Abdel-Karim, B. M., Hinz, O. F., & Kosfeld, M. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3675313 · Cited by 2 · see citing works
Learning by Design: Structuring and Documenting the Human Choices in Machine Learning DevelopmentPreprint
Enni, S. A., & Assent, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.00687 · Cited by 2 · see citing works
Smuha, N. A. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009367783.002 · Cited by 1 · see citing works
Hütt, M. ‐., & Schubert, C. (2020) · Springer Reference Geisteswissenschaften · doi:10.1007/978-3-658-23715-8_39-1 · Cited by 1 · see citing works
Mere algorithms can be demotivating
Karumathil, A., & Tripathi, R. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-022-00112-5 · Cited by 1 · see citing works
Caiza, J. C., López, G., & Guamán, D. S. (2023) · Revista Politécnica · doi:10.33333/rp.vol51n2.10
Revista Española de Discapacidad
(2023) · Revista Española de Discapacidad · doi:10.5569/2340-5104.11.01
Chuan, C., Sun, R., Tian, S., & Tsai, W. S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4624907
The hard problem of the androcentric context of AI: challenges for EU policy agendas
González-Martín, J. A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-18851-0.00013-5 · Cited by 1 · see citing works
Hütt, M. ‐., & Schubert, C. (2024) · Springer eBooks · doi:10.1007/978-3-658-19606-6_39
Gaps in Algorithmic AI Audits: Legal and Transparency Issues in Cryptocurrency Trading
Arsyad, I., Kharisma, D. B., & Wiwoho, J. (2026) · International Journal of Legal Information · doi:10.1515/ijli-2025-0013
- Authors
- Berendt, B., & Preibusch, S.
- Affiliations
- KU Leuven (BE)
- Microsoft Research (United Kingdom) (GB)
- Corresponding author
- Berendt, Bettina
- Venue
- Big Data
- Publisher
- SAGE Publications
- Published
- 2017-06
- Volume
- 5
- Issue
- 2
- Pages
- 135-152
- ISSN
- 2167-6461 (ISSN-L)
- 2167-647X
- OpenAlex
- W2623475374
- Cited by
- 32
- References
- 27
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 3 · 2023: 7 · 2022: 3 · 2021: 5 · 2020: 5 · 2019: 5 · 2018: 1
- FWCI
- 6.89
- Type
- journal article
- Language
- en
- Open access
- green
- lirias.kuleuven.be
- Locations
- Big Data (publishedVersion)
- PubMed (publishedVersion)
- Lirias (submittedVersion, other-oa)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- accountability, human-in-the-loop decision making, loan approval decisions, data protection by design, General Data Protection Regulation, algorithmic fairness, user interface design
- MeSH terms
- Decision Making, Humans, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review