Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making
Open accessPreprint — published as 10.1145/3173574.3174014 · Paper page
Abstract
Cite as:Michael Veale, Max Van Kleek and Reuben Binns (2018) Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. ACM Conference on Human Factors in Computing Systems (CHI'18). doi: 10.1145/3173574.3174014Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions—like taxation, justice, and child protection—are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The results suggest a disconnect between organisational and institutional realities, constraints and needs, and those addressed by current research into usable, transparent and 'discrimination-aware' machine learning—absences likely to undermine practical initiatives unless addressed. We see design opportunities in this disconnect, such as in supporting the tracking of concept drift in secondary data sources, and in building usable transparency tools to identify risks and incorporate domain knowledge, aimed both at managers and at the `street-level bureaucrats' on the frontlines of public service. We conclude by outlining ethical challenges and future directions for collaboration in these high-stakes applications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work on machine learning in the public sector suggests that such systems are often significantly shaped by (and sometimes overridden in light of) the values of those designing, operating, and responding to them on the ground.
“Recent work on the use of machine learning in the public sector suggests that such systems are often signi fi cantly shaped by (and sometimes overridden in light of) the values of those designing, operating, and responding to them on the ground in areas like taxation, law enforcement, and child welfare (Veale et al . 2018).”
From Human Judgment in algorithmic loops: Individual justice and automated <scp>decision‐making</scp> (2022) · cites this paper as Veale et al . 2018
The Introduction and Background and Motivation passages (page numbers unavailable) describe value-laden design challenges, subjective policy choices, and the exercise of public-service values through frontline discretion. The paper also identifies taxation, policing, and child protection as areas studied. These passages support the claim that people on the ground shape how systems are designed and used, but they do not establish that system recommendations are sometimes overridden in light of those values. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Street-Level Bureaucracy: Dilemmas of the Individual in Public Services
Lipsky, M. (1981) · Michigan Law Review · doi:10.2307/1288305 · Cited by 8,981 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · see citing works
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · see citing works
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · see citing works
Friedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203 · see citing works
A critical perspective on action research as a method for information systems research
Baskerville, R. L., & Wood-Harper, A. T. (1996) · Journal of Information Technology · doi:10.1080/026839696345289 · Cited by 1,082 · see citing works
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · see citing works
Veale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512 · see citing works
Self-citationAutomation Bias: Decision Making and Performance in High-Tech Cockpits
Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0801_3 · Cited by 353 · see citing works
Slave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244 · see citing works
Self-citationViewpoint: choosing appropriate information systems research methodologies
Galliers, R. D., & Land, F. F. (1987) · Communications of the ACM · doi:10.1145/32206.315753 · Cited by 181 · see citing works
Runaway Feedback Loops in Predictive Policing
Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., & Venkatasubramanian, S. (2017) · arXiv · doi:10.48550/arxiv.1706.09847 · Cited by 161 · see citing works
THE ‘IRON CAGE’ STRENGTHENED? DISCRETION AND DIGITAL DISCIPLINE
JORNA, F., & WAGENAAR, P. (2007) · Public Administration · doi:10.1111/j.1467-9299.2007.00640.x · Cited by 107 · see citing works
Yang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · see citing works
Adepeju, M., Rosser, G., & Cheng, T. (2016) · International Journal of Geographical Information Science · doi:10.1080/13658816.2016.1159684 · Cited by 75 · see citing works
"Shadow security" as a tool for the learning organization
Kirlappos, I., Parkin, S., & Sasse, M. A. (2015) · ACM SIGCAS Computers and Society · doi:10.1145/2738210.2738216 · Cited by 65 · see citing works
Le Dantec, C. A., & Edwards, W. K. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753345 · Cited by 58 · see citing works
More accountability for big-data algorithms
(2016) · Nature · doi:10.1038/537449a · Cited by 31 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · see citing works
Wieringa, M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372833 · Cited by 321 · see citing works
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229 · see citing works
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476089 · Cited by 134 · see citing works
Oswald, M. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0359 · Cited by 132 · see citing works
Human Judgment in algorithmic loops: Individual justice and automated <scp>decision‐making</scp>
Binns, R. (2022) · Regulation & Governance · doi:10.1111/rego.12358 · Cited by 134 · see citing works
Self-citationFAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14 · see citing works
Disclosure or Marketing? Analyzing the Efficacy of Vendor Self-reports for Vetting Public-sector AI
Kuehnert, B., Johnson, N., Dotan, R., & Heidari, H. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3812220
- Authors
- Veale, M., Van Kleek, M., & Binns, R.
- Affiliations
- University College London (GB)
- University of Oxford (GB)
- Publisher
- Center for Open Science
- Published
- 2018-02-04
- OpenAlex
- W3125947019
- Cited by
- 60
- References
- 22
- Citations by year
- 2026: 11 · 2025: 7 · 2024: 3 · 2023: 4 · 2022: 9 · 2021: 11 · 2020: 4 · 2019: 9 · 2018: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- SocArXiv (OSF Preprints) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/M507970/1, EP/J017728/2, EP/J017728, M507970)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Keywords
- algorithmic fairness, accountability, public sector decision-making, algorithmic transparency, concept drift, secondary data sources, street-level bureaucrats, public services, high-stakes decisions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
