The DOI matches its Crossref record; 87 of the 88 references it cites were found in a registry or an index; 10 of the 43 sources its claims cite were checked for support.
Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making
Open accessPublished version of a preprint
Abstract
Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions-like taxation, justice, and child protection-are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The results suggest a disconnect between organisational and institutional realities, constraints and needs, and those addressed by current research into usable, transparent and 'discrimination-aware' machine learning-absences likely to undermine practical initiatives unless addressed. We see design opportunities in this disconnect, such as in supporting the tracking of concept drift in secondary data sources, and in building usable transparency tools to identify risks and incorporate domain knowledge, aimed both at managers and at the 'street-level bureaucrats' on the frontlines of public service. We conclude by outlining ethical challenges and future directions for collaboration in these high-stakes applications.
Overview
CoChat Summary
TL;DR
Interviews with 27 public-sector machine-learning practitioners across five OECD countries reveal a disconnect between organisational realities and current fairness, accountability, and transparency research. The authors identify design opportunities around data drift, discretion, performance communication, and social practices, while cautioning that FAT/ML tools cannot simply be airdropped into high-stakes public decision-making.
Key findings
- Practitioners often chose simpler, more transparent models to secure internal buy-in: tax agencies used logistic regression or random forest, and one police contractor reduced 18,000 variables to 200, then 20, then 8, targeting around 75% accuracy.
- Performance communication was difficult: one police modeller reported “100% accuracy” but only 20% precision because of 40 million records with few crashes, while a tax analytics lead said people “tend to lose faith” if their preferred risk indicators are absent.
- Data changes and feedback loops were prominent: a human trafficking model became a car-wash detector after intelligence focused on car washes, and a predictive policing project saw the probability of being correct “tank” before re-evaluating learning every 2–3 weeks.
- Discretion and reliance varied by context: helicopter pilots resisted directions while motorcycle police liked sequential high-risk lists; an ethics committee advised working down rank-ordered vulnerability lists, with professional judgement always overriding the system.
Methods
- Semi-structured interviews with 27 individuals in late 2016, predominantly public servants and attached contractors in modelling or project management, across 5 OECD countries over 3 continents; 17 interviews were in person and 10 by telephone, each lasting 40–60 minutes with one interviewer; the US was not included.
- Ad hoc sampling drew on grey literature, freedom of information requests, snowball sampling, direct inquiries, and news databases; application areas included taxation, child protection, policing, justice, emergency response, and interior security.
- Conversations were not recorded; verbose notes were taken and typed the same day, then analysed with open coding in NVivo 11 and iteratively grouped using a public sector values framework.
Limitations
- Stated in the paper: the sample was not intended to be representative, used ad hoc sampling, and the countries are not named to reduce informant identification; the US was excluded; audio recording was avoided due to sensitivity, so the study relied on notes; the authors did not seek to validate existing theoretical frameworks.
- Evident from the text: findings rest on interviewer notes rather than transcripts, and the paper does not describe independent coding, member checking, or quantification of themes; the small, snowballed sample may overrepresent engaged practitioners.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 2 Not valid · 26 Not checked · 60 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Uptake of models in government is as much a social process as it is a technical one [50].”
What the sources say
- Supportedcites [50] 50. Daniel Antony Kolkman, Paolo Campo, Tina Balke-Visser, and Nigel Gilbert. 2016. How to build models for government: Criteria driving model acceptance in policymaking. Policy Sciences 49, 4 (2016), 489-504. DOI: [http://dx.doi.org/10.1007/s11077-016-9250-4](http://dx.doi.org/10.1007/s11077-016-9250-4) source
The Abstract concludes that model acceptance depends on model characteristics, supporting infrastructure, and organizational factors. The discussion of relative importance also identifies participation in development and the developer’s reputation as contributors, supporting the claim that uptake has social as well as technical dimensions. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This chimes with a common observation from clinical decision support-that designed tools, while promising in laboratories, often fail in practice in a variety of ways [47].”
What the sources say
- Partly supportedcites [47] 47. Kensaku Kawamoto, Caitlin A Houlihan, E Andrew Balas, and David F Lobach. 2005. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 330, 7494 (2005), 765. DOI: http://dx.doi.org/10.1136/bmj.38398.500764.8F sourceExperimental — accuracy not yet measured
The Introduction says clinical decision support systems do not always improve practice and reports that 34% of systems in a prior review did not significantly improve it. The Additional information section (p. 7) describes the systems as promising but not always effective in clinical practice. Neither passage establishes that they were promising in laboratories before failing in practice, and “often” overstates the reported failure rate. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Bias or opacity are far from newly observed characteristics of computer systems generally [26, 11, 24], but these issues have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.”
What the sources say
- Supportedcites [26] 26. Batya Friedman and Helen Nissenbaum. 1996. Bias in Computer Systems. ACM Trans. Inf. Syst. 14, 3 (July 1996), 330-347. DOI: http://dx.doi.org/10.1145/230538.230561 source
The citation is attached to the first clause, which says bias or opacity in computer systems is not newly observed. The Introduction describes biased systems and explains that their biases can remain hidden in complex code. The later claim about rapidly expanding data-driven systems is not part of the cited clause. (LLM, conf=0.88)
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [24] 24. Gerhard Fischer. 1991. The importance of models in making complex systems comprehensible. In Mental Models and Human-Computer Interaction , MJ Tauber and D Ackermann (Eds.). Elsevier, Noord Holland. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As observed in studies of clinical decision tools, which often fail to be adopted [83], organisational conditions are often missed in design contexts.”
What the sources say
- Supportedcites [83] 83. Qian Yang, John Zimmerman, Aaron Steinfeld, Lisa Carey, and James F Antaki. 2016. Investigating the Heart Pump Implant Decision Process: Opportunities for Decision Support Tools to Help. In Proceedings of the 2016 SIGCHI Conference on Human Factors in Computing Systems, CHI'16 . 4477-4488. DOI: http://dx.doi.org/10.1145/2858036.2858373 source
In Discussion, under “Barriers of DST Adoption” (page unavailable), the paper says that insufficient attention to social context in clinical decision-support tool design is a significant deficit. Its Related Work section also says prior design research offers few insights into integrating tools into clinical environments. Together, these support the citing sentence’s point about overlooked organisational conditions in this clinical design context. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Alexandra Chouldechova's impossibility theorem illustrating that independently plausible formal definitions of fairness can be statistically incompatible with one another [13]”
What the sources say
- Supportedcites [13] 13. Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5, 2 (2017), 153-163. DOI: http://dx.doi.org/10.1089/big.2016.0047 source
Section 2.1 (p. 2) shows that when recidivism prevalence differs between groups, a test-fair score cannot also have equal false-positive and false-negative rates across those groups. The Discussion (p. 4) likewise says balancing error rates will generally sacrifice freedom from predictive bias. This supports the claimed incompatibility, subject to the paper’s stated conditions. (LLM, conf=0.92)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
34 more claims · 2 with their cited sources checked
Create a free account to see all 98 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 48
- It may state two separate claims as one. 22
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work has argued that algorithms in the public sector is a domain in its own right [34,62,107,109,121] and must be characterized differently as compared to algorithms in the private sector where the decision outcomes are well-defined.”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [121]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Citizens and front-line bureaucrats are unaware of how data generated through their interactions might be used downstream for data mining and machine learning (Veale et al., 2018) raising concerns about consent.”
From AI adoption and diffusion in public administration: A systematic literature review and future research agenda (2023) · cites this paper as Veale et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another concern is internal gaming by public administration employees towards opportunistic behaviours similar to performance measures being manipulated to meet specific targets for funding”
From AI adoption and diffusion in public administration: A systematic literature review and future research agenda (2023) · cites this paper as Veale et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Investigative journalism tools such as Algorithm Tips [83], a database of algorithmic tools being used by public agencies, can also make information on algorithmic deployment more publicly accessible.”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [83]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Currently, the literature relies unilaterally on convenience datasets (enabling comparative studies), often from the UCI repository [13] or similar with limited industry context and engagement [193,286,287].”
From Fairness in machine learning: A survey (2024) · cites this paper as [287]
Experimental — accuracy not yet measured
15 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Gaming by decision-support users · p. 5
“One tax analytics lead [X2] worried that releasing the input variables and their weightings in a model could make their own auditors investigate according to their perception of the model structure, rather than the actual model outputs”
§ Gaming by decision-support users · p. 6
“The effect these conflicting incentives might have on a model were highlighted by a different tax agency [X2]”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 2×
“Research stemming from the early days of machine learning in expert systems attempts to make algorithmic logics more transparent [75, 81, 54, 66], or to better understand how users work with inductive systems and build mental models of them [60, 48, 77].”
BackgroundA PUBLIC MANAGEMENT FOR ALL SEASONS
HOOD, C. (1991) · Public Administration · doi:10.1111/j.1467-9299.1991.tb00779.x · Cited by 10,159 · cited 1×
“then steered at opportune moments in the discussion towards topics of fairness and accountability, effectiveness and complexity/robustness (mirroring the public sector values framework introduced by [36]). At times, this steering was unnecessary and avoided, particularly as the nature of the study was made clear to pa…”
BackgroundStreet-Level Bureaucracy: Dilemmas of the Individual in Public Services
Lipsky, M. (1981) · Michigan Law Review · doi:10.2307/1288305 · Cited by 8,981
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Many of the issues around algorithmic transparency so far have focussed on external algorithmic accountability and transparency-based rights, such as a ‘right to an explanation’ [20], although broad reasons to make systems transparent and interpretable exist [53].”
BackgroundThe Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“t shifts in distributions potentially relevant to a modelling task. Concept drift detection, particularly in complex real-world contexts, is difficult and daunting theoretically, let alone practically [65,28]. Some of the more recent reviews in the field call for the integration of domain knowledge in order to discern r…”
BackgroundDwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“The most well-known of the latter, the Workshop on Fairness, Accountability and Transparency in Machine Learning (FAT/ML), has fast become a convening venue and acronym for scholars, initially computer scientists and lawyers but increasingly ethnographers and political scientists, attempting to identify tangible ways…”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 2×
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
BackgroundKawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
Winner, L. (2017) · Computer Ethics · doi:10.4324/9781315259697-21 · Cited by 2,317 · cited 1×
“3174014 public and even their designers, slipping through processes of democracy and accountability by being misleadingly cloaked in the ‘neutral’ language of technology [82], and replicating problematic biases inherent in the historical datasets used to train them [5].”
Background
Research works printed but not linked: 13
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica (2016). https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing
- danah boyd. 2016. Undoing the neutrality of Big Data. Florida Law Review Forum 16 (2016), 226--232.
- Matthew Chalmers and Ian MacColl. 2003. Seamful and seamless design in ubiquitous computing. In Workshop at the crossroads: The interaction of HCI and systems issues in UbiComp, Vol. 8. https://perma.cc/2A3D-NMJP
- 10.5555/1213575 · doi:10.5555/1213575
- House of Common Science and Technology Committee. 2016. Robotics and artificial intelligence (HC 145). The House of Commons, London. http://www.publications.parliament.uk/pa/cm201617/ cmselect/cmsctech/145/145.pdf
- House of Commons Science and Technology Committee. 2016. The big data dilemma (HC 468). House of Commons, London. http://www.publications.parliament. uk/pa/cm201516/cmselect/cmsctech/468/468.pdf
- 10.5555/1964882 · doi:10.5555/1964882
- Robin Moore (Ed.). 2015. A compendium of research and analysis on the Offender Assessment System. Ministry of Justice Analytical Series London. DOI: http://dx.doi.org/https://perma.cc/W2FT-NFWZ
- BBC News. 2016. Kent slavery raids 'uncover 21 victims'. BBC News (7 Dec. 2016). https://perma.cc/AM4S-RMHR
- 10.5555/1462129 · doi:10.5555/1462129
- Nick Seaver. 2013. Knowing algorithms. Media in Transition 8 (2013). https://perma.cc/8USJ-VTWS
- The Royal Society. 2017. Machine learning: The power and promise of computers that learn by example. The Royal Society, London. https://royalsociety.org/~/ media/policy/projects/machine-learning/publications/ machine-learning-report.pdf
- The Royal Society and the British Academy. 2017. Data management and use: Governance in the 21st Century. The Royal Society and the British Academy, London. https://royalsociety.org/~/media/policy/projects/ data-governance/data-management-governance.pdf
Cited by
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
“[24, 53, 100, 106]), perhaps aided by tools for visualization and guided exploration [24, 69].”
BackgroundMethodInfluentialAlgorithmic bias: review, synthesis, and future research directions
Kordzadeh, N., & Ghasemaghaei, M. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1927212 · Cited by 770
“…the realm of algorithmic systems pertain to the socio-technical and interactive features provided through user interfaces, which can enhance the usability of those systems and enable users to detect biases and react to them accordingly (Binns et al., 2018; Dodge et al., 2019; Veale et al., 2018).”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
“[106] Michael Veale, Max Van Kleek, and Reuben Binns.”
BackgroundInfluentialKuziemski, M., & Misuraca, G. (2020) · Telecommunications Policy · doi:10.1016/j.telpol.2020.101976 · Cited by 610
“” (Veale, Van Kleek, & Binns, 2018).”
Metadata
- Authors
- Veale, M., Van Kleek, M., & Binns, R.
- Affiliations
- University College London (GB)
- University of Oxford, Oxford, Oxfordshire, United Kingdom
- University of Oxford (GB)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-14
- OpenAlex
- W2786242872
- Cited by
- 490
- References
- 88
- Citations by year
- 2026: 54 · 2025: 67 · 2024: 66 · 2023: 61 · 2022: 69 · 2021: 68 · 2020: 54 · 2019: 40 · 2018: 11
- FWCI
- 78.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- EPSRC (EP/M507970/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic fairness, accountability, public sector decision-making, algorithmic transparency, public value, concept drift, secondary data sources, street-level bureaucrats, public services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. https://app.cochat.ai/papers/10.1145/3173574.3174014
- The paper
- Veale, M., Van Kleek, M., & Binns, R. (2018). Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3174014
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.