The DOI matches its Crossref record; 87 of the 88 references it cites were found in a registry or an index; 10 of the 43 sources its claims cite were checked for support.
Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making
Open accessPublished version of a preprint
Abstract
Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions-like taxation, justice, and child protection-are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The results suggest a disconnect between organisational and institutional realities, constraints and needs, and those addressed by current research into usable, transparent and 'discrimination-aware' machine learning-absences likely to undermine practical initiatives unless addressed. We see design opportunities in this disconnect, such as in supporting the tracking of concept drift in secondary data sources, and in building usable transparency tools to identify risks and incorporate domain knowledge, aimed both at managers and at the 'street-level bureaucrats' on the frontlines of public service. We conclude by outlining ethical challenges and future directions for collaboration in these high-stakes applications.
Overview
CoChat Summary
TL;DR
Interviews with 27 public-sector machine-learning practitioners across five OECD countries reveal a disconnect between organisational realities and current fairness, accountability, and transparency research. The authors identify design opportunities around data drift, discretion, performance communication, and social practices, while cautioning that FAT/ML tools cannot simply be airdropped into high-stakes public decision-making.
Key findings
- Practitioners often chose simpler, more transparent models to secure internal buy-in: tax agencies used logistic regression or random forest, and one police contractor reduced 18,000 variables to 200, then 20, then 8, targeting around 75% accuracy.
- Performance communication was difficult: one police modeller reported “100% accuracy” but only 20% precision because of 40 million records with few crashes, while a tax analytics lead said people “tend to lose faith” if their preferred risk indicators are absent.
- Data changes and feedback loops were prominent: a human trafficking model became a car-wash detector after intelligence focused on car washes, and a predictive policing project saw the probability of being correct “tank” before re-evaluating learning every 2–3 weeks.
- Discretion and reliance varied by context: helicopter pilots resisted directions while motorcycle police liked sequential high-risk lists; an ethics committee advised working down rank-ordered vulnerability lists, with professional judgement always overriding the system.
Methods
- Semi-structured interviews with 27 individuals in late 2016, predominantly public servants and attached contractors in modelling or project management, across 5 OECD countries over 3 continents; 17 interviews were in person and 10 by telephone, each lasting 40–60 minutes with one interviewer; the US was not included.
- Ad hoc sampling drew on grey literature, freedom of information requests, snowball sampling, direct inquiries, and news databases; application areas included taxation, child protection, policing, justice, emergency response, and interior security.
- Conversations were not recorded; verbose notes were taken and typed the same day, then analysed with open coding in NVivo 11 and iteratively grouped using a public sector values framework.
Limitations
- Stated in the paper: the sample was not intended to be representative, used ad hoc sampling, and the countries are not named to reduce informant identification; the US was excluded; audio recording was avoided due to sensitivity, so the study relied on notes; the authors did not seek to validate existing theoretical frameworks.
- Evident from the text: findings rest on interviewer notes rather than transcripts, and the paper does not describe independent coding, member checking, or quantification of themes; the small, snowballed sample may overrepresent engaged practitioners.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 2 Not valid · 26 Not checked · 60 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Uptake of models in government is as much a social process as it is a technical one [50].”
What the sources say
- Supportedcites [50] 50. Daniel Antony Kolkman, Paolo Campo, Tina Balke-Visser, and Nigel Gilbert. 2016. How to build models for government: Criteria driving model acceptance in policymaking. Policy Sciences 49, 4 (2016), 489-504. DOI: [http://dx.doi.org/10.1007/s11077-016-9250-4](http://dx.doi.org/10.1007/s11077-016-9250-4) source
The Abstract concludes that model acceptance depends on model characteristics, supporting infrastructure, and organizational factors. The discussion of relative importance also identifies participation in development and the developer’s reputation as contributors, supporting the claim that uptake has social as well as technical dimensions. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This chimes with a common observation from clinical decision support-that designed tools, while promising in laboratories, often fail in practice in a variety of ways [47].”
What the sources say
- Partly supportedcites [47] 47. Kensaku Kawamoto, Caitlin A Houlihan, E Andrew Balas, and David F Lobach. 2005. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 330, 7494 (2005), 765. DOI: http://dx.doi.org/10.1136/bmj.38398.500764.8F sourceExperimental — accuracy not yet measured
The Introduction says clinical decision support systems do not always improve practice and reports that 34% of systems in a prior review did not significantly improve it. The Additional information section (p. 7) describes the systems as promising but not always effective in clinical practice. Neither passage establishes that they were promising in laboratories before failing in practice, and “often” overstates the reported failure rate. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Bias or opacity are far from newly observed characteristics of computer systems generally [26, 11, 24], but these issues have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.”
What the sources say
- Supportedcites [26] 26. Batya Friedman and Helen Nissenbaum. 1996. Bias in Computer Systems. ACM Trans. Inf. Syst. 14, 3 (July 1996), 330-347. DOI: http://dx.doi.org/10.1145/230538.230561 source
The citation is attached to the first clause, which says bias or opacity in computer systems is not newly observed. The Introduction describes biased systems and explains that their biases can remain hidden in complex code. The later claim about rapidly expanding data-driven systems is not part of the cited clause. (LLM, conf=0.88)
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [24] 24. Gerhard Fischer. 1991. The importance of models in making complex systems comprehensible. In Mental Models and Human-Computer Interaction , MJ Tauber and D Ackermann (Eds.). Elsevier, Noord Holland. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As observed in studies of clinical decision tools, which often fail to be adopted [83], organisational conditions are often missed in design contexts.”
What the sources say
- Supportedcites [83] 83. Qian Yang, John Zimmerman, Aaron Steinfeld, Lisa Carey, and James F Antaki. 2016. Investigating the Heart Pump Implant Decision Process: Opportunities for Decision Support Tools to Help. In Proceedings of the 2016 SIGCHI Conference on Human Factors in Computing Systems, CHI'16 . 4477-4488. DOI: http://dx.doi.org/10.1145/2858036.2858373 source
In Discussion, under “Barriers of DST Adoption” (page unavailable), the paper says that insufficient attention to social context in clinical decision-support tool design is a significant deficit. Its Related Work section also says prior design research offers few insights into integrating tools into clinical environments. Together, these support the citing sentence’s point about overlooked organisational conditions in this clinical design context. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Alexandra Chouldechova's impossibility theorem illustrating that independently plausible formal definitions of fairness can be statistically incompatible with one another [13]”
What the sources say
- Supportedcites [13] 13. Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5, 2 (2017), 153-163. DOI: http://dx.doi.org/10.1089/big.2016.0047 source
Section 2.1 (p. 2) shows that when recidivism prevalence differs between groups, a test-fair score cannot also have equal false-positive and false-negative rates across those groups. The Discussion (p. 4) likewise says balancing error rates will generally sacrifice freedom from predictive bias. This supports the claimed incompatibility, subject to the paper’s stated conditions. (LLM, conf=0.92)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
34 more claims · 2 with their cited sources checked
Create a free account to see all 98 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 48
- It may state two separate claims as one. 22
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work has argued that algorithms in the public sector is a domain in its own right [34,62,107,109,121] and must be characterized differently as compared to algorithms in the private sector where the decision outcomes are well-defined.”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [121]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Citizens and front-line bureaucrats are unaware of how data generated through their interactions might be used downstream for data mining and machine learning (Veale et al., 2018) raising concerns about consent.”
From AI adoption and diffusion in public administration: A systematic literature review and future research agenda (2023) · cites this paper as Veale et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another concern is internal gaming by public administration employees towards opportunistic behaviours similar to performance measures being manipulated to meet specific targets for funding”
From AI adoption and diffusion in public administration: A systematic literature review and future research agenda (2023) · cites this paper as Veale et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Investigative journalism tools such as Algorithm Tips [83], a database of algorithmic tools being used by public agencies, can also make information on algorithmic deployment more publicly accessible.”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [83]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Currently, the literature relies unilaterally on convenience datasets (enabling comparative studies), often from the UCI repository [13] or similar with limited industry context and engagement [193,286,287].”
From Fairness in machine learning: A survey (2024) · cites this paper as [287]
Experimental — accuracy not yet measured
15 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Gaming by decision-support users · p. 5
“One tax analytics lead [X2] worried that releasing the input variables and their weightings in a model could make their own auditors investigate according to their perception of the model structure, rather than the actual model outputs”
§ Gaming by decision-support users · p. 6
“The effect these conflicting incentives might have on a model were highlighted by a different tax agency [X2]”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 2×
“Research stemming from the early days of machine learning in expert systems attempts to make algorithmic logics more transparent [75, 81, 54, 66], or to better understand how users work with inductive systems and build mental models of them [60, 48, 77].”
BackgroundA PUBLIC MANAGEMENT FOR ALL SEASONS
HOOD, C. (1991) · Public Administration · doi:10.1111/j.1467-9299.1991.tb00779.x · Cited by 10,159 · cited 1×
“then steered at opportune moments in the discussion towards topics of fairness and accountability, effectiveness and complexity/robustness (mirroring the public sector values framework introduced by [36]). At times, this steering was unnecessary and avoided, particularly as the nature of the study was made clear to pa…”
BackgroundStreet-Level Bureaucracy: Dilemmas of the Individual in Public Services
Lipsky, M. (1981) · Michigan Law Review · doi:10.2307/1288305 · Cited by 8,981
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“Many of the issues around algorithmic transparency so far have focussed on external algorithmic accountability and transparency-based rights, such as a ‘right to an explanation’ [20], although broad reasons to make systems transparent and interpretable exist [53].”
BackgroundThe Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“t shifts in distributions potentially relevant to a modelling task. Concept drift detection, particularly in complex real-world contexts, is difficult and daunting theoretically, let alone practically [65,28]. Some of the more recent reviews in the field call for the integration of domain knowledge in order to discern r…”
BackgroundDwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“The most well-known of the latter, the Workshop on Fairness, Accountability and Transparency in Machine Learning (FAT/ML), has fast become a convening venue and acronym for scholars, initially computer scientists and lawyers but increasingly ethnographers and political scientists, attempting to identify tangible ways…”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 2×
“Given that performance metrics are one of the most valueladen parts of the machine learning process [5, 14], it will be key to discuss them both with statistical rigour and with practical relevance.”
BackgroundKawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
Winner, L. (2017) · Computer Ethics · doi:10.4324/9781315259697-21 · Cited by 2,317 · cited 1×
“3174014 public and even their designers, slipping through processes of democracy and accountability by being misleadingly cloaked in the ‘neutral’ language of technology [82], and replicating problematic biases inherent in the historical datasets used to train them [5].”
BackgroundFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 2×
“Yet issues like fairness have been shown to come with technically difficult to reconcile, or even irreconcilable trade-offs—something well-demonstrated by Alexandra Chouldechova’s impossibility theorem illustrating that independently plausible formal definitions of fairness can be statistically incompatible with one a…”
BackgroundCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 3×
“There are many points for intervention in a decision support system outside of the modelling process—for example, in the training data (many systems attempting to make fairer machine learning system intervene at this point [45, 23]) or after the model has been generated [46], such as the stage between model output and…”
BackgroundWHAT’S MEASURED IS WHAT MATTERS: TARGETS AND GAMING IN THE ENGLISH PUBLIC HEALTH CARE SYSTEM
BEVAN, G., & HOOD, C. (2006) · Public Administration · doi:10.1111/j.1467-9299.2006.00600.x · Cited by 1,580 · cited 1×
“This is despite how internal gaming is extensively highlighted in the public administration literature in relation to targets and the rise of New Public Management [7], a broad movement towards ‘rationalisation’ in the public sector that clearly affected informants around the world.”
BackgroundDataset Shift in Machine Learning
(2008) · doi:10.7551/mitpress/9780262170055.001.0001 · Cited by 1,553 · cited 1×
“t shifts in distributions potentially relevant to a modelling task. Concept drift detection, particularly in complex real-world contexts, is difficult and daunting theoretically, let alone practically [65,28]. Some of the more recent reviews in the field call for the integration of domain knowledge in order to discern r…”
BackgroundSome observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · cited 1×
“There are many points for intervention in a decision support system outside of the modelling process—for example, in the training data (many systems attempting to make fairer machine learning system intervene at this point [45, 23]) or after the model has been generated [46], such as the stage between model output and…”
BackgroundThe role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1×
“Over and under-reliance on decision support, extensively highlighted in the literature on automation bias [71, 18], featured considerably in informants’ responses.”
BackgroundFriedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · cited 1×
“Indeed, models might not fail in obvious ways, or might even be subject to attacks designed to change them in targeted ways [41].”
BackgroundJørgensen, T. B., & Bozeman, B. (2007) · Administration & Society · doi:10.1177/0095399707300703 · Cited by 1,082
(2015) · neural information processing systems · Cited by 912
Evaluating Learning Algorithms: A Classification Perspective
Japkowicz, N., & Shah, M. (2011) · doi:10.1017/cbo9780511921803 · Cited by 767
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 1×
“Over and under-reliance on decision support, extensively highlighted in the literature on automation bias [71, 18], featured considerably in informants’ responses.”
BackgroundDiscrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676 · cited 2×
“While these are early days for fairness/transparency methods, which are mostly under a decade old [64], the predominant mode of development often involves characterising a problem in a way that might often be at odds with the real world context—such as assuming you have access to the sensitive characteristics you are…”
Background(2011) · Cited by 675
Dunleavy, P., Margetts, H., Bastow, S., & Tinkler, J. (2006) · doi:10.1093/acprof:oso/9780199296194.001.0001 · Cited by 637
Varieties of knowledge elicitation techniques
Cooke, N. J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1083 · Cited by 579 · cited 1×
“Effective knowledge elicitation, as part of the hot topic of knowledge acquisition in heady days of expert systems, was thought to be a foundational building block of AI [15, 34].”
BackgroundEdwards, L., & Veale, M. (2017) · doi:10.31228/osf.io/97upg · Cited by 524
Self-citationVeale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512 · cited 2×
“While these are early days for fairness/transparency methods, which are mostly under a decade old [64], the predominant mode of development often involves characterising a problem in a way that might often be at odds with the real world context—such as assuming you have access to the sensitive characteristics you are…”
BackgroundSelf-citationVeale, M., & Binns, R. (2017) · doi:10.31235/osf.io/ustxg · Cited by 512
Self-citationThe relationship of action research to human-computer interaction
Hayes, G. R. (2011) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1993060.1993065 · Cited by 452 · cited 1×
“This assumption sits particularly uncomfortably with the value attributed to participatory design and action research in HCI and information systems [33, 6].”
BackgroundTickle, A. B., Andrews, R., Golea, M., & Diederich, J. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.728352 · Cited by 438 · cited 1×
“Research stemming from the early days of machine learning in expert systems attempts to make algorithmic logics more transparent [75, 81, 54, 66], or to better understand how users work with inductive systems and build mental models of them [60, 48, 77].”
BackgroundHoffman, R. R., Crandall, B., & Shadbolt, N. (1998) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872098779480442 · Cited by 403 · cited 1×
“With the inductive, data-driven turn, we may need to rediscover it as something which constrains and augments patterns learned from data, less around tough or rare cases [35] as much as around contentious, value-laden ones.”
BackgroundStreet-Level Bureaucracy and E-Government
Buffat, A. (2015) · Public Management Review · doi:10.1080/14719037.2013.771699 · Cited by 394 · cited 1×
“Where change does occur, managers might not notice it, as in their day-to-day roles or through their monitoring and evaluation tools, they only see part of the picture [10].”
BackgroundA Critical Perspective on Action Research as a Method for Information Systems Research
Baskerville, R. L., & Wood-Harper, A. T. (1996) · Journal of Information Technology · doi:10.1177/026839629601100305 · Cited by 390 · cited 2×
“This assumption sits particularly uncomfortably with the value attributed to participatory design and action research in HCI and information systems [33, 6].”
BackgroundAutomation Bias: Decision Making and Performance in High-Tech Cockpits
Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0801_3 · Cited by 353 · cited 1×
“Canonical domains in human factors have also focussed on even more dramatic life-and-death situations, such as issues around decision support in airplane cockpits [57] and air traffic control [37].”
BackgroundDigital Era Governance: IT Corporations, the State, and e-Government
Dunleavy, P., Margetts, H. Z., Bastow, S., & Tinkler, J. (2007) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · Cited by 340
Supersparse linear integer models for optimized medical scoring systems
Ustun, B., & Rudin, C. (2016) · Machine Learning · doi:10.1007/s10994-015-5528-6 · Cited by 313 · cited 1×
“The most well-known of the latter, the Workshop on Fairness, Accountability and Transparency in Machine Learning (FAT/ML), has fast become a convening venue and acronym for scholars, initially computer scientists and lawyers but increasingly ethnographers and political scientists, attempting to identify tangible ways…”
BackgroundDiscrimination Aware Decision Tree Learning
Kamiran, F., Calders, T., & Pechenizkiy, M. (2010) · 2010 IEEE International Conference on Data Mining · doi:10.1109/icdm.2010.50 · Cited by 303 · cited 1×
“There are many points for intervention in a decision support system outside of the modelling process—for example, in the training data (many systems attempting to make fairer machine learning system intervene at this point [45, 23]) or after the model has been generated [46], such as the stage between model output and…”
BackgroundThe governance of problems: Puzzling, powering and participation
Hoppe, R. (2010) · doi:10.1332/policypress/9781847429629.001.0001 · Cited by 286
Oswald, M., Grace, J., Urwin, S., & Barnes, G. C. (2018) · Information & Communications Technology Law · doi:10.1080/13600834.2018.1458455 · Cited by 247
Slave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244
Self-citationPage, E. C., & Jenkins, B. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199280414.001.0001 · Cited by 232
(2017) · Penn Carey Law Legal Scholarship Repository (University of Pennsylvania) · Cited by 228
Street-Level Bureaucracy: Dilemmas of the Individual in Public Services.
Goldner, F. H., & Lipsky, M. (1982) · Administrative Science Quarterly · doi:10.2307/2392554 · Cited by 212
Human Factors in Air Traffic Control
David Hopkin, V. (2017) · CRC Press eBooks · doi:10.1201/9780203751718 · Cited by 211 · cited 1×
“Canonical domains in human factors have also focussed on even more dramatic life-and-death situations, such as issues around decision support in airplane cockpits [57] and air traffic control [37].”
BackgroundDisparate Impact in Big Data Policing
Selbst, A. D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2819182 · Cited by 192 · cited 1×
“Those concerning race have taken centre stage, particularly in the US—ranging from discrimination in recidivism scores used in parole decisions [4, 13] to uneven demographic targeting in systems used for policing on the basis of spatiotemporal crime risk [70].”
BackgroundViewpoint: choosing appropriate information systems research methodologies
Galliers, R. D., & Land, F. F. (1987) · Communications of the ACM · doi:10.1145/32206.315753 · Cited by 181
Choosing Appropriate Information Systems Research Methodologies
Galliers, R. D., & Land, F. F. (2002) · Qualitative Research in Information Systems · doi:10.4135/9781849209687.n2 · Cited by 176 · cited 1×
“uggled to reconcile experimental traditions with the understanding that results from many methodologies fall apart when applied to political, noisy, stressful, complex and contested deployed settings [27]. In this paper, we seek to examine existing real-world contexts to assess whether FAT/ML tools might be airdropped…”
Background(2005) · Public law · Cited by 175
Mental models of robotic assistants
Kiesler, S. B., & Goetz, J. L. (2002) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/506443.506491 · Cited by 172 · cited 1×
“Research stemming from the early days of machine learning in expert systems attempts to make algorithmic logics more transparent [75, 81, 54, 66], or to better understand how users work with inductive systems and build mental models of them [60, 48, 77].”
BackgroundWetenschappelijke Raad voor het Regeringsbeleid
Godschalk, J. J. (1988) · Cited by 165
Runaway Feedback Loops in Predictive Policing
Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., & Venkatasubramanian, S. (2017) · arXiv · doi:10.48550/arxiv.1706.09847 · Cited by 161 · cited 2×
“In the worst cases, the model can have a polarising effect: directing police resources disproportionately to areas with slightly higher crime risk will, without corrections, skew future training data collection in those area, which might be demographically or socioeconomically disproportionate [21].”
BackgroundReconstructive expert system explanation
Wick, M. R., & Thompson, W. B. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90087-e · Cited by 158 · cited 1×
“d statistically assess it, assure it, or get closer to it [64, 32,23]. Research stemming from the early days of machine learning in expert systems attempts to make algorithmic logics more transparent [75,81,54,66], or to better understand how users work with inductive systems and build mental models of them [60,48,77]…”
BackgroundHow it works: a field study of non-technical users interacting with an intelligent system
Tullio, J., Dey, A. K., Chalecki, J., & Fogarty, J. A. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240630 · Cited by 149
Wetenschappelijke Raad voor het Regeringsbeleid
Elfring, T., & van der Aa, W. (1989) · Cited by 144
Some Observations on Mental Models
Norman, D. A. (2014) · Psychology Press eBooks · doi:10.4324/9781315802725-5 · Cited by 139 · cited 1×
“Research stemming from the early days of machine learning in expert systems attempts to make algorithmic logics more transparent [75, 81, 54, 66], or to better understand how users work with inductive systems and build mental models of them [60, 48, 77].”
BackgroundPolicy Bureaucracy: Government with a Cast of Thousands
Page, E. C., & Jenkins, B. (2005) · Cited by 109
THE ‘IRON CAGE’ STRENGTHENED? DISCRETION AND DIGITAL DISCIPLINE
JORNA, F., & WAGENAAR, P. (2007) · Public Administration · doi:10.1111/j.1467-9299.2007.00640.x · Cited by 107 · cited 2×
“It is a useful heuristic to assume systems are trained on ‘pure’ streams of data and then must be cleaned of bias downstream, but in real data collection environments, even upstream actors in the data collection process attempt to work in the discretionary places computer systems allow (and create) to inject fairness…”
BackgroundYang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · cited 2×
“ose bought arXiv:1802.01029v1 [cs.CY] 3 Feb 2018 from vendors, and are facing immediate, value laden challenges [14]. As observed in studies of clinical decision tools, which often fail to be adopted [83], organisational conditions are often missed in design contexts. Indeed fields studying technologies in practice hav…”
BackgroundDiscrimination and Privacy in the Information Society: Data Mining and Profiling in Large Databases
(2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3 · Cited by 100
Adepeju, M., Rosser, G., & Cheng, T. (2016) · International Journal of Geographical Information Science · doi:10.1080/13658816.2016.1159684 · Cited by 75 · cited 1×
“Other informants emphasised the importance of talking to users before determining performance metrics [X15], as in some cases only actionable knowledge is worth optimising for (see also [1]).”
Background"Shadow security" as a tool for the learning organization
Kirlappos, I., Parkin, S., & Sasse, M. A. (2015) · ACM SIGCAS Computers and Society · doi:10.1145/2738210.2738216 · Cited by 65 · cited 1×
“Lastly, researchers considering human factors in computer security have emphasised ‘shadow security practices’, which consist of ‘workarounds employees devise to ensure primary business goals are achieved’ and ‘reflect the working compromise staff find between security and “getting the job done”’ [49].”
BackgroundAviation Risk Perception: A Comparison Between Experts and Novices
Thomson, M. E., Önkal, D., Avcioğlu, A., & Goodwin, P. (2004) · Risk Analysis · doi:10.1111/j.0272-4332.2004.00552.x · Cited by 62 · cited 1×
“Research in risk perception by helicopter pilots has found additional disparities between experienced and inexperienced users which is also worth unpacking in this context [74].”
BackgroundUsing ethnography in the design of an explanation system
Forsythe, D. E. (1995) · Expert Systems with Applications · doi:10.1016/0957-4174(94)e0032-p · Cited by 62 · cited 1×
“Many ethical issues are only likely to surface far down the road of creation, procurement or deployment of a system, after many choices are baked-in [25].”
BackgroundSearching for build debt: experiences managing technical debt at Google
Morgenthaler, J. D., Gridnev, M., Sauciuc, R., & Bhansali, S. (2012) · doi:10.5555/2666036.2666037 · Cited by 61
Le Dantec, C. A., & Edwards, W. K. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753345 · Cited by 58 · cited 1×
“Their information systems are notorious for crossing scales and chains of accountability, engendering many practical challenges relating to performance and maintenance [51].”
BackgroundSearching for build debt: Experiences managing technical debt at Google
Morgenthaler, J. D., Gridnev, M., Sauciuc, R., & Bhansali, S. (2012) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1109/mtd.2012.6225994 · Cited by 55 · cited 1×
“The notion of a visibility debt has received some attention from engineers of both machine learning and traditional software engineering [56, 67].”
BackgroundDesigning Information for Remediating Cognitive Biases in Decision-Making
Zhang, Y., Bellamy, R. K. E., & Kellogg, W. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702239 · Cited by 51 · cited 1×
“The reason for this stems from an assumption of most work on decision-support systems: that it is relatively straight-forward to measure whether a design intervention would ‘reliably improve human decision-making, particularly in real-world contexts’ [84].”
BackgroundThe Importance of Models in Making Complex Systems Comprehensible
Fischer, G. (1991) · Human Factors in Information Technology · doi:10.1016/b978-0-444-88602-6.50005-0 · Cited by 43 · cited 1×
“Bias or opacity are far from newly observed characteristics of computer systems generally [26, 11, 24], but these issues have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.”
BackgroundA Comparative Analysis of Anti-Discrimination and Data Protection Legislations
Gellert, R. M., de Vries, K., Hert, P. D., & Gutwirth, S. (2012) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-642-30487-3_4 · Cited by 42 · cited 1×
“Techniques in discrimination-aware data mining attempt to define fairness (primarily in reference to anti-discrimination law [29]) and statistically assess it, assure it, or get closer to it [64, 32, 23].”
BackgroundHow to build models for government: criteria driving model acceptance in policymaking
Kolkman, D. A., Campo, P., Balke-Visser, T., & Gilbert, N. (2016) · Policy Sciences · doi:10.1007/s11077-016-9250-4 · Cited by 39 · cited 1×
“Uptake of models in government is as much a social process as it is a technical one [50].”
BackgroundOdendahl, T. J., & Shaw, A. M. (2001) · Sage eBooks · doi:10.4135/9781412973588.n19 · Cited by 37
Visualization in law enforcement
Chen, H., Atabakhsh, H., Tseng, C., Marshall, B. B., Kaza, S., Eggers, S., Gowda, H., Shah, A. H., Petersen, T., & Violette, C. (2005) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1056808.1056893 · Cited by 36 · cited 1×
“There are many points for intervention in a decision support system outside of the modelling process—for example, in the training data (many systems attempting to make fairer machine learning system intervene at this point [45, 23]) or after the model has been generated [46], such as the stage between model output and…”
BackgroundOdendahl, T. J., & Shaw, A. M. (2012) · Sage eBooks · doi:10.4135/9781412973588.d19 · Cited by 33
More accountability for big-data algorithms
(2016) · Nature · doi:10.1038/537449a · Cited by 31
Direct and Indirect Discrimination Prevention Methods
Hajian, S., & Domingo-Ferrer, J. (2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3_13 · Cited by 27 · cited 2×
“Techniques in discrimination-aware data mining attempt to define fairness (primarily in reference to anti-discrimination law [29]) and statistically assess it, assure it, or get closer to it [64, 32, 23].”
BackgroundDigital era governance: IT corporations, the state, and e-Government
Choi, I. (2016) · International Review of Public Administration · doi:10.1080/12294659.2016.1257549 · Cited by 23
Human Factors Contributions to Knowledge Elicitation
Hoffman, R. R. (2008) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x288475 · Cited by 22 · cited 1×
“Effective knowledge elicitation, as part of the hot topic of knowledge acquisition in heady days of expert systems, was thought to be a foundational building block of AI [15, 34].”
BackgroundMachines that learn in the wild: machine learning capabilities, limitations and implications
Armstrong, H. G. (2015) · Cited by 16
StatRec - Performance, Validation and Preservability of a Static Risk Prediction Instrument
Tollenaar, N., Wartna, B. S. J., Van der Heijden, P. G. M., & Bogaerts, S. (2016) · Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique · doi:10.1177/0759106315615504 · Cited by 7 · cited 1×
“Several public sector organisations using machine learning systems already publish information about the weights within their model, the variable importance scores, or record ethical experiences and challenges in the modelling process [76, 55, 62].”
BackgroundPolicy Bureaucracy: Government with a Cast of Thousands
Lowe, R. (2009) · Parliamentary Affairs · doi:10.1093/pa/gsp010 · Cited by 2 · cited 2×
“1See [63], who conducted 128 interviews in the UK civil service to understand the nature of policy work, asking only ‘what do you do?’ and ‘how do you come to be in this job?’.”
BackgroundVisualization in law enforcement
Chen, H., Homa Atabakhsh, A., Tseng, C., et al. (2005) · doi:10.5555/1065226.1065295 · Cited by 2
The governance of problems: Puzzling, powering and participation
Hoppe, R. (2010) · Policy Press eBooks · doi:10.56687/9781847426307 · Cited by 1
Cultures of public policy problems
Hoppe, R. (2010) · The governance of problems · doi:10.51952/9781847426307.ch004
Government with a Cast of Thousands
Page, E. C., & Jenkins, B. (2005) · Policy Bureaucracy · doi:10.1093/acprof:oso/9780199280414.003.0001
Research works printed but not linked: 13
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica (2016). https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing
- danah boyd. 2016. Undoing the neutrality of Big Data. Florida Law Review Forum 16 (2016), 226--232.
- Matthew Chalmers and Ian MacColl. 2003. Seamful and seamless design in ubiquitous computing. In Workshop at the crossroads: The interaction of HCI and systems issues in UbiComp, Vol. 8. https://perma.cc/2A3D-NMJP
- 10.5555/1213575 · doi:10.5555/1213575
- House of Common Science and Technology Committee. 2016. Robotics and artificial intelligence (HC 145). The House of Commons, London. http://www.publications.parliament.uk/pa/cm201617/ cmselect/cmsctech/145/145.pdf
- House of Commons Science and Technology Committee. 2016. The big data dilemma (HC 468). House of Commons, London. http://www.publications.parliament. uk/pa/cm201516/cmselect/cmsctech/468/468.pdf
- 10.5555/1964882 · doi:10.5555/1964882
- Robin Moore (Ed.). 2015. A compendium of research and analysis on the Offender Assessment System. Ministry of Justice Analytical Series London. DOI: http://dx.doi.org/https://perma.cc/W2FT-NFWZ
- BBC News. 2016. Kent slavery raids 'uncover 21 victims'. BBC News (7 Dec. 2016). https://perma.cc/AM4S-RMHR
- 10.5555/1462129 · doi:10.5555/1462129
- Nick Seaver. 2013. Knowing algorithms. Media in Transition 8 (2013). https://perma.cc/8USJ-VTWS
- The Royal Society. 2017. Machine learning: The power and promise of computers that learn by example. The Royal Society, London. https://royalsociety.org/~/ media/policy/projects/machine-learning/publications/ machine-learning-report.pdf
- The Royal Society and the British Academy. 2017. Data management and use: Governance in the 21st Century. The Royal Society and the British Academy, London. https://royalsociety.org/~/media/policy/projects/ data-governance/data-management-governance.pdf
Cited by
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
“[24, 53, 100, 106]), perhaps aided by tools for visualization and guided exploration [24, 69].”
BackgroundMethodInfluentialAlgorithmic bias: review, synthesis, and future research directions
Kordzadeh, N., & Ghasemaghaei, M. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1927212 · Cited by 770
“…the realm of algorithmic systems pertain to the socio-technical and interactive features provided through user interfaces, which can enhance the usability of those systems and enable users to detect biases and react to them accordingly (Binns et al., 2018; Dodge et al., 2019; Veale et al., 2018).”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
“[106] Michael Veale, Max Van Kleek, and Reuben Binns.”
BackgroundInfluentialKuziemski, M., & Misuraca, G. (2020) · Telecommunications Policy · doi:10.1016/j.telpol.2020.101976 · Cited by 610
“” (Veale, Van Kleek, & Binns, 2018).”
Metadata
- Authors
- Veale, M., Van Kleek, M., & Binns, R.
- Affiliations
- University College London (GB)
- University of Oxford, Oxford, Oxfordshire, United Kingdom
- University of Oxford (GB)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-14
- OpenAlex
- W2786242872
- Cited by
- 490
- References
- 88
- Citations by year
- 2026: 54 · 2025: 67 · 2024: 66 · 2023: 61 · 2022: 69 · 2021: 68 · 2020: 54 · 2019: 40 · 2018: 11
- FWCI
- 78.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- EPSRC (EP/M507970/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic fairness, accountability, public sector decision-making, algorithmic transparency, public value, concept drift, secondary data sources, street-level bureaucrats, public services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.