Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Abstract
Errors and biases are earning algorithms increasingly malignant reputations in society. A central challenge is that algorithms must bridge the gap between high-level policy and on-the-ground decisions, making inferences in novel situations where the policy or training data do not readily apply. In this paper, we draw on the theory of street-level bureaucracies, how human bureaucrats such as police and judges interpret policy to make on-the-ground decisions. We present by analogy a theory of street-level algorithms, the algorithms that bridge the gaps between policy and decisions about people in a socio-technical system. We argue that unlike street-level bureaucrats, who reflexively refine their decision criteria as they reason through a novel situation, street-level algorithms at best refine their criteria only after the decision is made. This loop-and-a-half delay results in illogical decisions when handling new or extenuating circumstances. This theory suggests designs for street-level algorithms that draw on historical design patterns for street-level bureaucracies, including mechanisms for self-policing and recourse in the case of error.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, regardless of the type of outcome, street-level bureaucrats in the public sector must still exercise discretion and contextualize the algorithmic results for each case and act within the constraints posed by bureaucratic processes as explained by the ADMAPS framework [2,62,91].”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [2]
In the citing paper
“machine-learning techniques in ADM for making administrative decisions cannot work in a rule of law situation because machine-learning is actually based on predictions, where a new decision is made on the basis of a data pool of previous decisions”
From Administrative due process when using automated decision-making in public administration: some notes from a Finnish perspective (2021) · cites this paper as Alkhatib and Bernstein 2019
In the citing paper
“more training data does not eliminate errors that lead to incorrect decision, because there will always be new real cases that do not conform to the patterns extracted from training data”
From Administrative due process when using automated decision-making in public administration: some notes from a Finnish perspective (2021) · cites this paper as Alkhatib and Bernstein 2019
In the citing paper
“machine-learning is actually based on predictions, where a new decision is made on the basis of a data pool of previous decisions”
From Administrative due process when using automated decision-making in public administration: some notes from a Finnish perspective (2021) · cites this paper as Alkhatib and Bernstein 2019
In the citing paper
“Alkhatib and Bernstein describe 'street-level algorithms' as systems operating in the gaps between general policy and concrete decisions, where technical classifications may acquire practical authority without being formally recognized as exercises of discretion (Alkhatib and Bernstein, 2019).”
From Human-centric AI governance in the European Union. Accountability, fundamental rights, and institutional resilience in public administration (2026) · cites this paper as Alkhatib and Bernstein, 2019
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567 · cited 2×
“[51] Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra”
BackgroundBureaucracy: what government agencies do and why they do it
(1990) · Choice Reviews Online · doi:10.5860/choice.27-4796 · Cited by 3,795
(2018) · Cited by 3,510
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279 · cited 1×
“This area in particular needs further study: the fault lines are here to stay, and we need to reflect on this shifting of discretion from the bureaucrat to the engineer [18].”
BackgroundThe Color of Law: A Forgotten History of How Our Government Segregated America
Pattillo, M. (2018) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306118792220ll · Cited by 2,247
Street-Level Bureaucracy: The Dilemmas of the Individual in Public Service
Lipsky, M. (1980) · Project Muse (Johns Hopkins University) · Cited by 2,135
Street Corner Society: The Social Structure of an Italian Slum
Meyer, C. H. Z., & Whyte, W. F. (1946) · Journal of Criminal Law and Criminology (1931-1951) · doi:10.2307/1138191 · Cited by 1,838
Çömlekçi, M. F. (2019) · Kırklareli University Institutional Repository (Kırklareli University) · Cited by 1,607
Flexibility Versus Efficiency? A Case Study of Model Changeovers in the Toyota Production System
Adler, P. S., Goldoftas, B., & Levine, D. I. (1999) · Organization Science · doi:10.1287/orsc.10.1.43 · Cited by 1,560
(2018) · Cited by 1,167
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165 · cited 1×
“Nearly every approach to crowd work requires a strategy for quality control, deciding which workers are allowed to continue working on the task, or at worst, which ones are denied payment [7, 50, 60, 61].”
BackgroundThe responsibility gap: Ascribing responsibility for the actions of learning automata
Matthias, A. (2004) · Ethics and Information Technology · doi:10.1007/s10676-004-3422-1 · Cited by 1,101 · cited 1×
“We also avoided a unique quality of machines in this nuanced tension: algorithmic systems operate at far greater speed than humans can [48], precipitating what Mittelstadt et al.”
BackgroundHuman Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Deliberate discretion?: the institutional foundations of bureaucratic autonomy
(2003) · Choice Reviews Online · doi:10.5860/choice.41-0598 · Cited by 904
Women Don't Ask: Negotiation and the Gender Divide
Babcock, L., & Laschever, S. (2003) · Princeton University Press eBooks · doi:10.1515/9780691212845 · Cited by 843
Gillespie, T. (2019) · doi:10.12987/9780300235029 · Cited by 806
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798
(2017) · bepress Legal Repository · Cited by 771
Integration of the Sales Force: An Empirical Examination
Anderson, E., & Schmittlein, D. C. (1984) · The RAND Journal of Economics · doi:10.2307/2555446 · Cited by 631 · cited 3×
“— Anderson and Schmittlein [3], as cited in [2] The challenge is that algorithmic review mechanisms are not well–equipped to understand unusual cases.”
BackgroundEslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
A Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530 · cited 1×
“However, often the most popular answer isn’t actually the correct one [53], and a gold standard answer may not be the only correct answer [36].”
BackgroundGillespie, T. (2018) · Cited by 511
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 3×
“go further to illustrate some of the ways that designers of algorithmic systems can better support street–level bureaucrats given these and other tensions [69].”
BackgroundIntersectional HCI: Engaging Identity through Gender, Race, and Class
Schlesinger, A., Edwards, W. K., & Grinter, R. E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025766 · Cited by 449
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444 · cited 3×
“[16] Sam Corbett-Davies and Sharad Goel.”
BackgroundInfluentialLanguage from police body camera footage shows racial disparities in officer respect
Voigt, R., Camp, N. P., Prabhakaran, V., Hamilton, W. L., Hetey, R. C., Griffiths, C. M., Jurgens, D., Jurafsky, D., & Eberhardt, J. L. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1702413114 · Cited by 407 · cited 1×
“, [70]), it prompts broad critiques and reviews of the policy, organization, or system.”
BackgroundAlgorithms in practice: Comparing web journalism and criminal justice
Christin, A. (2017) · Big Data & Society · doi:10.1177/2053951717718855 · Cited by 379 · cited 2×
“Indeed, Christin explores some of the tensions that emerge when algorithms begin to absorb bureaucrats’ responsibilities and shift the latitude that bureaucrats enjoyed, finding that bureaucrats work around and subvert these systems through foot-dragging, gaming, and open critique as a way of keeping their autonomy [1…”
BackgroundHalfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2013) · American Behavioral Scientist · doi:10.1177/0002764212469365 · Cited by 359
Martin, D., Hanrahan, B. V., O'Neill, J., & Gupta, N. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531663 · Cited by 278 · cited 1×
“fear [46], and the fear of getting work algorithmically rejected looms large [49].”
BackgroundWomen Don't Ask: Negotiation and the Gender Divide
Solnick, S., Babcock, L., & Laschever, S. (2004) · Southern Economic Journal · doi:10.2307/4135303 · Cited by 270
#thyghgapp: Instagram Content Moderation and Lexical Variation in Pro-Eating Disorder Communities
Chancellor, S., Pater, J. A., Clear, T., Gilbert, E., & De Choudhury, M. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2819963 · Cited by 255
The work of sustaining order in wikipedia: the banning of a vandal
Geiger, R. S., & Ribes, D. (2010) · Proceedings of the 2010 ACM conference on Computer supported cooperative work · doi:10.1145/1718918.1718941 · Cited by 245
Βλασσόπουλος, Γ. (2018) · Κοινωνική Πολιτική · doi:10.12681/sp.15989 · Cited by 244
Designing incentives for inexpert human raters
Shaw, A. D., Horton, J. J., & Chen, D. L. (2011) · Proceedings of the ACM 2011 conference on Computer supported cooperative work · doi:10.1145/1958824.1958865 · Cited by 236 · cited 1×
“Nearly every approach to crowd work requires a strategy for quality control, deciding which workers are allowed to continue working on the task, or at worst, which ones are denied payment [7, 50, 60, 61].”
BackgroundThe Crowd is a Collaborative Network
Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2819942 · Cited by 235 · cited 1×
“To manage the concern, workers use back–channels to share information about which requesters are reliable and don’t reject unnecessarily [24, 33].”
BackgroundLe, J., Edmonds, A., Hester, V., & Biewald, L. (2010) · Cited by 232
Out Online: Trans Self-Representation and Community Building on YouTube
Raun, T. (2016) · Routledge eBooks · doi:10.4324/9781315599229 · Cited by 232 · cited 1×
“YouTube was, at one time, a uniquely empowering space for members of the transgender community to be candid and to explore their shared experiences [52, 55], but arguably culture has grown in ways that YouTube and its content classification algorithms have not.”
BackgroundExpert crowdsourcing with flash teams
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T., Valentine, M., & Bernstein, M. S. (2014) · Proceedings of the 27th annual ACM symposium on User interface software and technology · doi:10.1145/2642918.2647409 · Cited by 221 · cited 2×
“[56] Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S.”
BackgroundSelf-citationValue-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Taming the System: The Control of Discretion in Criminal Justice, 1950-1990.
Savelsberg, J. J., & Walker, S. (1994) · Contemporary Sociology · doi:10.2307/2076090 · Cited by 198
Racial Equity in Algorithmic Criminal Justice
Huq, A. Z. (2018) · Duke Law Journal · Cited by 185
Even good bots fight: The case of Wikipedia
Tsvetkova, M., García-Gavilanes, R., Floridi, L., & Yasseri, T. (2017) · PLOS ONE · doi:10.1371/journal.pone.0171774 · Cited by 174 · cited 2×
“[65] Milena Tsvetkova, Ruth GarcÃŋa-Gavilanes, Luciano Floridi, and Taha”
BackgroundTask search in a human computation market
Chilton, L. B., Horton, J. J., Miller, R. C., & Azenkot, S. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837889 · Cited by 171 · cited 1×
“With all of the mistakes these systems make, and with the additional work that crowd workers have to do to make these systems work [13], it should come as little surprise that crowd workers are hesitant to attempt work from unknown and unreliable requesters [49].”
BackgroundUdry, J. R. (1994) · Demography · doi:10.2307/2061790 · Cited by 170 · cited 1×
“Discussions of transgender issues are not necessarily about sex at all; while sex and gender have historically been conflated [67], they refer to different ideas [66]; the state of being trans does not imply any particular sexuality.”
BackgroundLambrecht, A., & Tucker, C. E. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2852260 · Cited by 170 · cited 2×
“In some cases, it seems that problems stem from the data that informs models [11]; in others, recommendation systems are, as AI researchers say, reflecting a mirror back at our own society, itself steeped in racial prejudice [41, 64].”
ResultInstrumenting the crowd: using implicit behavioral measures to predict task performance
Rzeszotarski, J. M., & Kittur, A. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047199 · Cited by 164
Examining Crowd Work and Gig Work Through The Historical Lens of Piecework
Alkhatib, A., Bernstein, M. S., & Levi, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025974 · Cited by 148 · cited 2×
“— Anderson and Schmittlein [3], as cited in [2]”
BackgroundSelf-citationDisempowered by Data: Nonprofits, Social Enterprises, and the Consequences of Data-Driven Work
Bopp, C., Harmon, E., & Voida, A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025694 · Cited by 143
Screen Time Tantrums: How Families Manage Screen Media Experiences for Toddlers and Preschoolers
Hiniker, A., Suh, H., Cao, S., & Kientz, J. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858278 · Cited by 140
Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations
Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025811 · Cited by 123
Self-citationAvoiding the South Side and the Suburbs: The Geography of Mobile Crowdsourcing Markets
Thebault-Spieker, J., Terveen, L. G., & Hecht, B. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675278 · Cited by 121
McInnis, B., Cosley, D., Nam, C., & Leshed, G. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858539 · Cited by 119
Mitra, T., Hutto, C. J., & Gilbert, E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702553 · Cited by 102
(2013) · Routledge eBooks · doi:10.4324/9781315005140 · Cited by 97
Regulatory Structures: Enforcement as Implementation
Hanf, K. I. (1982) · European Journal of Political Research · doi:10.1111/j.1475-6765.1982.tb00015.x · Cited by 84
Valdés, F. (1995) · California Law Review · doi:10.2307/3480882 · Cited by 78 · cited 1×
“Discussions of transgender issues are not necessarily about sex at all; while sex and gender have historically been conflated [67], they refer to different ideas [66]; the state of being trans does not imply any particular sexuality.”
BackgroundHow Data Workers Cope with Uncertainty: A Task Characterisation Study
Boukhelifa, N., Perrin, M.-E., Huron, S., & Eagan, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025738 · Cited by 76
Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks
Kairam, S., & Heer, J. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2820016 · Cited by 72
Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision
Kim, J., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998196 · Cited by 71
Self-citationWhen the levee breaks: without bots, what happens to Wikipedia's quality control processes?
Geiger, R. S., & Halfaker, A. (2013) · Proceedings of the 9th International Symposium on Open Collaboration · doi:10.1145/2491055.2491061 · Cited by 67 · cited 2×
“When we ght with Wikipedia bots [20, 21, 65], street–level algorithms have erred in their enforcement of policies they’ve been taught either by programming or deep learning.”
BackgroundMethodHuman-Computer Interaction and Collective Intelligence
Bigham, J. P., Bernstein, M. S., & Adar, E. (2018) · Figshare · doi:10.1184/r1/6470123 · Cited by 61
Self-citationBeat the machine: Challenging workers to find the unknown unknowns
Attenberg, J., Ipeirotis, P. G., & Provost, F. J. (2011) · Cited by 57
Mechanical Novel: Crowdsourcing Complex Work Through Reflection and Revision
Kim, J. O., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017) · Understanding innovation · doi:10.1007/978-3-319-60967-6_5 · Cited by 56
Self-citationO’Neill, M. G. (2014) · Palgrave Macmillan UK eBooks · doi:10.1057/9781137383556_3 · Cited by 51 · cited 1×
“YouTube was, at one time, a uniquely empowering space for members of the transgender community to be candid and to explore their shared experiences [52, 55], but arguably culture has grown in ways that YouTube and its content classification algorithms have not.”
BackgroundStreet Theatre and Other Outdoor Performance
Mason, B. (1992) · Medical Entomology and Zoology · Cited by 48
Kou, Y., Gui, X., Zhang, S., & Nardi, B. A. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134697 · Cited by 48
Market Structure and Political Law: A Taxonomy of PowerPreprint
Teachout, Z., & Khan, L. M. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2490525 · Cited by 43
Hand book of Collective Intelligence
Wilson, D. R. (2017) · The Social Science Journal · doi:10.1016/j.soscij.2017.10.004 · Cited by 36
Grill, J. (2011) · Anthem Press eBooks · doi:10.7135/upo9780857288684.004 · Cited by 33
The Position and Problems of the Foreman
Whyte, W. F., & Gardner, B. B. (1945) · Human Organization · doi:10.17730/humo.4.2.4431114104856l63 · Cited by 25
Street theatre and other outdoor performance
(1993) · Choice Reviews Online · doi:10.5860/choice.31-1423 · Cited by 22 · cited 1×
“Experimentation is often the point of performance and art, and certainly part of the nature of public performance art like YouTube [15, 47].”
BackgroundThe Breastfeeding Controversy and Facebook: Politicization of Image, Privacy and Protest
Ibrahim, Y. (2010) · International Journal of E-Politics · doi:10.4018/jep.2010040102 · Cited by 17
'Long may the buskers carry on busking': street music and the law in Melbourne and Sydney
Quilter, J., & McNamara, L. (2015) · Research Online (University of Wollongong) · Cited by 13
The State of Video Hosting Websites from the Perspective of Content Creators
Leppänen, R. (2017) · Theseus (Ammattikorkeakoulujen) · Cited by 1
What Professors Don’t Tell You About Social Media
Summers, J. (2018) · Exhibit - A Showcase of Scholarship, Creativity and Preservation Provided by Xavier University Library (Xavier University) · Cited by 1
Research works printed but not linked: 7
- 10.5555/2908698.2908699 · doi:10.5555/2908698.2908699
- 10.5555/3208509 · doi:10.5555/3208509
- B.M. Jewell. 1921. The problem of piece work. Number nos. 1--16 in The Problem of Piece Work. Bronson Canode Print. Co. https: //books.google.com/books?id=NN5NAQAAIAAJ
- Eugene C Kim. 2007. YouTube: Testing the safe harbors of digital copyright law. S. Cal. Interdisc. LJ 17 (2007), 139.
- Michael Lipsky. 1969. Toward a theory of street-level bureaucracy. Institute for Research on Poverty, University of Wisconsin.
- Michael Winerip. 2012. Facing a robo-grader? Just keep obfuscating mellifluously. New York Times 22 (2012).
- Johanna Wright. 2018. Restricted Mode: How it works and what we can do better. https://youtube-creators.googleblog.com/2017/03/ restricted-mode-how-it-works-and-what.html
Cited by
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392
“Notably, although algorithms often make more accurate predictions than people do, their inability to reason re ex-ively and adapt to novel or marginal circumstances makes them poorly suited to achieving many principles of responsible and ethical decision making [2].”
BackgroundInfluentialThe Invisible Cage: Workers’ Reactivity to Opaque Algorithmic Evaluations
Rahman, H. A. (2021) · Administrative Science Quarterly · doi:10.1177/00018392211010118 · Cited by 375
“Second, and relatedly, this study highlights that algorithms lack reflexivity despite the speed and scale at which they operate; they simply carry out their programmed instructions regardless of the situation (Alkhatib and Bernstein, 2019).”
BackgroundOn the apparent conflict between individual and group fairness
Binns, R. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372864 · Cited by 312
“In contexts in which public administration and private decision-making are being automated or augmented with algorithms, the need for occasional individualised consideration is one of the reasons for keeping a human-in-the-loop, or ‘screen-level bureaucrat’ on hand [1, 8, 12, 48].”
BackgroundThe flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263
The Flaws of Policies Requiring Human Oversight of Government Algorithms
Green, B. (2021) · doi:10.2139/ssrn.3921216 · Cited by 259
Metadata
- Authors
- Alkhatib, A., & Bernstein, M.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-13
- OpenAlex
- W2942399136
- Cited by
- 188
- References
- 75
- Citations by year
- 2026: 21 · 2025: 26 · 2024: 24 · 2023: 16 · 2022: 43 · 2021: 30 · 2020: 23 · 2019: 5
- FWCI
- 26.34
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Stanford Cyber Initiative
- National Science Foundation (IIS-1351131)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Keywords
- street-level bureaucracy, algorithmic decision-making, algorithmic bias, algorithmic errors, policy implementation, socio-technical systems, novel situations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.