CoChat

The Flaws of Policies Requiring Human Oversight of Government Algorithms

Open access
Green, Ben · 2021 · Computer law & security review

The DOI matches its Crossref record; 105 of the 110 references it cites were found in a registry or an index.

Abstract

As algorithms become an influential component of government decision-making around the world, policymakers have debated how governments can attain the benefits of algorithms while preventing the harms of algorithms. One mechanism that has become a centerpiece of global efforts to regulate government algorithms is to require human oversight of algorithmic decisions. Despite the widespread turn to human oversight, these policies rest on an uninterrogated assumption: that people are able to effectively oversee algorithmic decision-making. In this article, I survey 41 policies that prescribe human oversight of government algorithms and find that they suffer from two significant flaws. First, evidence suggests that people are unable to perform the desired oversight functions. Second, as a result of the first flaw, human oversight policies legitimize government uses of faulty and controversial algorithms without addressing the fundamental issues with these tools. Thus, rather than protect against the potential harms of algorithmic decision-making in government, human oversight policies provide a false sense of security in adopting algorithms and enable vendors and agencies to shirk accountability for algorithmic harms. In light of these flaws, I propose a shift from human oversight to institutional oversight as the central mechanism for regulating government algorithms. This institutional approach operates in two stages. First, agencies must justify that it is appropriate to incorporate an algorithm into decision-making and that any proposed forms of human oversight are supported by empirical evidence. Second, these justifications must receive democratic public review and approval before the agency can adopt the algorithm.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2139/ssrn.3921216

Cites

105 of 105 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 105 references

Research works printed but not linked: 37

  1. Albright A. If you give a judge a risk score: Evidence from Kentucky bail decisions; 2019 https: //thelittledataset.com/about _ files/albright _ judge _ score.pdf.
  2. Belgian Federal Parliament. Loi Relative à la Protection des Personnes Physiques à L'égard des Traitements de Données à Caractère Personnel (Law on the Protection of Natural Persons with regard to the Processing of Personal Data); 2018 https:// www.ejustice.just.fgov.be/eli/loi/2018/07/30/2018040581/justel .
  3. Bundestag, 2019, Federal Data Protection Act (BDSG)
  4. California Legislature, 2021, AB-13 Public Contracts: Automated Decision Systems
  5. Campbell ET, editor. Detroiters Want to be Seen, not Watched. Riverwise; 2019 https://detroitcommunitytech.org/system/tdf/ librarypdfs/2019-2206 _ Riverwise-Surveillance.pdf?file=1 . CBS News. Police Departments Adopting Facial Recognition Tech Amid Allegations of Wrongful Arrests; 2021 60 Minutes https://www.cbsnews.com/news/ facial-recognition-60-minutes-2021-05-16/ .
  6. Charette RN. Michigan's MiDAS unemployment system: Algorithm alchemy created lead, not gold. IEEE Spectrum 2018 https://spectrum.ieee.org/riskfactor/computing/software/ michigans-midas-unemployment-system-algorithm- alchemy-that-created-lead-not-gold .
  7. Cambridge City, 2018, Surveillance technology ordinance
  8. Cummings ML. Automation and accountability in decision support system interface design. J. Technol. Stud. 2006;32(1):23-31 https://scholar.lib.vt.edu/ejournals/JOTS/v32/ v32n1/pdf/cummings.pdf.
  9. 2021, Proposal for a Regulation Laying Down Harmonised Rules on Artificial Intelligence, Artificial Intelligence Act
  10. 2021, Civil Society Calls for AI red lines in the European Union's Artificial Intelligence Proposal
  11. 2021, EU's New Artificial Intelligence Law Risks Enabling Orwellian Surveillance States
  12. National Congress of Brazil. General Data Protection Law; 2019 https://iapp.org/media/pdf/resource _ center/ Brazilian _ General _ Data _ Protection _ Law.pdf. New Jersey Courts. One Year Criminal Justice Reform Report to the Governor and the Legislature; 2017 https: //www.njcourts.gov/courts/assets/criminal/2017cjrannual.pdf.
  13. 2021, Ban Facial Recognition
  14. French Parliament, 2018, French Data Protection Act
  15. K Hill, 2018, How one state managed to actually write rules on facial recognition, Houses of the Oireachtas
  16. Houses of the Oireachtas. (2018). Data Protection Act 2018. https://data.oireachtas.ie/ie/oireachtas/act/2018/7/eng/ enacted/a0718.pdf Human Rights Watch. "Not in it for Justice": How California's Pretrial Detention and Bail System Unfairly Punishes Poor People; 2017 https://www.hrw.org/report/2017/04/11/not-it-justice/ how-californias-pretrial-detention-and-bail-system-unfairly . Hungarian Parliament. Data Protection Act; 2018 https://www.parlament.hu/irom41/00623/00623.pdf.
  17. Ibarra N. Santa Cruz, Calif., Bans predictive policing technology. Govern. Technol. 2020 https://www.govtech.com/public-safety/ santa-cruz-calif-bans-predictive-policing-technology.html . Indiana Supreme Court. Malenchik v. State; 2010 928
  18. Kelly E. EU lawmaker vows to tighten use of biometric tracking tech in Europe. Sifted 2021 https://sifted.eu/articles/eu-facial-recognition-tech/ .
  19. Kingdom of Bahrain. Personal Data Protection Law; 2018 https://bahrainbusinesslaws.com/laws/ Personal-Data-Protection-Law .
  20. Koebler J. Detroit police chief: facial recognition software misidentifies 96% of the time. Vice 2020 https://www.vice.com/en/article/dyzykz/ detroit-police-chief-facial-recognition-software-misidentifies -96-of-the-time .
  21. Metzinger T. Ethics washing made in Europe. Der Tagesspiegel 2019 https://www.tagesspiegel.de/politik/ eu-guidelines-ethics-washing-made-in-europe/24195496. html .
  22. L Miller, 2020, LAPD will end controversial program that aimed to predict where crimes would occur, Los Angeles Times
  23. 2017, An Act to Modernize Legislative Provisions as Regards the Protection of Personal Information, Bill, 64
  24. Northpointe, Inc.. Practitioner's Guide to COMPAS Core; 2015 http://www.northpointeinc.com/downloads/compas/ Practitioners-Guide-COMPAS-Core-_ 031915.pdf. Office of the Privacy Commissioner of Canada. A Regulatory Framework for AI: Recommendations for PIPEDA Reform; 2020 https://www.priv.gc.ca/en/about-the-opc/what-we-do/ consultations/completed-consultations/consultation-ai/ reg-fw _ 202011/ .
  25. Pasquale F . New Laws of Robotics: Defending Human Expertise in the Age of AI. Belknap Press; 2020 .
  26. Argentina Republic, 2013, Ley De Protecci�n De Los Datos Personales (Personal Data Protection Law)
  27. Richardson R, editor. Confronting Black Boxes: A Shadow Report of the New York City Automated Decision System Task Force. AI Now Institute; 2019 https://ainowinstitute.org/ads-shadowreport-2019.pdf.
  28. Richardson R, Schultz JM, Crawford K. Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. New York University Law Rev. 2019;94:15-55 https://www.nyulawreview.org/online-features/ dirty-data-bad-predictions-how-civil-rights-violations-impact- police-data-predictive-policing-systems-and-justice/ .
  29. Scharfenberg D. Computers can solve your problem. You may not like the answer. Boston Globe 2018 https://apps.bostonglobe. com/ideas/graphics/2018/09/equity-machine/ .
  30. J Skeem, 2016, Impact of risk assessment on judges' fairness in sentencing relatively poor defendants, Central Bond Court Report, 44, 51
  31. Slovenian Parliament. Zakon o varstvu Osebnih Podatkov na Podro čju Obravnavanja Kaznivih Dejanj (Personal Data Protection Act in the Field of Dealing with Criminal Offenses); 2020 http://www.pisrs.si/Pis.web/pregledPredpisa?id=ZAKO8157 .
  32. Solum L. Legal theory lexicon: Rules, standards, and principles. Legal Theory Blog 2009 https://lsolum.typepad.com/legaltheory/2009/09/ legal-theory-lexicon-rules-standards-and-principles.html .
  33. 2018, Statistics New Zealand. Algorithm Assessment Report
  34. Stein MI. New Orleans City Council bans facial recognition, predictive policing and other surveillance tech. Lens 2020 https://thelensnola.org/2020/12/18/new-orleans-city- council-approves-ban-on-facial-recognition-predictive- policing-and-other-surveillance-tech/ .
  35. Steinhart D. Juvenile detention risk assessment: A practice guide for juvenile detention reform. Annie E. Casey Found. 2006 https://www.aecf.org/m/resourceimg/ aecf-juveniledetentionriskassessment1-2006.pdf.
  36. Stop LAPD Spying Coalition. Before the Bullet Hits the Body: Dismantling Predictive Policing in Los Angeles; 2018 https://stoplapdspying.org/wp-content/uploads/2018/05/ Before-the-Bullet-Hits-the-Body-May-8-2018.pdf.
  37. Author Information Ben Green is a Postdoctoral Scholar in the Michigan Society of Fellows and an Assistant Professor in the Gerald R. Ford School of Public Policy. He holds a PhD in Applied Mathematics, with a secondary field in Science, Technology, and Society, from Har- vard University. His book, The Smart Enough City: Putting Tech- nology in Its Place to Reclaim Our Urban Future , was published in 2019 by MIT Press. Ben is also an Affiliate at the Berkman Klein Center for Internet & Society at Harvard and a Fellow at the Center for Democracy & Technology.

Cited by

Items 101–200 of 427

Metadata

Authors
Green, B.
Affiliations
University of Michigan (US)
Corresponding author
Green, Ben
Venue
Computer law & security review
Publisher
Elsevier BV
Published
2021
Volume
45
Pages
105681-105681
ISSN
2212-473X (ISSN-L)
OpenAlex
W3200421165
Cited by
259
References
110
Citations by year
2026: 108 · 2025: 56 · 2024: 43 · 2023: 28 · 2022: 24
FWCI
43.80
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Computer law & security review (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
  • Social Sciences › Political Science and International Relations › E-Government and Public Services
Keywords
human oversight, algorithmic decision-making, algorithmic regulation, institutional oversight, accountability, algorithmic harms, public trust in government, algorithmic governance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.