CoChat

Procurement As Policy: Administrative Process for Machine Learning

Open access
Mulligan, Deirdre K., Bamberger, Kenneth A. · 2019 · SSRN Electronic Journal

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.

Abstract

At every level of government, officials contract for technical systems that employ machine learning—systems that perform tasks without using explicit instructions, relying on patterns and inference instead. These systems frequently displace discretion previously exercised by policymakers or individual front-end government employees with an opaque logic that bears no resemblance to the reasoning processes of agency personnel. However, because agencies acquire these systems through government procurement processes, they and the public have little input into—or even knowledge about—their design or how well that design aligns with public goals and values. This Article explains the ways that the decisions about goals, values, risk, and certainty, along with the elimination of case-by-case discretion, inherent in machine-learning system design create policies—not just once when they are designed, but over time as they adapt and change. When the adoption of these systems is governed by procurement, the policies they embed receive little or no agency or outside expertise beyond that provided by the vendor. Design decisions are left to private third-party developers. There is no public participation, no reasoned deliberation, and no factual record, which abdicates Government responsibility for policymaking. This Article then argues for a move from a procurement mindset to policymaking mindset. When policy decisions are made through system design, processes suitable for substantive adm…

Overview

Integrity signals

Registry

Crossref · 10.2139/ssrn.3464203

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 24 references

Research works printed but not linked: 41

  1. See supra Section III.B.1 (discussing decision making by machine learning systems)
  2. See Mulligan, supra note 12 (discussing further the ways that values in technology change over time as technology is appropriated by users in new and unexpected ways, and how technology interactions with business models, organizational structures, and other technologies in ways that can transform its effects
  3. C See Matthew, 2005, Flexibility, like expertise, is often invoked to justify delegation of substantive policy choices to agencies, L. REV, 93
  4. E G See, Fox Television Stations, Inc., 556 U.S. 502 (2009) (applying arbitrary and capricious review to a change of agency policy applied in an adjudication)
  5. Motor Vehicle Mfrs. Assn. v. State Farm Mut., 463 U.S. 29, 30 (1983) (applying arbitrary and capricious review to a change in agency policy reached through rulemaking)
  6. Clark, 2004, at 463 (discussing how values tussles play out at design, redesign, configuration, and run time). This "developer" perspective on the ability and need to address values at every stage of the process is captured in the security adage, The Trustworthy Computing Security Development Lifecycle, 20TH ANN
  7. Taxpayer See, 2019, 206. Past: Forum #2: Transparency, NYC AUTOMATED DECISION SYSS. TASK FORCE
  8. 2010, The Commission was created in 1978 to develop and oversee the development of statewide guidelines to promote fairer and more uniform sentencing. 210. Judicial Code and Prisons and Parole Code, 95 PA, Testimony regarding Update on Local Law 49 of 2018 in Relation to Automated Decision Systems (ADS) Used by Agencies before NYC Council Comm. on Technology, 42
  9. 2018, proposedrisk-assessment-instrument/additional-information-about-the-proposed-sentence-riskassessment-instrument/research-overview-of-the-sentence-risk-assessment-instrument-1/view; for reports to date, see PA, PA. COMMISSION ON SENT'G, 4
  10. Stegall, 2014, Why Do We Have the Kidney Allocation System We Have Today? A History of the
  11. 2017, For details on the allocation formula and discussion of use of a software system-Kidney-Pancreas Simulated Allocation Model (KPSAM)-to assess policy proposals, see Bhavna Chopra & Kalathail K, Kidney Allocation System, HUM. IMMUNOLOGY 78.1, 4, 4
  12. U S Dep't Health & Hum, 2019, ) (detailing allocation rules and weighting for various organs). 221. The Final Rule requires organ allocation to be based on sound medical judgment, See Organ Procurement and Transplantation Network Policies
  13. 2019, make the best use of donated organs; avoid wasting organs and futile transplants; and promote patient access to transplants, C.F.R. �, 121
  14. Jason Accord, 2016, Digital Influencer Jez Humble: DevOps For 'Big Hairy Enterprises', FORBES, 9
  15. effects of their products[; i]n many cases, negative side effects come as a surprise to technology developers themselves
  16. Oakland See, 2016, 070 (2018) (establishing requirement for impact assessments but not requiring new staff), 64, 10
  17. E G See, 2018, What is a chief data ethics officer anyway, IAPP PRIVACY ADVISORY
  18. 2018, (announcing that JoAnn Stonier was moving from chief information governance and chief privacy officer to chief data officer, Mastercard Names JoAnn Stonier Chief Data Officer, MASTERCARD
  19. 2019, Rise of The Chief Ethics Officer, FORBES
  20. 1987, Jasanoff has documented how experts in science policy constantly attempted to demarcate the politics from the scientific in their practice in an effort to preserve claims of objectivity, Contested Boundaries in Policy-Relevant Science, 195
  21. States See Generally, 2004, presenting a collection of essays exploring the ways in which our methods of understanding and reasoning about the world and choices about how to live in the world are interdependent)
  22. William Dieterich, 2016, COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity, 9
  23. See Sam Corbett-Davies, 2016, A Computer Program Used for Bail and Sentencing Decisions Was Labeled Biased Against Blacks. It's Actually Not That Clear, WASH. POST, ARXIV
  24. E G See, 2014, Big Data: Seizing Opportunities, Preserving Values, EXECUTIVE OFF. PRESIDENT
  25. 2015, Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights, EXECUTIVE OFF. PRESIDENT
  26. See Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights
  27. & Holdren, at 2 & n.2 (citing President Obama's remarks on November 23, 2009 at the launch of his "Educate to Innovate" campaign for excellence in science
  28. 2014, See Big Data: A Tool for Inclusion Or Exclusion?, FED. TRADE COMMISSION
  29. Nat'l Sci, 2016, See The National Artificial Intelligence Research and Development Strategic Plan, 8
  30. See Mulligan, traditional sequential perspective of 'policymaking' (during which there is an opportunity for input) followed by 'implementation' is inconsistent with
  31. 2014, 113-101, 128 Stat. 1146. 286. See Implementing A Government-wide Law, Digital Accountability and Transparency Act, 18
  32. See Tad Hirsch, 2017, identifying "contestability" as a new design concern-focused on anticipating and designing for the ways technology can reshape knowledge production and power and describing three lower-level design principles to support contestability: 1) improving accuracy through phased and iterative deployment with expert users in environments that encourage feedback; 2) heightening legibility through mechanisms that "unpack aggregate measures" and "trac[e] system predictions all the way down" so that "users can follow, and if necessary, contest the reasoning behind each prediction"; and relatedly, in an effort to identify and vigilantly prevent system misuse and implicit bias, 3) identifying "aggregate effects" that may imperil vulnerable users through mechanisms that allow, Designing Contestability: Interaction Design, Machine Learning, and Mental Health, 95
  33. 2019, demanding that companies or governments that base decisions on machine learning must explore and enable alternative ways of datafying and modelling the same event, person or action, Mireille Hildebrandt, Privacy as Protection of the Incomputable Self: From Agnostic to Agonistic Machine Learning, 20 THEORETICAL INQUIRIES L, 83
  34. Also, this "responds to the need to call out the ethical and political implications of who decides task T, performance metric P and experience E, and to investigate how this is done, taking into account which (and whose) concerns are at stake
  35. For insights on how contestable systems advance individual understanding, see, for example, Motahhare Eslami, Understanding and Designing Around Users' Interaction with Hidden Algorithms in Sociotechnical Systems, 2017 CSCW '17 COMPANION 57 (describing several studies finding that seamful designs, which expose algorithmic reasoning to users, facilitated understanding, improved user engagement
  36. Susan Joslyn & Jared Leclerc, 2013, describing how displaying uncertainty in weather predictions can lead to more optimal decision making and trust in a forecast: transparency about probabilistic nature of prediction engenders, Decisions with Uncertainty: The Glass Half Full, 308
  37. Proc, ON INTELLIGENT USER INTERFACES, 82
  38. Saleema Amershi, 2014, providing an overview of interactive machine learning research with case studies, and discussing value of interactive machine learning approaches for the machine learning community as well as users)
  39. A Anrey, 2015, How to Discriminate Between Computer-Aided and ComputerHindered Decisions: A Case Study in Mammography, 33 MED. DECISION MAKING 98 (2013); see Goddard, supra note 182; see also Adrian Bussone et al., The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems, INT'L CONF, ON HEALTHCARE INFORMATICS, 160
  40. N Daniel, 2019, Automated Decision Support Technologies and the Legal Profession, 34 BERKELEY TECH, L.J
  41. 2013, missed the bus and can just blame the app"). 303. Communicating the uncertainty related with the probabilistic nature of machine learning systems to users improves decision making, and if it is coupled with feedback mechanism, it can leverage human knowledge to increase accuracy of model over time. See Rocio Garcia-Retamero & Edward T. Cokely, Communicating Health Risks with Visual Aids, 22 CURRENT DIRECTIONS, PSYCHOL. SCI, 392

Cited by

See all 78

Metadata

Authors
Mulligan, D. K., & Bamberger, K. A.
Affiliations
University of California, Berkeley (US)
Corresponding author
Bamberger, Kenneth A.
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2019
ISSN
1556-5068 (ISSN-L)
OpenAlex
W6924855210
Cited by
75
References
23
Citations by year
2026: 4 · 2025: 10 · 2024: 12 · 2023: 21 · 2022: 14 · 2021: 13 · 2020: 1
Type
posted content
Language
en
Open access
Locations
  • SSRN Electronic Journal (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • UC Berkeley
Topics
  • Social Sciences › Business, Management and Accounting › Strategy and Management › Public Procurement and Policy
  • Social Sciences › Political Science and International Relations › E-Government and Public Services
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
machine learning, procurement, administrative processes, public procurement, algorithmic decision-making, algorithmic transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.