Procurement As Policy: Administrative Process for Machine Learning
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
At every level of government, officials contract for technical systems that employ machine learning—systems that perform tasks without using explicit instructions, relying on patterns and inference instead. These systems frequently displace discretion previously exercised by policymakers or individual front-end government employees with an opaque logic that bears no resemblance to the reasoning processes of agency personnel. However, because agencies acquire these systems through government procurement processes, they and the public have little input into—or even knowledge about—their design or how well that design aligns with public goals and values. This Article explains the ways that the decisions about goals, values, risk, and certainty, along with the elimination of case-by-case discretion, inherent in machine-learning system design create policies—not just once when they are designed, but over time as they adapt and change. When the adoption of these systems is governed by procurement, the policies they embed receive little or no agency or outside expertise beyond that provided by the vendor. Design decisions are left to private third-party developers. There is no public participation, no reasoned deliberation, and no factual record, which abdicates Government responsibility for policymaking. This Article then argues for a move from a procurement mindset to policymaking mindset. When policy decisions are made through system design, processes suitable for substantive adm…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Private sector firms that develop such models regard this as intellectual property and are reluctant to provide design specifications”
From AI adoption and diffusion in public administration: A systematic literature review and future research agenda (2023) · cites this paper as Mulligan & Bamberger, 2019
Citations
Cites
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627
Using natural frequencies to improve diagnostic inferences
Hoffrage, U., & Gigerenzer, G. (1998) · Academic Medicine · doi:10.1097/00001888-199805000-00024 · Cited by 501
Proactive Policing: Effects on Crime and Communities
Committee on Proactive Policing: Effects on Crime, C., Committee on Law and Justice, Education, D. O. B. A. S. S. A., & National Academies of Sciences, E. (2018) · doi:10.17226/24928 · Cited by 313
Effects of Adaptive Task Allocation on Monitoring of Automated Systems
Parasuraman, R., Mouloua, M., & Molloy, R. (1996) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872096778827279 · Cited by 285
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239
Interacting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184
Intelligent user interfaces: an introduction
Maybury, M. T. (1998) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/291080.291081 · Cited by 124
Distributed Epistemic Responsibility in a Hyperconnected Era
Simon, J. (2015) · The Onlife Manifesto · doi:10.1007/978-3-319-04093-6_17 · Cited by 97
Automation use and automation bias
Mosier, K. L., & Skitka, L. J. (1999) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129904300346 · Cited by 69
Changing organ allocation policy for kidney transplantation in the United States
Chopra, B., & Sureshkumar, K. K. (2015) · World Journal of Transplantation · doi:10.5500/wjt.v5.i2.38 · Cited by 57
Research works printed but not linked: 41
- See supra Section III.B.1 (discussing decision making by machine learning systems)
- See Mulligan, supra note 12 (discussing further the ways that values in technology change over time as technology is appropriated by users in new and unexpected ways, and how technology interactions with business models, organizational structures, and other technologies in ways that can transform its effects
- C See Matthew, 2005, Flexibility, like expertise, is often invoked to justify delegation of substantive policy choices to agencies, L. REV, 93
- E G See, Fox Television Stations, Inc., 556 U.S. 502 (2009) (applying arbitrary and capricious review to a change of agency policy applied in an adjudication)
- Motor Vehicle Mfrs. Assn. v. State Farm Mut., 463 U.S. 29, 30 (1983) (applying arbitrary and capricious review to a change in agency policy reached through rulemaking)
- Clark, 2004, at 463 (discussing how values tussles play out at design, redesign, configuration, and run time). This "developer" perspective on the ability and need to address values at every stage of the process is captured in the security adage, The Trustworthy Computing Security Development Lifecycle, 20TH ANN
- Taxpayer See, 2019, 206. Past: Forum #2: Transparency, NYC AUTOMATED DECISION SYSS. TASK FORCE
- 2010, The Commission was created in 1978 to develop and oversee the development of statewide guidelines to promote fairer and more uniform sentencing. 210. Judicial Code and Prisons and Parole Code, 95 PA, Testimony regarding Update on Local Law 49 of 2018 in Relation to Automated Decision Systems (ADS) Used by Agencies before NYC Council Comm. on Technology, 42
- 2018, proposedrisk-assessment-instrument/additional-information-about-the-proposed-sentence-riskassessment-instrument/research-overview-of-the-sentence-risk-assessment-instrument-1/view; for reports to date, see PA, PA. COMMISSION ON SENT'G, 4
- Stegall, 2014, Why Do We Have the Kidney Allocation System We Have Today? A History of the
- 2017, For details on the allocation formula and discussion of use of a software system-Kidney-Pancreas Simulated Allocation Model (KPSAM)-to assess policy proposals, see Bhavna Chopra & Kalathail K, Kidney Allocation System, HUM. IMMUNOLOGY 78.1, 4, 4
- U S Dep't Health & Hum, 2019, ) (detailing allocation rules and weighting for various organs). 221. The Final Rule requires organ allocation to be based on sound medical judgment, See Organ Procurement and Transplantation Network Policies
- 2019, make the best use of donated organs; avoid wasting organs and futile transplants; and promote patient access to transplants, C.F.R. �, 121
- Jason Accord, 2016, Digital Influencer Jez Humble: DevOps For 'Big Hairy Enterprises', FORBES, 9
- effects of their products[; i]n many cases, negative side effects come as a surprise to technology developers themselves
- Oakland See, 2016, 070 (2018) (establishing requirement for impact assessments but not requiring new staff), 64, 10
- E G See, 2018, What is a chief data ethics officer anyway, IAPP PRIVACY ADVISORY
- 2018, (announcing that JoAnn Stonier was moving from chief information governance and chief privacy officer to chief data officer, Mastercard Names JoAnn Stonier Chief Data Officer, MASTERCARD
- 2019, Rise of The Chief Ethics Officer, FORBES
- 1987, Jasanoff has documented how experts in science policy constantly attempted to demarcate the politics from the scientific in their practice in an effort to preserve claims of objectivity, Contested Boundaries in Policy-Relevant Science, 195
- States See Generally, 2004, presenting a collection of essays exploring the ways in which our methods of understanding and reasoning about the world and choices about how to live in the world are interdependent)
- William Dieterich, 2016, COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity, 9
- See Sam Corbett-Davies, 2016, A Computer Program Used for Bail and Sentencing Decisions Was Labeled Biased Against Blacks. It's Actually Not That Clear, WASH. POST, ARXIV
- E G See, 2014, Big Data: Seizing Opportunities, Preserving Values, EXECUTIVE OFF. PRESIDENT
- 2015, Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights, EXECUTIVE OFF. PRESIDENT
- See Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights
- & Holdren, at 2 & n.2 (citing President Obama's remarks on November 23, 2009 at the launch of his "Educate to Innovate" campaign for excellence in science
- 2014, See Big Data: A Tool for Inclusion Or Exclusion?, FED. TRADE COMMISSION
- Nat'l Sci, 2016, See The National Artificial Intelligence Research and Development Strategic Plan, 8
- See Mulligan, traditional sequential perspective of 'policymaking' (during which there is an opportunity for input) followed by 'implementation' is inconsistent with
- 2014, 113-101, 128 Stat. 1146. 286. See Implementing A Government-wide Law, Digital Accountability and Transparency Act, 18
- See Tad Hirsch, 2017, identifying "contestability" as a new design concern-focused on anticipating and designing for the ways technology can reshape knowledge production and power and describing three lower-level design principles to support contestability: 1) improving accuracy through phased and iterative deployment with expert users in environments that encourage feedback; 2) heightening legibility through mechanisms that "unpack aggregate measures" and "trac[e] system predictions all the way down" so that "users can follow, and if necessary, contest the reasoning behind each prediction"; and relatedly, in an effort to identify and vigilantly prevent system misuse and implicit bias, 3) identifying "aggregate effects" that may imperil vulnerable users through mechanisms that allow, Designing Contestability: Interaction Design, Machine Learning, and Mental Health, 95
- 2019, demanding that companies or governments that base decisions on machine learning must explore and enable alternative ways of datafying and modelling the same event, person or action, Mireille Hildebrandt, Privacy as Protection of the Incomputable Self: From Agnostic to Agonistic Machine Learning, 20 THEORETICAL INQUIRIES L, 83
- Also, this "responds to the need to call out the ethical and political implications of who decides task T, performance metric P and experience E, and to investigate how this is done, taking into account which (and whose) concerns are at stake
- For insights on how contestable systems advance individual understanding, see, for example, Motahhare Eslami, Understanding and Designing Around Users' Interaction with Hidden Algorithms in Sociotechnical Systems, 2017 CSCW '17 COMPANION 57 (describing several studies finding that seamful designs, which expose algorithmic reasoning to users, facilitated understanding, improved user engagement
- Susan Joslyn & Jared Leclerc, 2013, describing how displaying uncertainty in weather predictions can lead to more optimal decision making and trust in a forecast: transparency about probabilistic nature of prediction engenders, Decisions with Uncertainty: The Glass Half Full, 308
- Proc, ON INTELLIGENT USER INTERFACES, 82
- Saleema Amershi, 2014, providing an overview of interactive machine learning research with case studies, and discussing value of interactive machine learning approaches for the machine learning community as well as users)
- A Anrey, 2015, How to Discriminate Between Computer-Aided and ComputerHindered Decisions: A Case Study in Mammography, 33 MED. DECISION MAKING 98 (2013); see Goddard, supra note 182; see also Adrian Bussone et al., The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems, INT'L CONF, ON HEALTHCARE INFORMATICS, 160
- N Daniel, 2019, Automated Decision Support Technologies and the Legal Profession, 34 BERKELEY TECH, L.J
- 2013, missed the bus and can just blame the app"). 303. Communicating the uncertainty related with the probabilistic nature of machine learning systems to users improves decision making, and if it is coupled with feedback mechanism, it can leverage human knowledge to increase accuracy of model over time. See Rocio Garcia-Retamero & Edward T. Cokely, Communicating Health Risks with Visual Aids, 22 CURRENT DIRECTIONS, PSYCHOL. SCI, 392
Cited by
Madan, R., & Ashok, M. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2022.101774 · Cited by 468
“…transparency (Chen et al., 2019; Harrison & Luna-Reyes, 2022; Henman, 2019; Janssen, Brous, et al., 2020; Janssen et al., 2022; Makasi et al., 2021; Mulligan & Bamberger, 2019; Ojo et al., 2019; Sousa et al., 2019; Veale et al., 2018; Veale & Brass, 2019; Wirtz et al., 2018; Young et al., 2019;…”
Ethical framework for Artificial Intelligence and Digital technologies
Ashok, M., Madan, R., Joha, A., & Sivarajah, U. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102433 · Cited by 469
van Noordt, C., & Misuraca, G. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2022.101714 · Cited by 356
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
“As a client making decisions on which projects to select, or permit for purchase, it can be good to set performance related requirements for procurement and leverage this procurement process in order to set expectations for functionality [127, 156, 165, 172].”
BackgroundAlgorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445935 · Cited by 195
“He notes, along with others (see [84]) that the vagaries of software procurement means that government agencies may deploy algorithmic systems without actually understanding what they do, which can”
Metadata
- Authors
- Mulligan, D. K., & Bamberger, K. A.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Bamberger, Kenneth A.
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2019
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W6924855210
- Cited by
- 75
- References
- 23
- Citations by year
- 2026: 4 · 2025: 10 · 2024: 12 · 2023: 21 · 2022: 14 · 2021: 13 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- UC Berkeley
- Topics
- Social Sciences › Business, Management and Accounting › Strategy and Management › Public Procurement and Policy
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- machine learning, procurement, administrative processes, public procurement, algorithmic decision-making, algorithmic transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.