Procurement As Policy: Administrative Process for Machine Learning
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
At every level of government, officials contract for technical systems that employ machine learning—systems that perform tasks without using explicit instructions, relying on patterns and inference instead. These systems frequently displace discretion previously exercised by policymakers or individual front-end government employees with an opaque logic that bears no resemblance to the reasoning processes of agency personnel. However, because agencies acquire these systems through government procurement processes, they and the public have little input into—or even knowledge about—their design or how well that design aligns with public goals and values. This Article explains the ways that the decisions about goals, values, risk, and certainty, along with the elimination of case-by-case discretion, inherent in machine-learning system design create policies—not just once when they are designed, but over time as they adapt and change. When the adoption of these systems is governed by procurement, the policies they embed receive little or no agency or outside expertise beyond that provided by the vendor. Design decisions are left to private third-party developers. There is no public participation, no reasoned deliberation, and no factual record, which abdicates Government responsibility for policymaking. This Article then argues for a move from a procurement mindset to policymaking mindset. When policy decisions are made through system design, processes suitable for substantive adm…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Private sector firms that develop such models regard this as intellectual property and are reluctant to provide design specifications”
From AI adoption and diffusion in public administration: A systematic literature review and future research agenda (2023) · cites this paper as Mulligan & Bamberger, 2019
Citations
Cites
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627
Using natural frequencies to improve diagnostic inferences
Hoffrage, U., & Gigerenzer, G. (1998) · Academic Medicine · doi:10.1097/00001888-199805000-00024 · Cited by 501
Proactive Policing: Effects on Crime and Communities
Committee on Proactive Policing: Effects on Crime, C., Committee on Law and Justice, Education, D. O. B. A. S. S. A., & National Academies of Sciences, E. (2018) · doi:10.17226/24928 · Cited by 313
Effects of Adaptive Task Allocation on Monitoring of Automated Systems
Parasuraman, R., Mouloua, M., & Molloy, R. (1996) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872096778827279 · Cited by 285
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239
Interacting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184
Intelligent user interfaces: an introduction
Maybury, M. T. (1998) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/291080.291081 · Cited by 124
Distributed Epistemic Responsibility in a Hyperconnected Era
Simon, J. (2015) · The Onlife Manifesto · doi:10.1007/978-3-319-04093-6_17 · Cited by 97
Automation use and automation bias
Mosier, K. L., & Skitka, L. J. (1999) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129904300346 · Cited by 69
Changing organ allocation policy for kidney transplantation in the United States
Chopra, B., & Sureshkumar, K. K. (2015) · World Journal of Transplantation · doi:10.5500/wjt.v5.i2.38 · Cited by 57
Research works printed but not linked: 41
- See supra Section III.B.1 (discussing decision making by machine learning systems)
- See Mulligan, supra note 12 (discussing further the ways that values in technology change over time as technology is appropriated by users in new and unexpected ways, and how technology interactions with business models, organizational structures, and other technologies in ways that can transform its effects
- C See Matthew, 2005, Flexibility, like expertise, is often invoked to justify delegation of substantive policy choices to agencies, L. REV, 93
- E G See, Fox Television Stations, Inc., 556 U.S. 502 (2009) (applying arbitrary and capricious review to a change of agency policy applied in an adjudication)
- Motor Vehicle Mfrs. Assn. v. State Farm Mut., 463 U.S. 29, 30 (1983) (applying arbitrary and capricious review to a change in agency policy reached through rulemaking)
- Clark, 2004, at 463 (discussing how values tussles play out at design, redesign, configuration, and run time). This "developer" perspective on the ability and need to address values at every stage of the process is captured in the security adage, The Trustworthy Computing Security Development Lifecycle, 20TH ANN
- Taxpayer See, 2019, 206. Past: Forum #2: Transparency, NYC AUTOMATED DECISION SYSS. TASK FORCE
- 2010, The Commission was created in 1978 to develop and oversee the development of statewide guidelines to promote fairer and more uniform sentencing. 210. Judicial Code and Prisons and Parole Code, 95 PA, Testimony regarding Update on Local Law 49 of 2018 in Relation to Automated Decision Systems (ADS) Used by Agencies before NYC Council Comm. on Technology, 42
- 2018, proposedrisk-assessment-instrument/additional-information-about-the-proposed-sentence-riskassessment-instrument/research-overview-of-the-sentence-risk-assessment-instrument-1/view; for reports to date, see PA, PA. COMMISSION ON SENT'G, 4
- Stegall, 2014, Why Do We Have the Kidney Allocation System We Have Today? A History of the
- 2017, For details on the allocation formula and discussion of use of a software system-Kidney-Pancreas Simulated Allocation Model (KPSAM)-to assess policy proposals, see Bhavna Chopra & Kalathail K, Kidney Allocation System, HUM. IMMUNOLOGY 78.1, 4, 4
- U S Dep't Health & Hum, 2019, ) (detailing allocation rules and weighting for various organs). 221. The Final Rule requires organ allocation to be based on sound medical judgment, See Organ Procurement and Transplantation Network Policies
- 2019, make the best use of donated organs; avoid wasting organs and futile transplants; and promote patient access to transplants, C.F.R. �, 121
- Jason Accord, 2016, Digital Influencer Jez Humble: DevOps For 'Big Hairy Enterprises', FORBES, 9
- effects of their products[; i]n many cases, negative side effects come as a surprise to technology developers themselves
- Oakland See, 2016, 070 (2018) (establishing requirement for impact assessments but not requiring new staff), 64, 10
- E G See, 2018, What is a chief data ethics officer anyway, IAPP PRIVACY ADVISORY
- 2018, (announcing that JoAnn Stonier was moving from chief information governance and chief privacy officer to chief data officer, Mastercard Names JoAnn Stonier Chief Data Officer, MASTERCARD
- 2019, Rise of The Chief Ethics Officer, FORBES
- 1987, Jasanoff has documented how experts in science policy constantly attempted to demarcate the politics from the scientific in their practice in an effort to preserve claims of objectivity, Contested Boundaries in Policy-Relevant Science, 195
- States See Generally, 2004, presenting a collection of essays exploring the ways in which our methods of understanding and reasoning about the world and choices about how to live in the world are interdependent)
- William Dieterich, 2016, COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity, 9
- See Sam Corbett-Davies, 2016, A Computer Program Used for Bail and Sentencing Decisions Was Labeled Biased Against Blacks. It's Actually Not That Clear, WASH. POST, ARXIV
- E G See, 2014, Big Data: Seizing Opportunities, Preserving Values, EXECUTIVE OFF. PRESIDENT
- 2015, Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights, EXECUTIVE OFF. PRESIDENT
- See Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights
- & Holdren, at 2 & n.2 (citing President Obama's remarks on November 23, 2009 at the launch of his "Educate to Innovate" campaign for excellence in science
- 2014, See Big Data: A Tool for Inclusion Or Exclusion?, FED. TRADE COMMISSION
- Nat'l Sci, 2016, See The National Artificial Intelligence Research and Development Strategic Plan, 8
- See Mulligan, traditional sequential perspective of 'policymaking' (during which there is an opportunity for input) followed by 'implementation' is inconsistent with
- 2014, 113-101, 128 Stat. 1146. 286. See Implementing A Government-wide Law, Digital Accountability and Transparency Act, 18
- See Tad Hirsch, 2017, identifying "contestability" as a new design concern-focused on anticipating and designing for the ways technology can reshape knowledge production and power and describing three lower-level design principles to support contestability: 1) improving accuracy through phased and iterative deployment with expert users in environments that encourage feedback; 2) heightening legibility through mechanisms that "unpack aggregate measures" and "trac[e] system predictions all the way down" so that "users can follow, and if necessary, contest the reasoning behind each prediction"; and relatedly, in an effort to identify and vigilantly prevent system misuse and implicit bias, 3) identifying "aggregate effects" that may imperil vulnerable users through mechanisms that allow, Designing Contestability: Interaction Design, Machine Learning, and Mental Health, 95
- 2019, demanding that companies or governments that base decisions on machine learning must explore and enable alternative ways of datafying and modelling the same event, person or action, Mireille Hildebrandt, Privacy as Protection of the Incomputable Self: From Agnostic to Agonistic Machine Learning, 20 THEORETICAL INQUIRIES L, 83
- Also, this "responds to the need to call out the ethical and political implications of who decides task T, performance metric P and experience E, and to investigate how this is done, taking into account which (and whose) concerns are at stake
- For insights on how contestable systems advance individual understanding, see, for example, Motahhare Eslami, Understanding and Designing Around Users' Interaction with Hidden Algorithms in Sociotechnical Systems, 2017 CSCW '17 COMPANION 57 (describing several studies finding that seamful designs, which expose algorithmic reasoning to users, facilitated understanding, improved user engagement
- Susan Joslyn & Jared Leclerc, 2013, describing how displaying uncertainty in weather predictions can lead to more optimal decision making and trust in a forecast: transparency about probabilistic nature of prediction engenders, Decisions with Uncertainty: The Glass Half Full, 308
- Proc, ON INTELLIGENT USER INTERFACES, 82
- Saleema Amershi, 2014, providing an overview of interactive machine learning research with case studies, and discussing value of interactive machine learning approaches for the machine learning community as well as users)
- A Anrey, 2015, How to Discriminate Between Computer-Aided and ComputerHindered Decisions: A Case Study in Mammography, 33 MED. DECISION MAKING 98 (2013); see Goddard, supra note 182; see also Adrian Bussone et al., The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems, INT'L CONF, ON HEALTHCARE INFORMATICS, 160
- N Daniel, 2019, Automated Decision Support Technologies and the Legal Profession, 34 BERKELEY TECH, L.J
- 2013, missed the bus and can just blame the app"). 303. Communicating the uncertainty related with the probabilistic nature of machine learning systems to users improves decision making, and if it is coupled with feedback mechanism, it can leverage human knowledge to increase accuracy of model over time. See Rocio Garcia-Retamero & Edward T. Cokely, Communicating Health Risks with Visual Aids, 22 CURRENT DIRECTIONS, PSYCHOL. SCI, 392
Cited by
Madan, R., & Ashok, M. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2022.101774 · Cited by 468
“…transparency (Chen et al., 2019; Harrison & Luna-Reyes, 2022; Henman, 2019; Janssen, Brous, et al., 2020; Janssen et al., 2022; Makasi et al., 2021; Mulligan & Bamberger, 2019; Ojo et al., 2019; Sousa et al., 2019; Veale et al., 2018; Veale & Brass, 2019; Wirtz et al., 2018; Young et al., 2019;…”
Ethical framework for Artificial Intelligence and Digital technologies
Ashok, M., Madan, R., Joha, A., & Sivarajah, U. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102433 · Cited by 469
van Noordt, C., & Misuraca, G. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2022.101714 · Cited by 356
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
“As a client making decisions on which projects to select, or permit for purchase, it can be good to set performance related requirements for procurement and leverage this procurement process in order to set expectations for functionality [127, 156, 165, 172].”
BackgroundAlgorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445935 · Cited by 195
“He notes, along with others (see [84]) that the vagaries of software procurement means that government agencies may deploy algorithmic systems without actually understanding what they do, which can”
Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150
Cooper, A. F., Moss, E., Laufer, B., & Nissenbaum, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533150 · Cited by 129
“, machine learning (ML) and artificial intelligence (AI)—which either have replaced or complemented rule-based software systems, or have been incorporated within them as essential elements [12, 26, 75, 92, 141].”
Backgroundvan Noordt, C., Medaglia, R., & Tangi, L. (2023) · Public Policy and Administration · doi:10.1177/09520767231198411 · Cited by 109
“…challenges to procurement, such as the legal use of data (Harrison and Luna-Reyes, 2022), ownership (Campion et al., 2022), opacity and secrecy (Mulligan and Bamberger, 2019) can bear serious risks if public administrations themselves cannot manage such partnerships successfully (Tangi et al.,…”
Wang, A., Kapoor, S., Barocas, S., & Narayanan, A. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3636509 · Cited by 57
“For example, standardized tests such as the SAT and the GRE are administered by private companies but those companies are monopolies—there is usually no alternative to these tests for university applications.”
BackgroundFrom Public E-Procurement 3.0 to E-Procurement 4.0; A Critical Literature Review
Mavidis, A., & Folinas, D. (2022) · Sustainability · doi:10.3390/su141811252 · Cited by 45
The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool
Gerchick, M., Jegede, T., Shah, T., Gutierrez, A., Beiers, S., Shemtov, N., Xu, K., Samant, A., & Horowitz, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594081 · Cited by 34
“Mulligan and Bamberger [70] argue that, under the current paradigm of government agencies procuring and using algorithmic tools developed by third-party entities, government decision-makers abdicate im-”
BackgroundInfluentialAI governance in India – law, policy and political economy
Joshi, D. (2024) · Communication Research and Practice · doi:10.1080/22041451.2024.2346428 · Cited by 50
“…in the way in which government policy is conceived and implemented, displacing administrative discretion and publicly accountable features of government administration with mechanisms implemented by private actors through algorithmic decision-making systems (Mulligan & Bamberger, 2019).”
Jomthanachai, S., Wong, W. P., & Khaw, K. W. (2023) · Computational Economics · doi:10.1007/s10614-023-10358-7 · Cited by 35
“This is because the ML system design generates policies not just once but over time as they adopt and evolve (Mulligan & Bamberger, 2019).”
BackgroundThe Fertile Dark Matter of Privacy takes on the Dark Patterns of Surveillance
Mulligan, D. K., Regan, P. M., & King, J. (2020) · Journal of Consumer Psychology · doi:10.1002/jcpy.1190 · Cited by 32
Self-citationDor, L. M. B., & Coglianese, C. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3111764 · Cited by 23
Attard-Frost, B., & Widder, D. G. (2025) · Big Data & Society · doi:10.1177/20539517251340603 · Cited by 59
“For example, public sector procurement, development, and use of automated decision systems in public service and courtroom contexts have caused harm to vulnerable groups (Angwin et al., 2016; Eubanks, 2018; Gans-Combe, 2022; Mulligan and Bamberger, 2019).”
Governing Algorithmic Systems with Impact Assessments: Six Observations
Watkins, E. A., Moss, E., Metcalf, J., Singh, R., & Elish, M. C. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462580 · Cited by 49
Beyond Ads: Sequential Decision-Making Algorithms in Law and Public Policy
Henderson, P., Chugg, B., Anderson, B., & Ho, D. E. (2022) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3511265.3550439 · Cited by 31
McBride, K., van Noordt, C., Misuraca, G. C., & Hammerschmid, G. (2021) · doi:10.31235/osf.io/un649 · Cited by 22
Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
Cooper, A. F., Levy, K. E. C., & De, C. (2021) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3465416.3483289 · Cited by 32
“44 Mulligan and Bamberger [65, 66] have notably written about this issue of technological transparency in public governance.”
BackgroundAI and the transformation of accountability and discretion in urban governance
Goldsmith, S., & Yang, J. (2026) · Urban Governance · doi:10.1016/j.ugj.2026.02.005 · Cited by 17
“At the city-level, when governments treat AI adoption as a technical procurement rather than a policymaking endeavor, they risk ceding value-laden decisions to algorithm designers beyond public scrutiny ( Mulligan & Bamberger, 2019 ).”
Non-Determinism and the Lawlessness of Machine Learning Code
Cooper, A. F., Frankle, J., & De, C. (2022) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3511265.3550446 · Cited by 39
“3 Either as a type of architecture [31, 33] or a modality on its own [24]. than using this widely-quoted shorthand to stand in for the more-precise (but still abbreviated) “code is constitutive of law” [3, p. 675], ML code could literally be used to generate law.”
MethodForks over knives: Predictive inconsistency in criminal justice algorithmic risk assessment tools
Greene, T., Shmueli, G., Fell, J., Lin, C.-F., & Liu, H.-W. (2022) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12966 · Cited by 15
Waldman, A. E. (2024) · Law & Social Inquiry · doi:10.1017/lsi.2023.44 · Cited by 14
“Scholars have also shown how agencies can turn technology procurement into a form of policy making, burying policy changes in the designs of computational systems purchased to facilitate administrative work (Mulligan & Bamberger 2019).”
Barnett, J., & Diakopoulos, N. A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534145 · Cited by 24
DoCarmo, T. E., Rea, S. C., Conaway, E., Emery, J. R., & Raval, N. (2021) · Law & Policy · doi:10.1111/lapo.12164 · Cited by 21
Johnson, N., Silva, E., Leon, H., Eslami, M., Schwanke, B., Dotan, R., & Heidari, H. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732049 · Cited by 12
“The past few years have marked an explosion of attention directed towards imagining how AI procurement might be reformed to prepare governments to anticipate AI harms [18, 45, 72, 87, 92, 108].”
InfluentialSanfilippo, M. R., & Frischmann, B. M. (2023) · Internet Policy Review · doi:10.14763/2023.1.1703 · Cited by 16
Meyer, M., Horowitz, A., Marshall, E., & Lum, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533104 · Cited by 14
“The make up of our team is itself a limiting factor that matches risk assessment development: the choices made by our team for a proposed tool for use in the service of the public are highly undemocratic, centralizing of our own perspectives, and generating of policy absent effective process, if used without substanti…”
BackgroundYurrita, M., Verma, H., Balayn, A., Gadiraju, U., Pont, S. C., & Bozzon, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713145 · Cited by 12
“Studies like these would address the need to encourage public participation and reasoned deliberation about public AI, moving away from procurement processes with limited visibility of design choices [76].”
Sheehy, B., & Ng, Y.-F. (2024) · Indiana Law Review · doi:10.18060/28360 · Cited by 7
Zahari, H. M., & Zain, R. (2024) · Springer eBooks · doi:10.1007/978-981-97-1375-2_6 · Cited by 10
Abdu, A. A., Chambers, L. M., Mulligan, D., & Jacobs, A. Z. (2024) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3630106.3658962 · Cited by 12
“Past literature has pointed to how the introduction or substitution of technologies can reconfigure values and social and political arrangements, and that understanding this context shift is key to responsible technology development [11, 79, 90, 110].”
BackgroundMethodInfluentialSelf-citationDiscussing The Legitimacy of Digital Market Surveillance
Goanță, C., & Spanakis, J. (2022) · doi:10.51868/12 · Cited by 6
Ethical Data Science: Prediction in the Public Interest
Washington, A. L. (2023) · doi:10.1093/oso/9780197693025.001.0001 · Cited by 12
Cerrillo i Martínez, A., & Velasco Rico, C. I. (2025) · doi:10.55323/9788413699752 · Cited by 9
Beyond Big Tech: Advocacy Technologists within Mission-Driven Civil Society Organizations
Chambers, L. M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757559 · Cited by 7
“They offer the “in-house technical capacity” that scholars like Mulligan and Bamberger [93, 94] have been calling for to bolster civil society and rectify the “the uneven technological expertise among stakeholders” [93:775].”
Widder, D. G., & Goues, C. L. (2024) · Communications of the ACM · doi:10.1145/3662730 · Cited by 11
The Many Facets of Data Equity
Jagadish, H. V., Stoyanovich, J., & Howe, B. (2022) · Journal of Data and Information Quality · doi:10.1145/3533425 · Cited by 14
“We seek contestability by design [33], an active principle that goes beyond explanation and focuses on user engagement, fostering user understanding of models and outputs, and collaboration in systems design [17, 26, 43].”
MethodReconfiguring Diversity and Inclusion for AI EthicsPreprint
Chi, N., Lurie, E., & Mulligan, D. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.02407 · Cited by 5
“IEEE Ethically Aligned Design); public policy initiatives attending to the values embedded in algorithmic systems used by government [47]; as well as government bans and limitations on the use of specific AI technologies in the policing context [17].”
BackgroundSelf-citationDesign versus reality: assessing the results and compliance of algorithmic impact assessments
Brandusescu, A., & Sieber, R. E. (2025) · Digital Society · doi:10.1007/s44206-025-00221-7 · Cited by 9
“A primary goal of AIAs is to establish a record of automated processes and their justifications, aiming to increase transparency and start a conversation about AI development with different stakeholder groups (Brandusescu, 2021; Mulligan & Bamberger, 2019).”
Suspect Development Systems: Databasing Marginality and Enforcing Discipline
Richardson, R., & Kak, A. (2021) · University of Michigan Journal of Law Reform · doi:10.36646/mjlr.55.4.suspect · Cited by 7
To “In-House” or To Outsource? Artificial Intelligence in Canadian Local Governments
Wan, S., & Sieber, R. E. (2025) · Journal of Urban Technology · doi:10.1080/10630732.2025.2477993 · Cited by 6
“Municipalities may need to commence a complicated procurement process to obtain a vendor for a product whose outcomes are uncertain (Hickok 2022; Mulligan and Bamberger, 2019; Sloane et al., 2021).”
One Bad NOFO? AI Governance in Federal Grantmaking
Bateyko, D., & Levy, K. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732109 · Cited by 3
“Mulligan and Bamberger [64] propose that procurement officials make policy choices about the purchase and use of machine learning tools through procurement processes.”
From Stem to Stern: Contestability Along AI Value Chains
Balayn, A., Pi, Y., Widder, D. G., Alfrink, K., Yurrita, M., Upadhyay, S., Karusala, N., Lyons, H., Turkay, Ç., Tessono, C., et al. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3681831 · Cited by 10
“…have made significant progress in directing the conversation towards making AI more responsive and accountable through ongoing learning based on feedback and contestation [16], their main focus remains on contesting AI design or outputs [4] within specific domains such as content moderation [19].”
Filho, C. E. Z., Kennelly, M., Fullagar, S. P., et al. (2023) · International Journal of Sport Policy and Politics · doi:10.1080/19406940.2023.2206410 · Cited by 3
The Rule of Law for Artificial Intelligence in Public Administration: A System Safety Perspective
Nouws, S., & Dobbe, R. (2024) · Information Technology and Law Series · doi:10.1007/978-94-6265-639-0_9 · Cited by 3
Standing Up a Regulatory Ecosystem for Governing AI Decision-making: Principles and Components
Hudson, V. M. (2022) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197579329.013.17 · Cited by 5
Attribute selection for stroke prediction based on Hadoop and machine learning
Hassan, M. F., & Manaa, M. E. (2023) · AIP conference proceedings · doi:10.1063/5.0121349 · Cited by 2
Expanding Civil Rights to Combat Digital Discrimination on the Basis of Poverty
Gilman, M. E. (2022) · SMU Law Review · doi:10.25172/smulr.75.3.6 · Cited by 7
The Sociology of Technical Choices in Predictive AI
Zanger-Tishler, M., & Zhang, S. (2025) · Social Science Computer Review · doi:10.1177/08944393251367045 · Cited by 2
Experience of administrative and legal support for control and supervision
Bortniak, K. V., Мердова, О. М., Tsomra, V. Y., Daraganova, N., & Mykhailov, R. (2023) · Revista Amazonia Investiga · doi:10.34069/ai/2023.63.03.27 · Cited by 6
Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Apollo (University of Cambridge) · doi:10.17863/cam.65493 · Cited by 1
Progressive AlgorithmsPreprint
Ravid, I., & Haim, A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3815744 · Cited by 1
Smart cities: private means, public ends?
Voorwinden, A. (2023) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.689858230 · Cited by 2
What Clinical Trials Can Teach Us about the Development of More Resilient AI for CybersecurityPreprint
Begoli, E., Bridges, R. A., Oesch, S., & Knight, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06545 · Cited by 1
Expanding Civil Rights to Combat Digital Discrimination on the Basis of PovertyPreprint
Gilman, M. E. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4105801
Nichtwissen bei maschinellem Lernen
Kostov, I. (2023) · Mohr Siebeck eBooks · doi:10.1628/978-3-16-162610-4
Introduction: Ethical Data Science
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0001
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.002.0004
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0003
Model: Dear Validity, Advice for Wayward Algorithms
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0004
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0007
Conclusion: Prediction in the Public Interest
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0009
Predict: Show Us Your Work or Someone Gets Hurt
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0008
Compare: No Comparison Without Representation
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0005
Optimize: Data Science Reasoning
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0006
Prologue: Tracing Ethics in the Prediction Supply Chain
Washington, A. L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197693025.003.0002
The Anti-Managerial Information CommissionPreprint
Kwoka, M. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4685260
Expanding Discretion and Accountability in the Context of AIPreprint
Goldsmith, S., & Yang, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4796633
Rethinking “Accountable Discretion” for Urban Governance in the Context of AiPreprint
Goldsmith, S., & Yang, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4968086 · Cited by 2
Machine learning: A multifaceted exploration of trends, regulations, and global impact
Rishwinder, S. B., Tusharika, S., Santosh, K., & Rakesh, K. (2024) · i-manager’s Journal on Future Engineering and Technology · doi:10.26634/jfet.19.4.20913
Everyday Legitimacy Practices and the Algorithmic Turn in the Administrative StatePreprint
Abdu, A. A., & Jacobs, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5160457
The cost of (data) community: error and repair in data processing pipelines
Metcalf, K. (2025) · BioSocieties · doi:10.1057/s41292-025-00367-0
Constitutional Equality in the Age of Artificial Intelligence
Jaiswal, R., & Bansal, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-7107-8.ch010
Singh, R., Kaur, G., Dsilva, J., Joshi, A., & Kumar, V. (2026) · De Gruyter eBooks · doi:10.1515/9783112237687-013
L'acte administratif algorithmique
Stepanov, A. (2026) · doi:10.70675/adc05be3za62ez4b4fz81e7zabf744e60dbf
Disclosure or Marketing? Analyzing the Efficacy of Vendor Self-reports for Vetting Public-sector AI
Kuehnert, B., Johnson, N., Dotan, R., & Heidari, H. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3812220
Metadata
- Authors
- Mulligan, D. K., & Bamberger, K. A.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Bamberger, Kenneth A.
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2019
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W6924855210
- Cited by
- 75
- References
- 23
- Citations by year
- 2026: 4 · 2025: 10 · 2024: 12 · 2023: 21 · 2022: 14 · 2021: 13 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- UC Berkeley
- Topics
- Social Sciences › Business, Management and Accounting › Strategy and Management › Public Procurement and Policy
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- machine learning, procurement, administrative processes, public procurement, algorithmic decision-making, algorithmic transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.