CoChat
Sign Up
Cited by 81Public Procurement and PolicyEthics and Social Impacts of AIE-Government and Public Services

Public procurement of artificial intelligence systems: new risks and future proofing

Open access
Hickok, Merve·2024·AI & SOCIETY
DOI 10.1007/s00146-022-01572-2PMC PMC9526785
Open this paper in CoChat — free accountOpen PDF

Abstract

Public entities around the world are increasingly deploying artificial intelligence (AI) and algorithmic decision-making systems to provide public services or to use their enforcement powers. The rationale for the public sector to use these systems is similar to private sector: increase efficiency and speed of transactions and lower the costs. However, public entities are first and foremost established to meet the needs of the members of society and protect the safety, fundamental rights, and wellbeing of those they serve. Currently AI systems are deployed by the public sector at various administrative levels without robust due diligence, monitoring, or transparency. This paper critically maps out the challenges in procurement of AI systems by public entities and the long-term implications necessitating AI-specific procurement guidelines and processes. This dual-prong exploration includes the new complexities and risks introduced by AI systems, and the institutional capabilities impacting the decision-making process. AI-specific public procurement guidelines are urgently needed to protect fundamental rights and due process.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00146-022-01572-2

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI could amplify existing challenges due to its technical complexity, increased autonomy, system opacity that impedes error detection, distributed responsibility across multiple suppliers and systems, and risks that propagate across linked systems and models.

    “While some of the discussed risks may also apply for traditional IT systems [61], AI could amplify existing challenges due to its technical complexity, increased autonomy, system opacity that impedes error detection, distributed responsibility across multiple suppliers and systems, and risks that propagate across linked systems and models [88, 91].”

    From “It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism (2026) · cites this paper as [91]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. ACLU (2022b) Settlement secures the right to consideration of release for people arrested by ICE in New York. ACLU New York. https://www.nyclu.org/en/press-releases/settlement-secures-right-consideration-release-people-arrested-ice-new-york
  2. Barocas S, Selbst AD (2016) Big data's disparate impact. California Law Review 104: 671. https://www.californialawreview.org/wp-content/uploads/2016/06/2Barocas-Selbst.pdf
  3. Birhane A, Kalluri P, Card D, Agnew W, Dotan R, Bao M (2022) The values encoded in machine learning research. FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency. June 2022.173–184. 10.1145/3531146.3533083 · DOI · Paper page
  4. Brayne S Predict and surveil: data, discretion, and the future of policing 2020 1 Oxford University Press
  5. Brown S, Carrier R, Hickok M, Smith AL (2021). Bias mitigation in data sets. 10.31235/osf.io/z8qrb · DOI
  6. Cooper AF, Laufer B, Moss E, Nissenbaum H (2022) Accountability in an algorithmic society: relationality, responsibility, and robustness in machine learning. FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency. June 2022. 864–876. 10.1145/3531146.3533150 · DOI
  7. Dobbe R, Dean S, Gilbert TK, Kohli N (2018) A broader view on bias in automated decision-making: reflecting on epistemology and dynamics. Workshop on Fairness, Accountability and Transparency in Machine Learning during ICML 2018, Stockholm, Sweden. 10.48550/arXiv.1807.00553 · DOI
  8. Dunleavy P Margetts HZ Bastow S Tinkler J Digital era governance: IT corporations, the state, and e-government 2007 Oxford University Press
  9. Dwork C, Hardt M, Pitassi T, Reingold O, Zemel RS (2012) Fairness through awareness. ITCS '12: Proc. of the 3rd Innovations in Theoretical Computer Science Conference, 214–226. 10.1145/2090236.2090255 · DOI · Paper page
  10. Eckhouse L Lum K Conti-Cook C Ciccolini J Layers of bias: a unified approach for understanding problems with risk assessment Crim Justice Behav 2019 46 2 185 209 10.1177/0093854818811379 · DOI · Paper page
  11. Engstrom DF, Ho DE, Sharkey CM, Cuéllar M (2020) Government by algorithm: artificial intelligence in federal administrative agencies. Administrative Conference of the United States. https://www-cdn.law.stanford.edu/wp-content/uploads/2020/02/ACUS-AI-Report.pdf
  12. Eubanks V Automating inequality: how high-tech tools profile, police, and punish the poor 2018 St. Martin's Press
  13. Hajian S Domingo-Ferrer J Custers B Calders T Schermer B Zarsky T Direct and indirect discrimination prevention methods Discrimination and privacy in the information society studies in applied philosophy, epistemology and rational ethics 2013 Heidelberg Springer, Berlin
  14. Hickok M, Dorsey C, O’Brien T, Baur D, Ingram K, Chauhan C, Gamundani A (2022) Case study: the distilling of a biased algorithmic decision system through a business lens. 10.2139/ssrn.4019672. https://osf.io/preprints/socarxiv/t5dhu/ · DOI
  15. Holland S, Hosny A, Newman S, Joseph J, Chmielinski K (2018) The dataset nutrition label: a framework to drive higher data quality standards. ArXiv, abs/1805.03677 · arXiv
  16. Koulish R (2016) Immigration detention in the risk classification assessment era. Connecticut Public Interest Law Journal, Vol. 16, No.1. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2865972#
  17. Mitchell S Potash E Barocas S D'Amour A Lum K Algorithmic fairness: choices, assumptions, and definitions Annu Rev Stat Appl 2021 8 1 10.1146/annurev-statistics-042720-125902 · DOI · Paper page
  18. Moss E Watkins EA Singh R Elish MC Metcalf J Assembling accountability: algorithmic impact assessment for the public interest SSRN J 2021 10.2139/ssrn.3877437 · DOI
  19. Mulligan DK Bamberger KA Procurement as policy: administrative process for machine learning Berkley Technol Law J 2019 34 773 10.15779/Z385X25D2W · DOI
  20. O'Neil C Weapons of math destruction: how big data increases inequality and threatens democracy 2016 London Penguin
  21. Pasquale F The black box society: the secret algorithms that control money and information 2015 Cambridge, MA and London Harvard University Press
  22. Ramos G (2022) Ethics of AI and democracy: UNESCO recommendation’s insights. Turkish Policy Quarterly. http://turkishpolicy.com/article/1091/ethics-of-ai-and-democracy-unesco-recommendations-insights
  23. Ribeiro M, Singh S, Guestrin C (2016) Why should I trust you?: explaining the predictions of any classifier. In Proc. of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. 97–101, San Diego, California. Association for Computational Linguistics.
  24. Richardson R, Schultz JM, Crawford K (2019) Dirty data, bad predictions: how civil rights violations impact police data, predictive policing systems and justice. New York University Law Review Online. https://www.nyulawreview.org/wp‐content/uploads/2019/04/NYULawReview‐94‐Richardson_etal‐FIN.pdf
  25. Rudin C Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead Nat Mach Intell 2019 1 206 215 10.1038/s42256-019-0048-x 35603010 PMC9122117 · DOI · Paper page
  26. Schwartz P (1992) Data processing and government administration: the failure of the American legal response to the computer, 43 Hastings Law Journal. 1321
  27. Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M, Crespo J, Dennison D (2015) Hidden technical debt in machine learning systems. NIPS 2503-2511
  28. Selbst AD, Boyd D, Friedler S, Venkatasubramanian S, Vertesi J (2018) Fairness and abstraction in sociotechnical systems (August 23, 2018). 2019 ACM Conference on Fairness, Accountability, and Transparency (FAT*), 59–68. 10.1145/3287560.3287598 · DOI · Paper page
  29. Shin D User perceptions of algorithmic decisions in the personalized AI system: perceptual evaluation of fairness, accountability, transparency, and explainability J Broadcast Electron Media 2020 64 4 541 565 10.1080/08838151.2020.1843357 · DOI
  30. Shin D The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI Int J Hum Comput Stud 2021 146 102551 10.1016/j.ijhcs.2020.102551 · DOI · Paper page
  31. Shin D Lim JS Ahmad N Understanding user sensemaking in fairness and transparency in algorithms: algorithmic sensemaking in over-the-top platform AI Soc 2022 10.1007/s00146-022-01525-9 · DOI
  32. Sloane M, Chowdhury R, Havens JC, Lazovich T, Rincon AL (2021) AI and procurement – a primer. 10.17609/bxzf-df18 · DOI
  33. Veale M, Kleek MV, Binns R (2018) Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. Proc. of the 2018 CHI Conference on Human Factors in Computing Systems.
  34. Wieringa M (2020). What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability. Proc.of the 2020 Conference on Fairness, Accountability, and Transparency. 1–18. 10.1145/3351095.3372833 · DOI · Paper page
  35. Young MM, Bullock JB, Lecy JD (2019) Artificial discretion as a tool of governance: a framework for understanding the impact of artificial intelligence on public administration. Perspectives on Public Management and Governance 2(4). https://academic.oup.com/ppmg/article-abstract/2/4/301/5602198

Cited by

Sorted by citations

Authors
Hickok, M.
Corresponding author
Hickok, Merve
ORCID
Hickok, Merve
Venue
AI & SOCIETY
Publisher
Springer Science and Business Media LLC
Published
2022-10-02
Volume
39
Issue
3
Pages
1213-1227
ISSN
  • 0951-5666 (ISSN-L)
  • 1435-5655
DOI
10.1007/s00146-022-01572-2
PMC
PMC9526785
OpenAlex
W4299285312
Cited by
81
References
18
Citations by year
2026: 31 · 2025: 29 · 2024: 13 · 2023: 8
FWCI
9.34
Type
journal article
Language
en
Open access
Licence
Locations
  • AI & Society (publishedVersion, public-domain)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Business, Management and Accounting › Strategy and Management › Public Procurement and Policy
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Political Science and International Relations › E-Government and Public Services
Keywords
public procurement, artificial intelligence, algorithmic decision-making, public sector AI, procurement guidelines, due diligence, transparency, fundamental rights, due process, risk management, institutional capacity, public services

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review