CoChat
Sign Up
Cited by 3Ethics and Social Impacts of AIDigitalization, Law, and RegulationE-Government and Public Services

Directions of Technical Innovation for Regulatable AI Systems

Open access
Shen, Xudong, Brown, Hannah, Tao, Jiashu, Strobel, Martin, Tong, Yao, Narayan, Akshay, Soh, Harold, Doshi-Velez, Finale·2024·Communications of the ACM
DOI 10.1145/3653670
Open this paper in CoChat — free account

Abstract

Public sector AI procurement checklists can help guide efforts to create regulatable AI systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3653670

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Public procurement can help bridge the gap between HCI research and public policy because it sits at the intersection of policy, governance, and human-centred design.

    “Public procurement can help bridge this gap: because procurement sits at the intersection of policy, governance, and human-centred design, it provides fertile ground for HCI to contribute [see also 148].”

    From “It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism (2026) · cites this paper as [148]

    The discussion of the two public-sector AI procurement checklists says they inform regulation and identifies HCI as a technical area receiving insufficient attention. That supports procurement as a setting where HCI could contribute, but the paper does not explicitly say procurement bridges HCI research and public policy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. [2][2] Andrews, R.W., Lilly, J.M., Srivastava, D., and Feigh, K.M. The role of shared mental models in human-AI teams: A theoretical review. Theoretical Issues in Ergonomics Science 24, 2 (2023), 129–175;. Go to Citation Crossref Google Scholar
  2. [3][3] Arora, S. and Doshi, P. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence 297 (2021), 103500. Crossref Google Scholar a [...] desired given observed reward functions. b [...] is not identifiable. Advances in inverse RL
  3. [4][4] Bagdasaryan, E., Poursaeed, O., and Shmatikov, V. Differential privacy has disparate impact on model accuracy. Advances in Neural Information Processing Systems 32 (2019). Go to Citation Google Scholar
  4. [5][5] Bayerl, P.S. and Paul, K.I. What determines inter-coder agreement in manual annotations? A meta-analytic investigation. Computational Linguistics 37, 4 (2011), 699–725. Go to Citation Digital Library Google Scholar
  5. [6][6] Belsley, D.A., Kuh, E., and Welsch, R.E. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley & Sons, (2005). Go to Citation Google Scholar
  6. [7][7] Ben-Gal, I. Outlier detection. In Data Mining and Knowledge Discovery Handbook. Springer, (2005), 131–146. Go to Citation Crossref Google Scholar
  7. [10][10] Bondi, E. et al. Role of human-AI interaction in selective prediction. In Proceedings of the AAAI Conf. on Artificial Intelligence, (2022). Go to Citation Google Scholar
  8. [11][11] Brodley, C.E. and Friedl, M.A. Identifying mislabeled training data. J. of Artificial Intelligence Research 11 (1999), 131–167. Go to Citation Digital Library Google Scholar
  9. [12][12] Brown, D.S., Schneider, J., Dragan, A., and Niekum, S. Value alignment verification. In Proceedings of PMLR 139 (2021), 1105–1115. Go to Citation Google Scholar
  10. [14][14] Chuang, C.-Y., Torralba, A., and Jegelka, S. Estimating generalization under distribution shifts via domain-invariant representations. In Proceedings of the Intern. Conf. on Machine Learning, PMLR 119 (2020), 1984–1994. Go to Citation Google Scholar
  11. [15][15] de Man, Y. et al. Opt-in and opt-out consent procedures for the reuse of routinely recorded health data in scientific research and their consequences for consent rate and consent bias: Systematic review. J. of Medical Internet Research 25 (2023), e42131. Go to Citation Crossref Google Scholar
  12. [17][17] Dwork, C. Differential privacy. In Proceedings of Automata, Languages and Programming: 33rd Intern. Colloquium. Springer, (2006), 1–12. Go to Citation Digital Library Google Scholar
  13. [18][18] Feng, J. et al. Clinical artificial intelligence quality improvement: Towards continual monitoring and updating of AI algorithms in healthcare. npj Digital Medicine 5, 1 (2022), 66. Go to Citation Crossref Google Scholar
  14. [19][19] Gajcin, J. et al. Contrastive explanations for comparing preferences of reinforcement learning. In Proceedings of AAAI Conf. on Artificial Intelligence, (2022). Go to Citation Google Scholar
  15. [20][20] Gama, J. et al. A survey on concept drift adaptation. ACM Computing Surveys 46, 4 (2014), 1–37. Go to Citation Digital Library Google Scholar
  16. [21][21] Gehr, T. et al. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In Proceedings of 2018 IEEE Symp. on Security and Privacy. IEEE, (2018), 3–18. Go to Citation Crossref Google Scholar
  17. [22][22] Ghorbani, A., Wexler, J., Zou, J.Y., and Kim, B. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems 32 (2019). Go to Citation Google Scholar
  18. [23][23] Guidotti, R. Counterfactual explanations and how to find them: Literature review and benchmarking. Data Mining and Knowledge Discovery (2022), 1–55. Go to Citation Google Scholar
  19. [24][24] Hadfield-Menell, D. et al. Inverse reward design. Advances in Neural Information Processing Systems 30 (2017). Go to Citation Google Scholar
  20. [25][25] Isaac, W.S. Hope, hype, and fear: The promise and potential pitfalls of artificial intelligence in criminal justice. Ohio St. J. Crim. L. 15 (2017), 543. Go to Citation Google Scholar
  21. [26][26] Karimi, A.-H., Schölkopf, B., and Valera, I. Algorithmic recourse: From counterfactual explanations to interventions. In Proceedings of the 2021 ACM Conf. on Fairness, Accountability, and Transparency. ACM, (2021), 353–362. Digital Library Google Scholar a [...] actionable ways to change the decision. b [...] the foundation for algorithmic recourse. c [...] the purposes of explanation and recourse,
  22. [27][27] Krishna, S., Ma, J., and Lakkaraju, H. Towards bridging the gaps between the right to explanation and the right to be forgotten. In Proceedings of the 40th Intern. Conf. on Machine Learning. JMLR.org, (2023). Digital Library Google Scholar a [...] end, including unlearning and fine-tuning. b [...] nascent, especially for generative models.
  23. [28][28] MacGlashan, J. et al. Interactive learning from policy-dependenthuman feedback. In Proceedings of the 34th Intern. Conf. on Machine Learning, PLMR 70 (2017), 2285–2294. Go to Citation Google Scholar
  24. [30][30] Moos, J. et al. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction 4, 1 (2022), 276–315. Go to Citation Crossref Google Scholar
  25. [31][31] Ong, D.C., Zaki, J., and Goodman, N.D. Computational models of emotion inference in theory of mind: A review and roadmap. Topics in Cognitive Science 11, 2 (2019), 338–357. Go to Citation Crossref Google Scholar
  26. [32][32] Pineau, J. et al. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). The J. of Machine Learning Research 22, 1 (2021), 7459–7478. Go to Citation Digital Library Google Scholar
  27. [34][34] Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206–215. Go to Citation Crossref Google Scholar
  28. [35][35] Samek, W. and Müller, K.-R. Towards explainable artificial intelligence. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, (2019), 5–22. Go to Citation Crossref Google Scholar
  29. [36][36] Sawaragi, Y., Nakayama, H., and Tanino, T. Theory of Multiobjective Optimization. Elsevier, (1985). Go to Citation Google Scholar
  30. [37][37] Shokri, R., Strobel, M., and Zick, Y. On the privacy risks of model explanations. In Proceedings of the 2021 AAAI/ACM Conf. on AI, Ethics, and Society. ACM, (2021), 231–241. Go to Citation Digital Library Google Scholar
  31. [39][39] Sosa, D. Meaningful explanation. Philosophical. Issues 8 (1997), 351–356. Go to Citation Crossref Google Scholar
  32. [41][41] Suresh, H. and Guttag, J.V. A framework for understanding unintended consequences of machine learning. (2019), arXiv preprint arXiv:1901.10002. Go to Citation Google Scholar · arXiv
  33. [42][42] Tsirtsis, S. and Gomez Rodriguez, M. Decisions, counterfactual explanations and strategic behavior. Advances in Neural Information Processing Systems 33 (2020), 16749–16760. Go to Citation Google Scholar
  34. [43][43] Vyas, N., Kakade, S.M., and Barak, B. On provable copyright protection for generative models. In Proceedings of the 40th Intern. Conf. on Machine Learning, PMLR 202 (2023), 35277–35299. Go to Citation Google Scholar
  35. [44][44] Wang, K.R. et al. Interpretability in the wild: A circuit for indirect object identification in GPT-2 small. In Proceedings of Intern. Conf. on Learning Representation, (2023). Go to Citation Google Scholar
  36. [45][45] World Economic Forum. AI Procurement in a Box. Technical report, World Economic Forum, (2020); https://www.weforum.org/reports/ai-procurement-in-a-box/ Go to Citation Google Scholar Cited By View all Hudig AKallina ESingh J(2026)“It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance MechanismProceedings of the 2026 CHI Conference on Human Factors in Computing Systems10.1145/3772318.3791968(1-22)Online publication date: 13-Apr-2026 https://dl.acm.org/doi/10.1145/3772318.3791968 Asrifan ARismawati RSulaiman IAbdullah ANoni N(2026)Enhancing Student Engagement With AIEthical, Legal, and Pedagogical Perspectives on AI in Education10.4018/979-8-3373-3000-6.ch007(189-218)Online publication date: 21-Jan-2026 https://doi.org/10.4018/979-8-3373-3000-6.ch007 Jain RMusengamana A(2025)AI-Integrated Data Governance Frameworks for Real-Time, Scalable, and Ethical Data ArchitecturesGenerative AI-Powered Data Architectures10.4018/979-8-3373-5616-7.ch006(135-164)Online publication date: 21-Nov-2025 https://doi.org/10.4018/979-8-3373-5616-7.ch006 Show More Cited By Index Terms Index Terms (Premium feature) Directions of Technical Innovation for Regulatable AI Systems Computing methodologies Artificial intelligence Social and professional topics Computing / technology policy Government technology policy Governmental regulations Recommendations Recommendations (Premium feature) Public innovation and digital government dg.o '08: Proceedings of the 2008 international conference on Digital government research The potential for digital government has long been recognized, but many governments are only now realizing that simply putting services online is insufficient to reap benefits. Pockets of innovation are occurring all around the world, but they are ... Read More eGovernance implementation model for public sector innovation ICEGOV '12: Proceedings of the 6th International Conference on Theory and Practice of Electronic Governance This study examines the process of innovation in the Federal Government Agencies of Pakistan to enhance the efficiency and effectiveness of public services. Research objective of this study is to provide an eGovernance Implementation Model of how ... Read More Public value innovation: a theoretical framework based on system dynamics dg.o '18: Proceedings of the 19th Annual International Conference on Digital Government Research: Governance in the Data Age The capability to innovate is seen as fundamental to solve the current challenges posed to governments and to create public value to society. However, many studies in the public sector field did not link innovation to existing theories. There are calls ... Read More Comments Comments (Premium feature) Information & Contributors Information Published In cover image Communications of the ACM Communications of the ACM Volume 67, Issue 11 November 2024 · DOI · Paper page

Cited by

Sorted by citations

Authors
Shen, X., Brown, H., Tao, J., Strobel, M., Tong, Y., Narayan, A., Soh, H., & Doshi-Velez, F.
Affiliations
  • National University of Singapore (SG)
  • Harvard University, School of Engineering and Applied Sciences, Cambridge, Massachusetts, United States
ORCID
Venue
Communications of the ACM
Publisher
Association for Computing Machinery (ACM)
Published
2024-10-25
Volume
67
Issue
11
Pages
82-89
ISSN
  • 0001-0782 (ISSN-L)
  • 1557-7317
DOI
10.1145/3653670
OpenAlex
W4403011922
Cited by
3
References
4
Citations by year
2026: 2 · 2025: 1
FWCI
1.25
Type
journal article
Language
en
Open access
Licence
Locations
Communications of the ACM (publishedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
  • Social Sciences › Political Science and International Relations › E-Government and Public Services
Keywords
AI regulation, technological innovation, responsible AI deployment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review