CoChat
Sign Up
PreprintCited by 8Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Safety Systems Engineering in Autonomy

System Cards for AI-Based Decision-Making for Public Policy

Open access
Gursoy, Furkan, Kakadiaris, Ioannis A.·2022·arXiv
DOI 10.48550/arxiv.2203.04754arXiv 2203.04754
Open this paper in CoChat — free accountOpen PDF

Abstract

Decisions impacting human lives are increasingly being made or assisted by automated decision-making algorithms. Many of these algorithms process personal data for predicting recidivism, credit risk analysis, identifying individuals using face recognition, and more. While potentially improving efficiency and effectiveness, such algorithms are not inherently free from bias, opaqueness, lack of explainability, maleficence, and the like. Given that the outcomes of these algorithms have a significant impact on individuals and society and are open to analysis and contestation after deployment, such issues must be accounted for before deployment. Formal audits are a way of ensuring algorithms meet the appropriate accountability standards. This work, based on an extensive analysis of the literature and an expert focus group study, proposes a unifying framework for a system accountability benchmark for formal audits of artificial intelligence-based decision-aiding systems. This work also proposes system cards to serve as scorecards presenting the outcomes of such audits. It consists of 56 criteria organized within a four-by-four matrix composed of rows focused on (i) data, (ii) model, (iii) code, (iv) system, and columns focused on (a) development, (b) assessment, (c) mitigation, and (d) assurance. The proposed system accountability benchmark reflects the state-of-the-art developments for accountable systems, serves as a checklist for algorithm audits, and paves the way for sequential work in future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.04754

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Data sheets, model cards, and system cards are frameworks that can help structure the systematic presentation of metadata about models and systems and that feed into auditing procedures.

    “Important constituent elements are the notions of data sheets [18, 25, 38], model cards [40], and system cards [22]: frameworks that can help structure the systematic presentation of metadata about models and systems and that feed into auditing procedures [37].”

    From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [22]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 93

  1. [1][1] R. Wexler, 'Life, liberty, and trade secrets: Intellectual property in the criminal justice system,' Stanford Law Review , vol. 70, no. 5, pp. 1343-1430, 2018. -
  2. [3][3] S. Bhatore, L. Mohan, and Y. R. Reddy, 'Machine learning techniques for credit risk evaluation: A systematic literature review,' Journal of Banking and Financial Technology , vol. 4, no. 1, pp. 111-138, 2020. -
  3. [4][4] G. T. Allison and M. H. Halperin, 'Bureaucratic politics: A paradigm and some policy implications,' World Politics , vol. 24, no. S1, pp. 4079, 1972. -
  4. [5][5] R. M. Dawes, 'The robust beauty of improper linear models in decision making,' American Psychologist , vol. 34, no. 7, pp. 571-582, 1979. -
  5. [6][6] B. J. Dietvorst, J. P. Simmons, and C. Massey, 'Algorithm aversion: People erroneously avoid algorithms after seeing them err.' Journal of Experimental Psychology: General , vol. 144, no. 1, pp. 114-126, 2015. -
  6. [7][7] J. Gogoll and M. Uhl, 'Rage against the machine: Automation in the moral domain,' Journal of Behavioral and Experimental Economics , vol. 74, pp. 97-103, 2018. -
  7. [8][8] M. K. Lee, 'Understanding perception of algorithmic decisions: Fairness, trust, and emotion in response to algorithmic management,' Big Data & Society , vol. 5, no. 1, 2018, Art. no. 2053951718756684. -
  8. [9][9] R. P. Kennedy, P. D. Waggoner, and M. Ward, 'Trust in public policy algorithms,' The Journal of Politics , vol. 84, no. 2, pp. 1132-1148, 2021. -
  9. [10][10] J. M. Logg, J. A. Minson, and D. A. Moore, 'Algorithm appreciation: People prefer algorithmic to human judgment,' Organizational Behavior and Human Decision Processes , vol. 151, pp. 90-103, 2019. -
  10. [11][11] S. Brayne, Predict and Surveil: Data, Discretion, and the Future of Policing . New York, NY, USA: Oxford University Press, 2020. -
  11. [14][14] M. Broussard, Artificial Unintelligence: How Computers Misunderstand the World . Cambridge, MA, USA: The MIT Press, 2018. -
  12. [15][15] R. Benjamin, Race After Technology: Abolitionist Tools for the New Jim Code . Medford, MA, USA: Polity Press, 2019. -
  13. [16][16] A. Lambrecht and C. Tucker, 'Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads,' Management Science , vol. 65, no. 7, pp. 2966-2981, 2019. -
  14. [18][18] J. Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan, 'Human decisions and machine predictions,' The Quarterly Journal of Economics , vol. 133, no. 1, pp. 237-293, 2018. -
  15. [19][19] W. Austin and T. A. Williams, 'A survey of judges' responses to simulated legal cases: Research note on sentencing disparity,' The Journal of Criminal Law and Criminology , vol. 68, no. 2, pp. 306310, 1977. -
  16. [20][20] M. K. Dhami and P. Ayton, 'Bailing and jailing the fast and frugal way,' Journal of Behavioral Decision Making , vol. 14, no. 2, pp. 141168, 2001. -
  17. [21][21] M. Loi and M. Spielkamp, 'Towards accountability in the use of artificial intelligence for public administrations,' in Proc. AAAI/ACM Conference on AI, Ethics, and Society , 2021, pp. 757-766. -
  18. [22][22] F. Cech, 'The agency of the forum: Mechanisms for algorithmic accountability through the lens of agency,' Journal of Responsible Technology , vol. 7-8, 2021, Art. no. 100015. -
  19. [23][23] M. Wieringa, 'What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2020, pp. 1-18. -
  20. [24][24] A. Henriksen, S. Enni, and A. Bechmann, 'Situated accountability: Ethical principles, certification standards, and explanation methods in applied AI,' in Proc. AAAI/ACM Conference on AI, Ethics, and Society , 2021, pp. 574-585. -
  21. [26][26] I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes, 'Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2020, pp. 33-44. -
  22. [27][27] J. M¨ okander, J. Morley, M. Taddeo, and L. Floridi, 'Ethics-based auditing of automated decision-making systems: Nature, scope, and limitations,' Science and Engineering Ethics , vol. 27, no. 4, p. 44, 2021. -
  23. [28][28] R. V. Zicari, J. Brodersen, J. Brusseau, B. D¨ udder, T. Eichhorn, T. Ivanov, G. Kararigas, P. Kringen, M. McCullough, F. M¨ oslein, N. Mushtaq, G. Roig, N. St¨ urtz, K. Tolle, J. J. Tithi, I. van Halem, and M. Westerlund, 'Z-Inspection®: A process to assess trustworthy AI,' IEEE Transactions on Technology and Society , vol. 2, no. 2, pp. 83-97, 2021. -
  24. [29][29] J. Bandy, 'Problematic machine behavior: A systematic literature review of algorithm audits,' in Proc. ACM Conference on HumanComputer Interaction , 2021, pp. 1-34. -
  25. [30][30] C. Wilson, A. Ghosh, S. Jiang, A. Mislove, L. Baker, J. Szary, K. Trindel, and F. Polli, 'Building and auditing fair algorithms: A case study in candidate screening,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2021, pp. 666-677. -
  26. [32][32] S. Brown, J. Davidovic, and A. Hasan, 'The algorithm audit: Scoring the algorithms that score us,' Big Data & Society , vol. 8, no. 1, 2021, Art. no. 2053951720983865. -
  27. [33][33] A. Koshiyama, E. Kazim, and P. Treleaven, 'Algorithm auditing: Managing the legal, ethical, and technological risks of artificial intelligence, machine learning, and associated algorithms,' Computer , vol. 55, no. 4, pp. 40-50, 2022. -
  28. [34][34] I. Naja, M. Markovic, P. Edwards, and C. Cottrill, 'A semantic framework to support AI system accountability and audit,' in Proc. European Semantic Web Conference , 2021, pp. 160-176. -
  29. [35][35] E. Tagiou, Y. Kanellopoulos, C. Aridas, and C. Makris, 'A tool supported framework for the assessment of algorithmic accountability,' in Proc. International Conference on Information, Intelligence, Systems and Applications , 2019, pp. 1-9. -
  30. [36][36] P. M. Krafft, M. Young, M. Katell, J. E. Lee, S. Narayan, M. Epstein, D. Dailey, B. Herman, A. Tam, V. Guetler, C. Bintz, D. Raz, P. O. Jobe, F. Putz, B. Robick, and B. Barghouti, 'An action-oriented AI policy toolkit for technology audits by community advocates and activists,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2021, pp. 772-781. -
  31. [37][37] R. P. Bartlett, A. Morse, N. Wallace, and R. Stanton, 'Algorithmic Discrimination and Input Accountability under the Civil Rights Acts,' Berkeley Technology Law Journal, , vol. 36, pp. 675-736, 2020. -
  32. [38][38] P. Saleiro, B. Kuester, L. Hinkson, J. London, A. Stevens, A. Anisfeld, K. T. Rodolfa, and R. Ghani, 'Aequitas: A bias and fairness audit toolkit,' 2019, arXiv:1811.05577. - · arXiv
  33. [39][39] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. III, and K. Crawford, 'Datasheets for datasets,' Communications of The ACM , vol. 64, no. 12, pp. 86-92, 2021. -
  34. [40][40] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru, 'Model cards for model reporting,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2019, pp. 220-229. -
  35. [41][41] L. McGregor, D. Murray, and V. Ng, 'International human rights law as a framework for algorithmic accountability,' International & Comparative Law Quarterly , vol. 68, no. 2, pp. 309-343, 2019. -
  36. [42][42] A. Buhmann, J. Paßmann, and C. Fieseler, 'Managing algorithmic accountability: Balancing reputational concerns, engagement strategies, and the potential of rational discourse,' Journal of Business Ethics , vol. 163, no. 2, pp. 265-280, 2020. -
  37. [43][43] H. Shen, A. DeVos, M. Eslami, and K. Holstein, 'Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors,' in Proc. ACM on Human-Computer Interaction , 2021, pp. 1-29. -
  38. [44][44] F. A. Batarseh, L. Freeman, and C.-H. Huang, 'A survey on artificial intelligence assurance,' Journal of Big Data , vol. 8, no. 1, 2021, Art. no. 60. -
  39. [45][45] E. Kazim and A. Koshiyama, 'AI assurance processes,' 2020, SSRN:3685087. -
  40. [50][50] M. MacCarthy, 'An examination of the algorithmic accountability act of 2019,' 2019, SSRN:3615731. -
  41. [51][51] E. Kazim, D. M. T. Denny, and A. Koshiyama, 'AI auditing and impact assessment: According to the UK information commissioner's office,' AI and Ethics , vol. 1, no. 3, pp. 301-310, 2021. -
  42. [52][52] F. Gursoy, R. Kennedy, and I. Kakadiaris, 'A Critical Assessment of the Algorithmic Accountability Act of 2022,' 2022, SSRN:4193199. -
  43. [53][53] R. Cooper and M. Foster, 'Sociotechnical systems,' American Psychologist , vol. 26, no. 5, pp. 467-474, 1971. -
  44. [54][54] J. A. Kroll, 'The fallacy of inscrutability,' Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , vol. 376, no. 2133, 2018, Art. no. 20180084. -
  45. [55][55] V. Dignum, 'Responsible autonomy,' in Proc. International Joint Conference on Artificial Intelligence , 2017, pp. 4698-4704. -
  46. [56][56] A. D. Selbst, D. Boyd, S. A. Friedler, S. Venkatasubramanian, and J. Vertesi, 'Fairness and abstraction in sociotechnical systems,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2019, pp. 59-68. -
  47. [57][57] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, 'A survey on bias and fairness in machine learning,' ACM Computing Surveys , vol. 54, no. 6, 2021, Art. no. 115. -
  48. [58][58] M. M. Maas, 'Aligning AI regulation to sociotechnical change,' in The Oxford Handbook of AI Governance , J. Bullock, Ed., 2022. -
  49. [59][59] F. K. Doˇ silovi´ c, M. Brˇ ci´ c, and N. Hlupi´ c, 'Explainable artificial intelligence: A survey,' in Proc. International Convention on Information and Communication Technology, Electronics and Microelectronics , 2018, pp. 210-215. -
  50. [60][60] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal, 'Explaining explanations: An overview of interpretability of machine learning,' in Proc. IEEE International Conference on Data Science and Advanced Analytics , 2018, pp. 80-89. -
  51. [61][61] R. Marcinkeviˇ cs and J. E. Vogt, 'Interpretability and explainability: A machine learning zoo mini-tour,' 2020, arXiv:2012.01805. - · arXiv
  52. [62][62] Z. C. Lipton, 'The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.' Queue , vol. 16, no. 3, pp. 31-57, 2018. -
  53. [63][63] A. Erasmus, T. D. P. Brunet, and E. Fisher, 'What is interpretability?' Philosophy & Technology , vol. 34, no. 4, pp. 833-862, 2021. -
  54. [65][65] P. P. Angelov, E. A. Soares, R. Jiang, N. I. Arnold, and P. M. Atkinson, 'Explainable artificial intelligence: An analytical review,' WIREs Data Mining and Knowledge Discovery , vol. 11, no. 5, 2021, Art. no. e1424. -
  55. [66][66] L. Longo, R. Goebel, F. Lecue, P. Kieseberg, and A. Holzinger, 'Explainable artificial intelligence: Concepts, applications, research challenges and visions,' in Proc. Cross Domain Conference for Machine Learning and Knowledge Extraction , 2020, pp. 1-16. -
  56. [67][67] A. D. Selbst and J. Powles, 'Meaningful information and the right to explanation,' International Data Privacy Law , vol. 7, no. 4, pp. 233242, 2017. -
  57. [68][68] L. Edwards and M. Veale, 'Enslaving the algorithm: From a 'right to an explanation' to a 'right to better decisions'?' IEEE Security and Privacy , vol. 16, no. 3, pp. 46-54, 2018. -
  58. [69][69] S. Barocas, A. D. Selbst, and M. Raghavan, 'The hidden assumptions behind counterfactual explanations and principal reasons,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2020, pp. 80-89. -
  59. [70][70] A. D. Selbst and S. Barocas, 'The intuitive appeal of explainable machines,' Fordham Law Review , vol. 87, no. 3, pp. 1085-1139, 2018. -
  60. [71][71] J. A. Kroll, 'Outlining traceability: A principle for operationalizing accountability in computing systems,' in Proc. ACM Conference on Fairness, Accountability, and Transparency , 2021, pp. 758-771. -
  61. [72][72] D. J. Weitzner, H. Abelson, T. Berners-Lee, J. Feigenbaum, J. Hendler, and G. J. Sussman, 'Information accountability,' Communications of the ACM , vol. 51, no. 6, pp. 82-87, 2008. -
  62. [73][73] H. Felzmann, E. F. Villaronga, C. Lutz, and A. Tam` o-Larrieux, 'Transparency you can trust: Transparency requirements for artificial intelligence between legal norms and contextual concerns,' Big Data & Society , vol. 6, no. 1, 2019, Art. no. 2053951719860542. -
  63. [74][74] E. Moss, E. A. Watkins, R. Singh, M. C. Elish, and J. Metcalf, 'Assembling accountability: Algorithmic impact assessment for the public interest,' 2021, SSRN:3877437. -
  64. [75][75] K. Crawford, 'Can an algorithm be agonistic? Ten scenes from life in calculated publics,' Science, Technology, & Human Values , vol. 41, no. 1, pp. 77-92, 2015. -
  65. [76][76] R. Berk, H. Heidari, S. Jabbari, M. Kearns, and A. Roth, 'Fairness in criminal justice risk assessments: The state of the art,' Sociological Methods & Research , vol. 50, no. 1, pp. 3-44, 2021. -
  66. [77][77] S. Wachter, B. Mittelstadt, and C. Russell, 'Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI,' Computer Law & Security Review , vol. 41, 2021, Art. no. 105567. -
  67. [78][78] B. Kenward and T. Sinclair, 'Machine morality, moral progress, and the looming environmental disaster,' Cognitive Computation and Systems , vol. 3, no. 2, pp. 83-90, 2021. -
  68. [79][79] S. Feuerriegel, M. Dolata, and G. Schwabe, 'Fair AI,' Business & Information Systems Engineering , vol. 62, no. 4, pp. 379-384, 2020. -
  69. [80][80] A. L. Hoffmann, 'Where fairness fails: Data, algorithms, and the limits of antidiscrimination discourse,' Information, Communication & Society , vol. 22, no. 7, pp. 900-915, 2019. -
  70. [81][81] P. Hacker, 'Teaching fairness to artificial intelligence: Existing and novel strategies against algorithmic discrimination under EU law,' Common Market Law Review , vol. 55, no. 4, pp. 1143-1185, 2018. -
  71. [82][82] X. Wang, J. Li, X. Kuang, Y.-a. Tan, and J. Li, 'The security of machine learning in an adversarial setting: A survey,' Journal of Parallel and Distributed Computing , vol. 130, pp. 12-23, 2019. -
  72. [83][83] S. Sharma, J. Henderson, and J. Ghosh, 'CERTIFAI: A common framework to provide explanations and analyse the fairness and robustness of black-box models,' in Proc. AAAI/ACM Conference on AI, Ethics, and Society , 2020, pp. 166-172. -
  73. [84][84] H. Xu and S. Mannor, 'Robustness and generalization,' Machine Learning , vol. 86, no. 3, pp. 391-423, 2012. -
  74. [85][85] O. C. Jenkins, D. Lopresti, and M. Mitchell, 'Next wave artificial intelligence: Robust, explainable, adaptable, ethical, and accountable,' 2020, arXiv:2012.06058. - · arXiv
  75. [88][88] B. Custers, F. Dechesne, A. M. Sears, T. Tani, and S. van der Hof, 'A comparison of data protection legislation and policies across the EU,' Computer Law & Security Review , vol. 34, no. 2, pp. 234-243, 2018. -
  76. [89][89] M. L. Rustad and T. H. Koenig, 'Towards a global data privacy standard,' Florida Law Review , vol. 71, no. 2, pp. 365-454, 2019. -
  77. [90][90] M. Gong, Y. Xie, K. Pan, K. Feng, and A. Qin, 'A survey on differentially private machine learning,' IEEE Computational Intelligence Magazine , vol. 15, no. 2, pp. 49-64, 2020. -
  78. [91][91] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, 'Federated learning: Challenges, methods, and future directions,' IEEE Signal Processing Magazine , vol. 37, no. 3, pp. 50-60, 2020. -
  79. [92][92] F. Bieker, M. Friedewald, M. Hansen, H. Obersteller, and M. Rost, 'A process for data protection impact assessment under the European eneral data protection regulation,' in Proc. Privacy Technologies and Policy: Annual Privacy Forum , 2016, pp. 21-37. -
  80. [93][93] R. J. Torraco, 'Writing integrative literature reviews: Using the past and present to explore the future,' Human Resource Development Review , vol. 15, no. 4, pp. 404-428, 2016. -
  81. [94][94] M. R. Lynn, 'Determination and quantification of content validity,' Nursing Research , vol. 35, no. 6, pp. 382-386, 1986. -
  82. [95][95] J. W. Creswell and D. L. Miller, 'Determining validity in qualitative inquiry,' Theory Into Practice , vol. 39, no. 3, pp. 124-130, 2000. -
  83. [96][96] Y. S. Lincoln and E. G. Guba, Naturalistic Inquiry . Thousand Oaks, CA, USA: SAGE Publications, 2022. -
  84. [97][97] P. P. Uhrowczik, 'Data dictionary/directories,' IBM Systems Journal , vol. 12, no. 4, pp. 332-350, 1973. -
  85. [99][99] P. D. Waggoner, R. Kennedy, H. Le, and M. Shiran, 'Big data and trust in public policy automation,' Statistics, Politics and Policy , vol. 10, no. 2, pp. 115-136, 2019. -
  86. [101][101] J. Kleinberg, S. Mullainathan, and M. Raghavan, 'Inherent Trade-Offs in the Fair Determination of Risk Scores,' 2016, arXiv:1609.05807. - · arXiv
  87. [103][103] Y. Wang and K. Chaudhuri, 'Data poisoning attacks against online learning,' 2018, arXiv:1808.08994. - · arXiv
  88. [104][104] C. Atkinson, F. Barth, and D. Brenner, 'Software testing using test sheets,' in Proc. International Conference on Software Testing, Verification, and Validation Workshops , 2010, pp. 454-459. -
  89. [106][106] M. T. Ribeiro, S. Singh, and C. Guestrin, ''Why should I trust you?': Explaining the predictions of any classifier,' in Proc. ACM International Conference on Knowledge Discovery and Data Mining , 2016, pp. 1135-1144. -
  90. [107][107] S. M. Lundberg and S.-I. Lee, 'A unified approach to interpreting model predictions,' in Proc. International Conference on Neural Information Processing Systems , 2017, pp. 4768-4777. -
  91. [108][108] T.-P. Liang, C.-C. Liu, T.-M. Lin, and B. Lin, 'Effect of team diversity on software project performance,' Industrial Management & Data Systems , vol. 107, no. 5, pp. 636-653, 2007. -
  92. [110][110] C. Wang, X.-H. Li, H. Han, S. Wang, L. Wang, C. C. Cao, and L. Chen, 'Counterfactual explanations in explainable AI: A tutorial,' in Proceedings ACM SIGKDD Conference on Knowledge Discovery & Data Mining , 2021, pp. 4080-4081. -
  93. [113][113] M. Bovens, 'Analysing and assessing accountability: A conceptual framework,' European Law Journal , vol. 13, no. 4, pp. 447-468, 2007.

Cited by

Sorted by citations

Related

Authors
Gursoy, F., & Kakadiaris, I. A.
Affiliations
University of Houston (US)
ORCID
Kakadiaris, Ioannis A.
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2203.04754
arXiv
2203.04754
OpenAlex
W4225760721
Cited by
7
References
0
Citations by year
2026: 1 · 2025: 1 · 2024: 2 · 2023: 2 · 2022: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2131504)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
algorithmic accountability, algorithmic auditing, AI decision-making, public policy, algorithmic bias, explainability, algorithmic transparency, algorithmic fairness, benchmarking framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review