CoChat
Sign Up
PreprintCited by 292Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Digitalization, Law, and Regulation

The DOI matches its DataCite record.

Auditing large language models: a three-layered approach

Open access
Mökander, Jakob, Schuett, Jonas, Kirk, Hannah Rose, Floridi, Luciano·2023·arXiv
DOI 10.48550/arxiv.2302.08500arXiv 2302.08500
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) represent a major advance in artificial intelligence (AI) research. However, the widespread use of LLMs is also coupled with significant ethical and social challenges. Previous research has pointed towards auditing as a promising governance mechanism to help ensure that AI systems are designed and deployed in ways that are ethical, legal, and technically robust. However, existing auditing procedures fail to address the governance challenges posed by LLMs, which display emergent capabilities and are adaptable to a wide range of downstream tasks. In this article, we address that gap by outlining a novel blueprint for how to audit LLMs. Specifically, we propose a three-layered approach, whereby governance audits (of technology providers that design and disseminate LLMs), model audits (of LLMs after pre-training but prior to their release), and application audits (of applications based on LLMs) complement and inform each other. We show how audits, when conducted in a structured and coordinated manner on all three levels, can be a feasible and effective mechanism for identifying and managing some of the ethical and social risks posed by LLMs. However, it is important to remain realistic about what auditing can reasonably be expected to achieve. Therefore, we discuss the limitations not only of our three-layered approach but also of the prospect of auditing LLMs at all. Ultimately, this article seeks to expand the methodological toolkit available to technology providers and policymakers who wish to analyse and evaluate LLMs from technical, ethical, and legal perspectives.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.08500

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Detailing safety measures will often not require divulging implementable details, and when it does, private disclosure to second-party auditors offers a solution.

    “However, detailing safety measures will often not require divulging implementable details, and when it does, private disclosure to second-party auditors (Mökander et al., 2023;ARC, 2022;Hadfield and Clark, 2023;Shevlane et al., 2023) offers a solution.”

    From Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) · cites this paper as Mökander et al., 2023

    Section 4.2 (Governance audits; page unavailable) supports confidential access: auditors may review safeguards under nondisclosure agreements, and their full reports are usually not public. But the paper does not say that detailing safety measures often avoids implementable details, and Section 3.2 specifies independent third-party auditors rather than second-party auditors. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 210

  1. Sandvig, C., Hamilton, K., Karahalios, K., Langbort, C.: Audit- ing algorithms. In: ICA 2014 Data and Discrimination Preconfer- ence, pp. 1-23 (2014). https:// doi. org/ 10. 1109/ DEXA. 2009. 55 · DOI
  2. Diakopoulos, N.: Algorithmic accountability: journalistic investi- gation of computational power structures. Digit. J. 3(3), 398-415 (2015). https:// doi. org/ 10. 1080/ 21670 811. 2014. 976411 · DOI · Paper page
  3. Mökander, J., Floridi, L.: Ethics-based auditing to develop trustworthy AI. Minds Mach. (Dordr) 0123456789, 2-6 (2021). https:// doi. org/ 10. 1007/ s11023-021-09557-8 · DOI · Paper page
  4. Brundage, M., et al.: Toward trustworthy AI development: mechanisms for supporting verifiable claims. ArXiv, no. 2004.07213[cs.CY])., 2020, [Online]. http:// arxiv. org/ abs/ 2004. 07213
  5. Raji, I.D., Buolamwini, J.: Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. In: AIES 2019-Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 429-435, (2019). https:// doi. org/ 10. 1145/ 33066 18. 33142 44 · DOI · Paper page
  6. Mökander, J., Morley, J., Taddeo, M., Floridi, L.: Ethics-based auditing of automated decision-making systems: nature, scope, and limitations. Sci. Eng. Ethics (2021). https:// doi. org/ 10. 1007/ s11948-021-00319-4ORIG INAL · DOI
  7. Cobbe, J., Lee, M.S.A., Singh, J.: Reviewable automated decision-making: a framework for accountable algorithmic systems. In: FAccT 2021-Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 598-609 (2021). https:// doi. org/ 10. 1145/ 34421 88. 34459 21 · DOI · Paper page
  8. Floridi, L.: Infraethics-on the conditions of possibility of morality. Philos. Technol. 30(4), 391-394 (2017). https:// doi. org/ 10. 1007/ s13347-017-0291-1 · DOI · Paper page
  9. Raji, I.D. et al.: Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In: FAT* 2020-Proceedings of the 2020 Conference on Fair- ness, Accountability, and Transparency, pp. 33-44, 2020, doi: https:// doi. org/ 10. 1145/ 33510 95. 33728 73 · DOI · Paper page
  10. Koshiyama, A., Kazim, E., Treleaven, P.: Algorithm auditing: managing the legal, ethical, and technological risks of artifi- cial intelligence, machine learning, and associated algorithms. IEEE 55(4), 40-50 (2022). https:// doi. org/ 10. 1109/ MC. 2021. 30672 25 · DOI · Paper page
  11. Power, M.: The Audit Society: Rituals of Verification. Oxford University Press, Oxford (1997)
  12. Raji, I.D., Xu, P., Honigsberg, C., Ho, D.: Outsider oversight: designing a third party audit ecosystem for AI governance. In: AIES 2022 -Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 557-571, Jul. 2022, https:// doi. org/ 10. 1145/ 35140 94. 35341 81 · DOI · Paper page
  13. Kazim, E., Koshiyama, A.S., Hilliard, A., Polle, R.: Systematiz- ing audit in algorithmic recruitment. J. Intell. 9(3), 1-11 (2021). https:// doi. org/ 10. 3390/ jinte llige nce90 30046 · DOI · Paper page
  14. Robertson, R.E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., Wilson, C.: Auditing partisan audience bias within Google search. In: Proc ACM Hum Comput Interact, vol. 2, no. CSCW, 2018, https:// doi. org/ 10. 1145/ 32744 17 · DOI · Paper page
  15. Buolamwini, J., Gebru, T.: Gender shades: intersectional accu- racy disparities in commercial gender classification. In: Confer- ence on Fairness, Accountability, and Transparency, 2018, pp. 1-15. https:// doi. org/ 10. 2147/ OTT. S1269 05 · DOI · Paper page
  16. Oakden-Rayner, L., et al.: Validation and algorithmic audit of a deep learning system for the detection of proximal femoral fractures in patients in the emergency department: a diagnostic accuracy study. Lancet Digit. Health 4(5), e351-e358 (2022). https:// doi. org/ 10. 1016/ S2589-7500(22) 00004-8 · DOI
  17. Liu, X., Glocker, B., McCradden, M.M., Ghassemi, M., Den- niston, A.K., Oakden-Rayner, L.: The medical algorithmic audit. Lancet Digit. Health 4(5), e384-e397 (2022). https:// doi. org/ 10. 1016/ S2589-7500(22) 00003-6 · DOI · Paper page
  18. Bommasani, R., et al.: On the opportunities and risks of founda- tion models. ArXiv, Aug. 2021, [Online]. http:// arxiv. org/ abs/ 2108. 07258
  19. Floridi, L., Chiriatti, M.: GPT-3: its nature, scope, limits, and consequences. Minds Mach. 30(4), 681-694 (2020). https:// doi. org/ 10. 1007/ s11023-020-09548-1 · DOI · Paper page
  20. Rosenfeld, R.: Two decdes of statistical language modeling where do we go form here? Where do we go from here? Proc. IEEE 88(8), 1270-1275 (2000). https:// doi. org/ 10. 1109/5. 880083 · DOI
  21. Brown, T.B., et al.: Language models are few-shot learners. Adv. Neural Inf. Process. Syst. (2020). https:// doi. org/ 10. 48550/ arxiv. 2005. 14165 · DOI · Paper page
  22. OpenAI.: GPT-4 Technical Report. Mar. 2023. [Online]. https:// arxiv. org/ abs/ 2303. 08774 v3. Accessed 12 Apr 2023
  23. Chowdhery, A., et al.: PaLM: scaling language modeling with pathways. ArXiv (2022). https:// doi. org/ 10. 48550/ arxiv. 2204. 02311 · DOI · Paper page
  24. Thoppilan, R., et al.: LaMDA: language models for dialog appli- cations. ArXiv (2022)
  25. Rae, J.W. et al.: Scaling language models: methods, analysis & insights from training Gopher. ArXiv (2022)
  26. S. Zhang et al., "OPT: Open Pre-trained Transformer Language Models," ArXiv, May 2022, [Online]. Available: http:// arxiv. org/ abs/ 2205. 01068
  27. Hu, Y., Jing, X., Ko, Y., Rayz, J.T.: Misspelling correction with pre-trained contextual language model. In: 2020 IEEE 19th International Conference on Cognitive Informatics & Cognitive Computing (ICCICC), pp. 144-149. (2020)
  28. Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.S., Mel- lor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., Biles, C., Brown, S., Kenton, Z., Hawkins, W., Stepleton, T., Birhane, A., Hendricks, A.L., Rimell, L., Isaac, W., Haas, J., Legassick, S., Irving, G., Gabriel, I.: Taxonomy of risks posed by language models. In: 2022 ACM Conference on Fairness, Accountabil- ity, and Transparency (FAccT '22). Association for Computing Machinery, pp. 214-229. New York, NY, USA. (2022). https:// doi. org/ 10. 1145/ 35311 46. 35330 88 · DOI · Paper page
  29. Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: FAccT 2021-Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, Association for Computing Machinery, Inc, Mar. 2021, pp. 610-623. https:// doi. org/ 10. 1145/ 34421 88. 34459 22 · DOI · Paper page
  30. Shelby, R., et al.: Sociotechnical harms: scoping a taxonomy for harm reduction. ArXiv (2022). https:// doi. org/ 10. 48550/ arxiv. 2210. 05791 · DOI
  31. Perez, E., et al.: Discovering language model behaviors with model-written evaluations. ArXiv, Dec. 2022. [Online]. https:// arxiv. org/ abs/ 2212. 09251 v1. Accessed 22 Mar 2023
  32. Liang, P., et al.: Holistic evaluation of language models; holistic evaluation of language models. ArXiv, 2022. [Online]. https:// arxiv. org/ pdf/ 2211. 09110. pdf. Accessed 13 Feb 2023
  33. Radford, A., et al.: Learning transferable visual models from nat- ural language supervision. In: Proceedings of the 38th Interna- tional Conference on Machine Learning, 2021. [Online]. https:// github. com/ OpenAI/ CLIP. Accessed 20 Jan 2023
  34. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M, Ope- nAI.: Hierarchical text-conditional image generation with CLIP latents. 2022. https:// doi. org/ 10. 48550/ arxiv. 2204. 06125 · DOI · Paper page
  35. Peyrard, M., et al.: Invariant language modeling. ArXiv (2021). https:// doi. org/ 10. 48550/ arxiv. 2110. 08413 · DOI
  36. Ganguli, D., et al.: Predictability and surprise in large generative models. In: ACM International Conference Proceeding Series, pp. 1747-1764, Jun. 2022. https:// doi. org/ 10. 1145/ 35311 46. 35332 29. · DOI · Paper page
  37. Ganguli, D., et al.: Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. ArXiv, 2022. [Online]. https:// github. com/ anthr opics/ hh-rlhf. Accessed 2 Apr 2023
  38. Perez, E., et al.: Red teaming language models with language models. ArXiv (2022). https:// doi. org/ 10. 48550/ arxiv. 2202. 03286 · DOI
  39. Office of U.S. Senator Ron Wyden.: Algorithmic Accountability Act of 2022. In: 117th Congress 2D Session, 2022, https:// doi. org/ 10. 1016/ S0140-6736(02) 37657-8 · DOI
  40. Joshi, A.K.: Natural language processing. Science (1979) 253(5025), 1242-1249 (1991). https:// doi. org/ 10. 1126/ SCIEN CE. 253. 5025. 1242 · DOI
  41. Hirschberg, J., Manning, C.D.: Advances in natural language processing. Science (1979) 349(6245), 261-266 (2015). https:// doi. org/ 10. 1126/ SCIEN CE. AAA86 85/ ASSET/ D33AB 763- A443-444C-B766-A6B69 883BF D7/ ASSETS/ GRAPH IC/ 349_ 261_ F5. JPEG · DOI
  42. Chernyavskiy, A., Ilvovsky, D., Nakov, P.: Transformers: 'The End of History' for Natural Language Processing?," Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12977 LNAI, pp. 677-693, 2021. https:// doi. org/ 10. 1007/ 978-3-030-86523-8_ 41/ TABLES/5 · DOI
  43. Adiwardana, D., et al.: Towards a human-like open-domain Chat- bot. ArXiv (2020). https:// doi. org/ 10. 48550/ arxiv. 2001. 09977 · DOI · Paper page
  44. Wang, A., et al.: SuperGLUE: a stickier benchmark for general- purpose language understanding systems.In: NIPS'19, 2019. https:// doi. org/ 10. 5555/ 34542 87. 34545 81 · DOI
  45. Bai, Y., et al.: Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, Apr. 2022. [Online]. https:// arxiv. org/ abs/ 2204. 05862 v1. Accessed 2 Apr 2023
  46. Stiennon, N., et al. Learning to summarize from human feedback. Adv Neural Inf Process Syst. vol. 2020-December, Sep. 2020. [Online]. https:// arxiv. org/ abs/ 2009. 01325 v3. Accessed 2 Apr 2023
  47. Ouyang, L., et al.: Training language models to follow instruc- tions with human feedback. ArXiv, Mar. 2022. [Online]. https:// arxiv. org/ abs/ 2203. 02155 v1. Accessed 2 Apr 2023
  48. Arcas, B.A.Y.: Do large language models understand us? Dae- dalus 151(2), 183-197 (2022). https:// doi. org/ 10. 1162/ daed_a_ · DOI
  49. Suzgun, M., et al.: Challenging BIG-bench tasks and whether chain-of-thought can solve them. ArXiv, Oct. 2022. [Online]. https:// arxiv. org/ abs/ 2210. 09261 v1. Accessed 2 Apr 2023
  50. Villalobos, P., Sevilla, J., Besiroglu, T., Heim, L., Ho, A., Hobbhahn, M.: Machine learning model sizes and the parameter gap. ArXiv, Jul. 2022, [Online]. http:// arxiv. org/ abs/ 2207. 02852
  51. Hoffmann, J. et al.: Training compute-optimal large language models. ArXiv, Mar. 2022, [Online]. http:// arxiv. org/ abs/ 2203. 15556
  52. Bowman, S.R.: Eight things to know about large language mod- els. Apr. 2023. [Online]. https:// arxiv. org/ abs/ 2304. 00612 v1 Accessed 12 Apr 2023
  53. Rasmy, L., Xiang, Y., Xie, Z., Tao, C., Zhi, D.: Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digit. Med. 4(1), 1-13 (2021). https:// doi. org/ 10. 1038/ s41746-021-00455-y · DOI · Paper page
  54. Wang, Y., Wang, W., Joty, S., Hoi, S.C.H.: CodeT5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation. In: EMNLP 2021-2021 Con- ference on Empirical Methods in Natural Language Processing, Proceedings, pp. 8696-8708, 2021. https:// doi. org/ 10. 18653/ V1/ 2021. EMNLP-MAIN. 685 · DOI · Paper page
  55. Chen, M., et al.: Evaluating large language models trained on code. ArXiv. [Online]. https:// www. github. com/ openai/ human- eval (2021). Accessed 20 Jan 2023
  56. Wang, S., Tu, Z., Tan, Z., Wang, W., Sun, M., Liu, Y.: Language models are good translators. ArXiv (2021). https:// doi. org/ 10. 48550/ arxiv. 2106. 13627 · DOI
  57. Kojima, T., Shane Gu, S., Reid, M., Matsuo, Y., Iwasawa, Y., Google Research: Large language models are zero-shot reason- ers. ArXiv (2022). https:// doi. org/ 10. 48550/ arxiv. 2205. 11916 · DOI · Paper page
  58. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scal- ing laws for neural language models (2020). https:// doi. org/ 10. 48550/ arXiv. 2001. 08361 · DOI · Paper page
  59. Brown, A.R., Kluska, A., Nie, A., Gupta, A., Venkatesh, A., Gholamidavoodi, A., et al.: Beyond the imitation game: Quan- tifying and extrapolating the capabilities of language models (2022). arXiv: 2206. 04615
  60. Borji, A., Ai, Q.: A categorical archive of ChatGPT failures. Feb. 2023. [Online]. https:// arxiv. org/ abs/ 2302. 03494 v7. Accessed 22 Mar 2023
  61. Shevlane, T.: Structured access. In: Bullock, J., Chen, Y.-C., Himmelreich, J., Hudson, V.M., Korinek, A., Young, M., Zhang, B. (eds.) The Oxford Handbook of AI Governance. Oxford Uni- versity Press (2022). https:// doi. org/ 10. 1093/ oxfor dhb/ 97801 97579 329. 013. 39 · DOI
  62. Tamkin, A., Brundage, M., Clark, J., Ganguli, D.: Understanding the capabilities, limitations, and societal impact of large language models. ArXiv, Feb. 2021, [Online]. http:// arxiv. org/ abs/ 2102. 02503
  63. Avin, S., et al.: Filling gaps in trustworthy development of AI. Science 374(6573), 1327-1329 (2021). https:// doi. org/ 10. 1126/ SCIEN CE. ABI71 76 · DOI
  64. Wang, Z.J., Choi, D., Xu, S., Yang, D.: Putting humans in the natural language processing loop: a survey. In: Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing, pp. 47-52 (2021)
  65. Marda, V., Narayan, S.: On the importance of ethnographic meth- ods in AI research. Nat. Mach. Intell. 3(3. Nature Research), 187-189 (2021). https:// doi. org/ 10. 1038/ s42256-021-00323-0 · DOI
  66. Mitchell, M., et al.: Model cards for model reporting. In: FAT* 2019-Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, pp. 220-229, Jan. 2019. https:// doi. org/ 10. 1145/ 32875 60. 32875 96 · DOI · Paper page
  67. Derczynski, L., et al.: Assessing language model deployment with risk cards. [Online]. https:// arxiv. org/ abs/ 2303. 18190 v1 (2023). Accessed 2 Apr 2023
  68. Gebru, T., et al.: Datasheets for datasets. Commun. ACM 64(12), 86-92 (2021). https:// doi. org/ 10. 1145/ 34587 23 · DOI · Paper page
  69. Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., Gold- stein, T.: A watermark for large language models (2023). arXiv: 2301. 10226
  70. Hacker, P., Engel, A., Mauer, M.: Regulating ChatGPT and other large generative AI models (2023). arXiv: 2302. 02337
  71. Helberger, N., Diakopoulos, N.: ChatGPT and the AI Act. Inter- net Policy Rev. (2023). https:// doi. org/ 10. 14763/ 2023.1. 1682 · DOI
  72. Mökander, J., Axente, M., Casolari, F., Floridi, L.: Conform- ity assessments and post-market monitoring: a guide to the role of auditing in the Proposed European AI regulation. Minds Mach. (Dordr) 32(2), 241-268 (2022). https:// doi. org/ 10. 1007/ s11023-021-09577-4 · DOI · Paper page
  73. Senft, S., Gallegos, F.: Information Technology Control and Audit, 3rd edn. CRC Press/Auerbach Publications, Boca Raton (2009)
  74. Dai, W., Berleant, D.: Benchmarking contemporary deep learn- ing hardware and frameworks: A survey of qualitative metrics. In: Proceedings-2019 IEEE 1st International Conference on Cognitive Machine Intelligence, CogMI 2019, pp. 148-155, Dec. 2019. https:// doi. org/ 10. 1109/ COGMI 48466. 2019. 00029. · DOI
  75. Voas, J., Miller, K.: Software certification services: encouraging trust and reasonable expectations. In: IEEE Computer Society, pp. 39-44. [Online]. https:// ieeex plore. ieee. org/ stamp/ stamp. jsp? arnum ber= 17173 42 (2016). Accessed 2 Apr 2023
  76. Dean, S., Gilbert, T.K., Lambert, N., Zick, T.: Axes for Socio- technical Inquiry in AI Research. IEEE Trans. Technol. Soc. 2(2), 62-70 (2021). https:// doi. org/ 10. 1109/ tts. 2021. 30740 97 · DOI
  77. European Commission.: AI liability directive. In: Proposal for a Directive of the European Parliament and of the Council on adapting non-contractual civil liability rules to artificial intelli- gence, pp. 1-29. [Online]. https:// ec. europa. eu/ commi ssion/ sites/ beta-polit ical/ files/ polit ical-guide lines-next-commi ssion_ en. pdf (2022). Accessed 21 Jan 2023
  78. Berlin, I.: The pursuit of the ideal. In: The Crooked Timber of Mankind: Chapters in the History of Ideas, 1988, pp. 1-20. [Online]. https:// www-jstor-org. ezpro xy-prd. bodle ian. ox. ac. uk/ stable/ j. ctt2t t8nd.6# metad ata_ info_ tab_ conte nts. Accessed 20 Jan 2023
  79. Gabriel, I.: Artificial intelligence, values, and alignment. Minds Mach. (Dordr) 30(3), 411-437 (2020). https:// doi. org/ 10. 1007/ s11023-020-09539-2 · DOI · Paper page
  80. Goodman, B.: Hard choices and hard limits in artificial intel- ligence. In: AIES 2021-Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 112-120, Jul. 2021, https:// doi. org/ 10. 1145/ 34617 02. 34625 39 · DOI
  81. Kirk, H.R., Vidgen, B., Röttger, P., Hale, S.A.: Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. ArXiv (2023)
  82. Mittelstadt, B.: Principles alone cannot guarantee ethical AI. Nat. Mach. Intell. 1(11), 501-507 (2019). https:// doi. org/ 10. 1038/ s42256-019-0114-4 · DOI · Paper page
  83. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: a survey on bias and fairness in machine learning. ACM Comput. Surv. (CSUR) (2021). https:// doi. org/ 10. 1145/ 34576 07 · DOI · Paper page
  84. Kleinberg, J.: Inherent trade-offs in algorithmic fairness. ACM SIGMETRICS Perform. Eval. Rev. 46(1), 40-40 (2018). https:// doi. org/ 10. 1145/ 32920 40. 32196 34 · DOI
  85. Kusner, M., Loftus, J., Russell, C., Silva, R.: Counterfactual fairness. In: 31st Conference on Neural Information Processing Systems. [Online]. https:// obama white house. archi ves. gov/ blog/ 2016/ 05/ 04/ big-risks-big-oppor tunit ies-inter secti on-big-data (2017). Accessed 20 Jan 2023
  86. Whittlestone, J., Alexandrova, A., Nyrup, R., Cave, S.: The role and limits of principles in AI ethics: Towards a focus on tensions. In: AIES 2019-Proceedings of the 2019 AAAI/ACM Confer- ence on AI, Ethics, and Society, pp. 195-200 (2019). doi: https:// doi. org/ 10. 1145/ 33066 18. 33142 89 · DOI · Paper page
  87. Gururangan, S, et al.: Don't stop pretraining: adapt language models to domains and tasks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Jul. 2020, https:// doi. org/ 10. 18653/ V1/ 2020. ACL- MAIN. 740 100. O'Neill, O.: A Philosopher Looks at Digital Communication. Cambridge University Press, Cambridge (2021) 101. · DOI · Paper page
  88. Kasirzadeh, A., Gabriel, I.: In conversation with Artificial Intel- ligence: aligning language models with human values. Minds Mach. (Dordr) (2023). https:// doi. org/ 10. 48550/ arxiv. 2209. 00731 102. · DOI · Paper page
  89. Steed, R., Panda, S., Kobren, A., Wick, M.: Upstream mitiga- tion is not all you need: testing the bias transfer hypothesis in pre-trained language models. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 3524-3542 (2022). https:// doi. org/ 10. 18653/ V1/ 2022. ACL-LONG. 247 103. Gupta, K.: Comtemporary Auditing," p. 1095. [Online]. https:// books. google. com/ books/ about/ Conte mpora ry_ Audit ing. html? id= neDFW DyUWu QC (2004). Accessed 18 Feb 2023 104. · DOI
  90. LaBrie, R.C., Steinke, G.H.: Towards a framework for ethical audits of AI algorithms. In: 25th Americas Conference on Infor- mation Systems, AMCIS 2019, pp. 1-5 (2019) 106.
  91. Stodt, J., Reich, C.: Machine learning development audit frame- work: assessment and inspection of risk and quality of data, model and development process. Int. J. Comput. Inform. Eng. 15(3), 187-193, (2021) 107.
  92. Adler, P., et al.: Auditing black-box models for indirect influence. Knowl. Inf. Syst. 54(1), 95-122 (2018). https:// doi. org/ 10. 1007/ s10115-017-1116-3 108. · DOI · Paper page
  93. Kearns, M., Neel, S., Roth, A., Wu, Z.S.: Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In: 35th International Conference on Machine Learning, ICML 2018, vol. 6, pp. 4008-4016 (2018) 109.
  94. Laux, J., Wachter, S., Mittelstadt, B.: Taming the few: platform regulation, independent audits, and the risks of capture created by the DMA and DSA. Comput. Law Secur. Rev. 43, 105613 (2021). https:// doi. org/ 10. 1016/j. clsr. 2021. 105613 110. Selbst, A.D.: An institutional view of algorithmic impact assessments. Harv. J. Law Technol., vol. 35, 2021, [Online]. https:// papers. ssrn. com/ sol3/ papers. cfm? abstr act_ id= 38676 34. Accessed 10 Feb 2023 111. · DOI · Paper page
  95. Bandy, J.: Problematic machine behavior: a systematic literature review of algorithm audits. Proc. ACM Hum. Comput. Interact. 5(1), 1-34 (2021). https:// doi. org/ 10. 1145/ 34491 48 112. Floridi, L., Holweg, M., Taddeo, M., Amaya Silva, J., Mökander, J., Wen, Y.: capAI-a procedure for conducting conformity assessment of AI systems in line with the EU Artificial Intel- ligence Act. SSRN (2022). https:// doi. org/ 10. 2139/ ssrn. 40640 91 113. Minkkinen, M., Laine, J., Mäntymäki, M.: Continuous auditing of artificial intelligence: a conceptualization and assessment of tools and frameworks. Digit. Soc. 1(3), 21 (2022). https:// doi. org/ 10. 1007/ s44206-022-00022-2 114. · DOI
  96. Metaxa, D., et al.: Auditing algorithms. Found. Trends Human- Comput. Interact. 14(4), 272-344 (2021). https:// doi. org/ 10. 1561/ 11000 00083 115. · DOI
  97. Berghout, E., Fijneman, R., Hendriks, L., de Boer, M., Butijn, B.J.: Advanced digital auditing. In: Progress in IS. Cham: Springer Nature, (2023). https:// doi. org/ 10. 1007/ 978-3-031-11089-4 116. · DOI
  98. Mökander, J., Axente, M.: Ethics-based auditing of auto- mated decision-making systems : intervention points and policy implications. AI Soc (2021). https:// doi. org/ 10. 1007/ s00146-021-01286-x 117. · DOI
  99. Brown, S., Davidovic, J., Hasan, A.: The algorithm audit: scoring the algorithms that score us. Big Data Soc 8(1), 205395172098386 (2021). https:// doi. org/ 10. 1177/ 20539 51720 983865 118. Gibson Dunn.: New York City Proposes Rules to Clarify Upcom- ing Artificial Intelligence Law for Employers. https:// www. gibso ndunn. com/ new-york-city-propo ses-rules-to-clari fy-upcom ing- artifi cial-intel ligen ce-law-for-emplo yers/ (2023). Accessed 2 Apr 2023 119. PwC: PwC ethical AI framework (2020). https:// www. pwc. com/ gx/ en/ issues/ data-and-analy tics/ artifi cial-intel ligen ce/ what-is- respo nsible-ai. html. Accessed 10 Feb 2023 120. Deloitte.: Deloitte Introduces Trustworthy AI Framework to Guide Organizations in Ethical Application of Technology. Press release. https:// www2. deloi tte. com/ us/ en/ pages/ about-deloi tte/ artic les/ press-relea ses/ deloi tte-intro duces-trust worthy-ai-frame work. html (2020). Accessed 18 Sep 2020) 121. KPMG.: KPMG offers ethical AI Assurance using CIO Strategy Council standards. Press release. https:// home. kpmg/ ca/ en/ home/ media/ press-relea ses/ 2020/ 11/ kpmg-offers-ethic al-ai-assur ance- using-ciosc-stand ards. html (2020). Accessed 10 Nov 2021 122. EY.: Assurance in the age of AI," 2018, [Online]. https:// assets. ey. com/ conte nt/ dam/ ey-sites/ ey-com/ en_ gl/ topics/ digit al/ ey- assur ance-in-the-age-of-ai. pdf. Accessed 12 Feb 2023 123. NIST: AI Risk Management Framework: Second Draft Notes for Reviewers: Call for comments and contributions. National Institute of Standards and Technology (2022) 124. ISO.: ISO/IEC 23894-Information technology-Artificial intel- ligence-Guidance on risk management. International Organi- zation for Standardization. https:// www. iso. org/ stand ard/ 77304. html, (2023). Accessed 20 Jan 2023 125. NIST.: Risk management guide for information technology sys- tems recommendations of the National Institute of Standards and Technology. National Institute of Standards and Technol- ogy. [Online]. https:// www. hhs. gov/ sites/ defau lt/ files/ ocr/ priva cy/ hipaa/ admin istra tive/ secur ityru le/ nist8 00-30. pdf (2002). Accessed 20 Jan 2023 126. VDE.: VDE SPEC 900012 V1.0 (en). Verband Der Elektrotech- nik. [Online]. www. vde. com (2022). Accessed 20 Jan 2023 127. ICO.: Guidance on the AI auditing framework: Draft guidance for consultation. Information Commissioner's Office, [Online]. https:// ico. org. uk/ media/ about-the-ico/ consu ltati ons/ 26172 19/ guida nce-on-the-ai-audit ing-frame work-draft-for-consu ltati on. pdf (2020). Accessed 12 Feb 2023 128. Institute of Internal Auditors: The IIA's Artificial Intelligence Auditing Framework. The Institute of Internal Auditors-Global Perspectives (2018) 129. ISO.: ISO/IEC 38507:2022-Information technology-Govern- ance of IT-Governance implications of the use of artificial intelligence by organizations. International Organization for Standardization. https:// www. iso. org/ stand ard/ 56641. html? browse= tc (2022). Accessed 20 Jan 2023 130. Institute of Internal Auditors. About Internal Audit. The Institute of Internal Auditors. https:// www. theiia. org/ en/ about-us/ about- inter nal-audit/ (2022). Accessed 20 Jan 2023 131. · DOI · Paper page
  100. Yanisky-Ravid, S., Hallisey, S.K.: Equality and privacy by design: a new model of artificial data transparency via auditing, certification, and safe harbor regimes. Fordham Urban Law J. 46(2), 428-486, (2019). 132. Saleiro, P., et al.: Aequitas: a bias and fairness audit toolkit. ArXiv, no. 2018, 2018, [Online]. http:// arxiv. org/ abs/ 1811. 05577 133.
  101. Costanza-Chock, S., Raji, I.D., Buolamwini, j.: Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem; Who Audits the Auditors? Recommenda- tions from a field scan of the algorithmic auditing ecosystem. In:FAccT'22, vol. 22 (2022). https:// doi. org/ 10. 1145/ 35311 46. 35332 13 134 · DOI · Paper page
  102. Slee, T.: The incompatible incentives of private-sector AI. In: The Oxford Handbook of Ethics of AI, Oxford University Press, pp. 106-123 (2020). https:// doi. org/ 10. 1093/ OXFOR DHB/ 97801 90067 397. 013.6 135 · DOI
  103. Naudé, W., Dimitri, N.: The race for an artificial general intel- ligence: implications for public policy. AI Soc. 35(2), 367-379 (2020). https:// doi. org/ 10. 1007/ S00146-019-00887-X/ METRI CS 136. · DOI
  104. Lauer, D.: You cannot have AI ethics without ethics. AI and Ethics 0123456789, 1-5 (2020). https:// doi. org/ 10. 1007/ s43681-020-00013-4 138. · DOI
  105. Danks, D., London, A.J.: Regulating autonomous systems: beyond standards. IEEE Intell Syst 32(1), 88-91 (2017). https:// doi. org/ 10. 1109/ MIS. 2017.1 139. Mahajan, V., Venugopal, V.K., Murugavel, M., Mahajan, H.: The algorithmic audit: working with vendors to validate radiology-AI algorithms-how we do it. Acad. Radiol. 27(1), 132-135 (2020). https:// doi. org/ 10. 1016/j. acra. 2019. 09. 009 140. · DOI
  106. Zerbino, P., Aloini, D., Dulmin, R., Mininno, V.: Process-min- ing-enabled audit of information systems: methodology and an application. Expert Syst. Appl. 110, 80-92 (2018). https:// doi. org/ 10. 1016/j. eswa. 2018. 05. 030 141. Mittelstadt, B.: Auditing for transparency in content personali- zation systems. Int. J. Commun. 10(June), 4991-5002 (2016) 142. · DOI
  107. Kroll, J.A.: The fallacy of inscrutability. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. (2018). https:// doi. org/ 10. 1098/ rsta. 2018. 0084 143. OECD.: OECD Framework for the Classification of AI sys- tems. Paris. [Online]. https:// doi. org/ 10. 1787/ cb6d9 eca-en (2022) Accessed 11 Apr 2022 144. · DOI
  108. Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., Song, D.: Fooling vision and language models despite localization and attention mechanism. In: 2018 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 4951-4961, Jun. 2018. https:// doi. org/ 10. 1109/ CVPR. 2018. 00520 145. · DOI
  109. Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H., Pierrehumbert, J.B.: HateCheck: functional tests for hate speech detection models. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 41-58 (2021). https:// doi. org/ 10. 18653/ V1/ 2021. ACL-LONG.4 146. Aspillaga, C., Carvallo, A., Araujo, V.: Stress Test evaluation of transformer-based models in natural language understand- ing tasks. In: Proceedings of the 12th Conference on Language Resources and Evaluation, pp. 11-16. [Online]. https:// github. com/ (2020). Accessed 20 Jan 2023 147. · DOI
  110. Dignum, V.: Responsible autonomy. In: Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, vol. 1, p. 5, (2017). .24963/ ijcai.2017/655 148.
  111. Wei, J. et al.: Emergent abilities of large language models. ArXiv (2022) 149.
  112. Sheng, E., Chang, K.W., Natarajan, P., Peng, N.: The woman worked as a babysitter: on biases in language generation. In: 2019 Conference on Empirical Methods in Natural Language Processing, pp. 3407-3412 (2019). https:// doi. org/ 10. 18653/ V1/ D19-1339. 150. · DOI · Paper page
  113. Gehman, S., Gururangan, S., Sap, M., Choi, Y., Smith, N.A.: RealToxicityPrompts: evaluating neural toxic degeneration in language models. In: Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pp. 3356-3369, Sep. 2020, [Online]. http:// arxiv. org/ abs/ 2009. 11462 151.
  114. Song, C., Shmatikov, V.: Auditing data provenance in text-gener- ation models. In: KDD'19, 2019. https:// doi. org/ 10. 1145/ 32925 00. 33308 85 152. EPRS: Auditing the quality of datasets used in algorithmic deci- sion-making systems. European Parliamentary Research Service (2022). https:// doi. org/ 10. 2861/ 98930 153. · DOI
  115. Floridi, L., Strait, A.: Ethical foresight analysis: what it is and why it is needed? Minds Mach. (Dordr) 30(1), 77-97 (2020). https:// doi. org/ 10. 1007/ s11023-020-09521-y 154. · DOI · Paper page
  116. Hodges, C.: Ethics in business practice and regulation. In: Law and Corporate Behaviour : Integrating Theories of Regulation, Enforcement, Compliance and Ethics, pp. 1-21 (2015). https:// doi. org/ 10. 5040/ 97814 74201 124 155. ISO.: ISO/IEC 38500:2015-Information technology-Gov- ernance of IT for the organization. International Organization for Standardization. https:// www. iso. org/ stand ard/ 62816. html (2015). Accessed 20 Jan 2023). 156. Iliescu, F.-M.: Auditing IT Governance. In: Informatica Eco- nomica, 14(1), 93-102. [Online]. https:// www. proqu est. com/ docvi ew/ 14332 36144/ fullt extPDF/ A2EAF E83CB FA461 APQ/1? accou ntid= 13042 & force dol= true (2010). Accessed 20 Jan 2023 157. · DOI
  117. Falco, G., et al.: Governing AI safety through independent audits. Nat. Mach. Intell. 3(7), 566-571 (2021). https:// doi. org/ 10. 1038/ s42256-021-00370-7 158. · DOI · Paper page
  118. Leveson, N.: Engineering a safer world: systems thinking applied to safety. In: Engineering systems. Cambridge: MIT Press (2011) 159.
  119. Dobbe, R.I.J.: System safety and artificial intelligence. In: The Oxford Handbook of AI Governance, p. C67.S1-C67.S18, Oct. 2022. https:// doi. org/ 10. 1093/ OXFOR DHB/ 97801 97579 329. 013. 67 160 · DOI · Paper page
  120. Schuett, J.: Three lines of defense against risks from AI. (2022). https:// doi. org/ 10. 48550/ arxiv. 2212. 08364 161. Bauer, J.: The necessity of auditing artificial intelligence. SSRN 577, 1-16 (2016) 162. · DOI
  121. Chopra, A.K., Singh, M.P.: Sociotechnical systems and ethics in the large. In: AIES 2018-Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp 48-53 (2018). https:// doi. org/ 10. 1145/ 32787 21. 32787 40 163 · DOI
  122. Contractor, D., et al.: Behavioral use licensing for responsible AI. In" 2022 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA: ACM, Jun. 2022, pp. 778-788. https:// doi. org/ 10. 1145/ 35311 46. 35331 43. 164. Schuett, J.: Risk management in the artificial intelligence act. Eur. J. Risk Regul. (2023). https:// doi. org/ 10. 1017/ ERR. 2023.1 165. Carlini, N., et al.: Extracting training data from large language models. In: Proceedings of the 30th USENIX Security Sympo- sium, 2021. [Online]. https:// www. usenix. org/ confe rence/ useni xsecu rity21/ prese ntati on/ carli ni-extra cting. Accessed 20 Jan 2023 166. · DOI · Paper page
  123. Dwork, C.: Differential privacy. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intel- ligence and Lecture Notes in Bioinformatics), vol. 4052 LNCS, pp. 1-12 (2006). https:// doi. org/ 10. 1007/ 11787 006_1/ COVER 167. · DOI
  124. Kaissis, G.A., Makowski, M.R., Rückert, D., Braren, R.F.: Secure, privacy-preserving and federated machine learning in medical imaging. Nat. Mach. Intell. 2(6), 305-311 (2020). https:// doi. org/ 10. 1038/ s42256-020-0186-1 168. · DOI · Paper page
  125. Bharadwaj, K.B.P., Kanagachidambaresan, G.R.: Pattern rec- ognition and machine learning. (2021). https:// doi. org/ 10. 1007/ 978-3-030-57077-4_ 11 169. · DOI
  126. Crisan, A., Drouhard, M., Vig, J., Rajani, N.: Interactive model cards: a human-centered approach to model documentation. In: ACM International Conference Proceeding Series, vol. 22, pp. 427-439, Jun. 2022. https:// doi. org/ 10. 1145/ 35311 46. 35331 08 170 · DOI · Paper page
  127. Pushkarna, M., Zaldivar, A., Kjartansson, O.: Data cards: pur- poseful and transparent dataset documentation for responsible AI. In: ACM International Conference Proceeding Series, pp. 1776-1826, Jun. 2022. https:// doi. org/ 10. 1145/ 35311 46. 35332 31 171 · DOI
  128. Jernite, Y. et al.: Data governance in the age of large-scale data- driven language technology. In 2022 ACM Conference on Fair- ness, Accountability, and Transparency, New York, NY, USA: ACM, Jun. 2022, pp. 2206-2222. https:// doi. org/ 10. 1145/ 35311 46. 35346 37 172 · DOI · Paper page
  129. Paullada, A., Raji, I.D., Bender, E.M., Denton, E., Hanna, A.: Data and its (dis)contents: a survey of dataset development and use in machine learning research. Patterns 2(11), 100336 (2021). https:// doi. org/ 10. 1016/J. PATTER. 2021. 100336 173. · DOI · Paper page
  130. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: EMNLP 2018 -2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Proceedings of the 1st Workshop, pp. 353-355 (2018). https:// doi. org/ 10. 18653/ V1/ W18-5446. 174. Rudner, T.J., Toner, H.: Key concepts in AI Safety: robustness and adversarial examples. In: CSET Issue Brief (2021) 175. · DOI · Paper page
  131. Sohoni, N.S., Dunnmon, J.A., Angus, G., Gu, A., Ré, C.: No Sub- class Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems. In: 34th Conference on Neural Informa- tion Processing Systems, Nov. 2020, [Online]. http:// arxiv. org/ abs/ 2011. 12945 176.
  132. Garg, S., Ramakrishnan, G.: BAE: BERT-based adversarial examples for text classification. In: EMNLP 2020 -2020 Con- ference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 6174-6181, (2020). https:// doi. org/ 10. 18653/ V1/ 2020. EMNLP-MAIN. 498 177. · DOI
  133. Li, L., Ma, R., Guo, Q., Xue, X., Qiu, X.: BERT-ATTACK: adversarial attack against BERT using BERT. In: EMNLP 2020-2020 Conference on Empirical Methods in Natural Lan- guage Processing, Proceedings of the Conference, pp. 6193- 6202, 2020. https:// doi. org/ 10. 18653/ V1/ 2020. EMNLP-MAIN. 500. 178. Goel, K., Rajani, N.. Vig, J.. Taschdjian, Z., Bansal, M., Ré, C.: Robustness gym: unifying the NLP evaluation landscape. In: NAACL-HLT 2021-2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Demonstrations, pp. 42-55 (2021). https:// doi. org/ 10. 18653/ V1/ 2021. NAACL-DEMOS.6. 179. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., Kiela, D.: Adversarial NLI: A new benchmark for natural language understanding. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, Jul. 2020. https:// doi. org/ 10. 18653/ V1/ 2020. ACL-MAIN. 441 180. · DOI · Paper page
  134. Kiela, D., et al.: Dynabench: rethinking benchmarking in NLP. In: NAACL-HLT 2021-2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Confer- ence, pp. 4110-4124, (2021). https:// doi. org/ 10. 18653/ V1/ 2021. NAACL-MAIN. 324 181. Wang, B., et al.: Adversarial GLUE: a multi-task benchmark for robustness evaluation of language models. In: NeurIPS 2021, Nov. 2021. https:// doi. org/ 10. 48550/ arxiv. 2111. 02840. 182. Zhang, M., Ré, C.: Contrastive adapters for foundation model group robustness. In: ICML 2022 Workshop on Spurious Cor- relations, Jul. 2022. https:// doi. org/ 10. 48550/ arxiv. 2207. 07180 183. · DOI
  135. McMahan, H.B., Ramage, D., Talwar, K., Zhang, L.: Learning differentially private recurrent language models. In: ICLR 2018 Conference Blind Submission. Feb. 24, 2018 184.
  136. Jayaraman, B., Evans, D.: Evaluating differentially private machine learning in practice. In: Proceedings of the 28th USE- NIX Security Symposium, Feb. 2019, [Online]. http:// arxiv. org/ abs/ 1902. 08874 185.
  137. Carlini, N., Brain, G., Liu, C., Erlingsson, Ú., Kos, J., Song, D.: The secret sharer: evaluating and testing unintended memoriza- tion in neural networks. In: Proceedings of the 28th USENIX Security Symposium, 2019, [Online]. https:// www. usenix. org/ confe rence/ useni xsecu rity19/ prese ntati on/ carli ni. Accessed 10 Feb 2023 186.
  138. Evans, O., Cotton-Barratt, O., Finnveden, L., Bales, A., Balwit, A., Wills, P., et al.: Truthful AI: developing and governing AI that does not lie (2021). arXiv: 2110. 06674 187.
  139. Lin, S., Openai, J.H., Evans, O.: TruthfulQA: measuring how models mimic human falsehoods. In: Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics, vol. 1, pp. 3214-3252, Jun. 2022. https:// doi. org/ 10. 18653/ V1/ 2022. ACL-LONG. 229 188. Nejadgholi., I., Kiritchenko, S.: On cross-dataset generalization in automatic detection of online abuse. In: Proceedings of the Fourth Workshop on Online Abuse and Harms, pp. 173-183 (2020). https:// doi. org/ 10. 18653/ v1/ P17 189. · DOI
  140. Caliskan, A., Bryson, J.J., Narayanan, A.: Semantics derived automatically from language corpora contain human-like biases. Science (1979) 356(6334), 183-186 (2017). https:// doi. org/ 10. 1126/ SCIEN CE. AAL42 30/ SUPPL_ FILE/ CALIS KAN-SM. PDF 190. · DOI
  141. Jo, E.S., Gebru, T.: Lessons from archives: Strategies for col- lecting sociocultural data in machine learning. In: FAT* 2020 - Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 306-316, Jan. 2020. doi: https:// doi. org/ 10. 1145/ 33510 95. 33728 29 191. Dodge, J., et al.: Documenting large Webtext corpora: a case study on the colossal clean crawled corpus. In: EMNLP 2021 -2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pp. 1286-1305 (2021). https:// doi. org/ 10. 18653/ V1/ 2021. EMNLP-MAIN. 98 192. Webster, K., et al.: Measuring and reducing gendered correlations in pre-trained models. ArXiv (2020). https:// doi. org/ 10. 48550/ arxiv. 2010. 06032 193. · DOI · Paper page
  142. May, C., Wang, A., Bordia, S., Bowman, S.R., Rudinger, R.: On measuring social biases in sentence encoders. In: NAACL HLT 2019-2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies -Proceedings of the Conference, vol. 1, pp. 622-628 (2019). https:// doi. org/ 10. 18653/ V1/ N19-1063 194. · DOI · Paper page
  143. Nadeem, M., Bethke, A., Reddy, S.: StereoSet: Measuring stereo- typical bias in pretrained language models. In: Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, pp. 5356-5371. [Online]. https://stereoset (2021). Accessed 20 Jan 2023 195.
  144. Bender, E.M., Friedman, B.: Data statements for natural language processing: toward mitigating system bias and enabling better science. Trans. Assoc. Comput. Linguist. 6, 587-604 (2018). https:// doi. org/ 10. 1162/ TACL_A_ 00041 196. · DOI · Paper page
  145. Schat, E., van de Schoot, R., Kouw, W.M., Veen, D., Mendrik, A.M.: The data representativeness criterion: Predicting the per- formance of supervised classification based on data set similar- ity. PLoS ONE 15(8), e0237009 (2020). https:// doi. org/ 10. 1371/ JOURN AL. PONE. 02370 09 197. Kreutzer, J., et al.: Quality at a glance: an audit of web-crawled multilingual datasets. Trans. Assoc. Comput. Linguist. 10, 50-72 (2022). https:// doi. org/ 10. 1162/ TACL_A_ 00447/ 109285 198. · DOI
  146. Simig, D., et al.: Text Characterization Toolkit (TCT). In: Pro- ceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Processing: System Demonstrations. pp. 72-87. [Online]. https:// aclan tholo gy. org/ 2022. aacl-demo.9 (2022). Accessed 21 Jan 2023 199.
  147. Hancox-Li, L., Kumar, I.E.: Epistemic values in feature impor- tance methods: Lessons from feminist epistemology. In: FAccT 2021-Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 817-826, Mar. 2021. https:// doi. org/ 10. 1145/ 34421 88. 34459 43. 200. Dash, A., Mukherjee, A., Ghosh, S.: A network-centric frame- work for auditing recommendation systems. In: Proceedings- IEEE INFOCOM, vol. April, pp. 1990-1998 (2019). https:// doi. org/ 10. 1109/ INFOC OM. 2019. 87374 86. 201. Idowu, S.O.: Legal compliance. In: Idowu, S.O., Capaldi, N., Zu, L., Gupta, A.D. (eds.) Encyclopedia of Corporate Social Respon- sibility, pp. 1578-1578. Springer. Berlin (2013). https:// doi. org/ 10. 1007/ 978-3-642-28036-8_ 100980 202. · DOI
  148. Jobin, A., Ienca, M., Vayena, E.: The global landscape of AI eth- ics guidelines. Nat. Mach. Intell. 1(9), 389-399 (2019). https:// doi. org/ 10. 1038/ s42256-019-0088-2 203. · DOI · Paper page
  149. Green, R.M., Donovan, A.: The methods of business ethics. The Oxford Handbook of Business Ethics (2009). https:// doi. org/ 10. 1093/ OXFOR DHB/ 97801 95307 955. 003. 0002 204. · DOI
  150. Raji, I.D., Kumar, I.E., Horowitz, A., Selbst, A.: The fallacy of AI functionality. In: ACM International Conference Proceeding Series, pp. 959-972, Jun. 2022. https:// doi. org/ 10. 1145/ 35311 46. 35331 58 205. Rahwan, I.: Society-in-the-loop: programming the algorithmic social contract. Ethics Inf. Technol. 20(1), 5-14 (2018). https:// doi. org/ 10. 1007/ s10676-017-9430-8 206. · DOI · Paper page
  151. Dafoe, A.: AI governance: a research agenda, no. July 2017 (2017). https:// doi. org/ 10. 1176/ ajp. 134.8. aj134 8938. 207. · DOI
  152. Truby, J., Brown, R.D., Ibrahim, I.A., Parellada, O.C.: A sand- box approach to regulating high-risk artificial intelligence applications. Eur. J. Risk Regul. 13(2), 270-294 (2022). https:// doi. org/ 10. 1017/ ERR. 2021. 52 208. Akpinar, N.-J., et al.: A sandbox tool to bias(Stress)-test fairness algorithms. ArXiv (2022). https:// doi. org/ 10. 48550/ arxiv. 2204. 10233 209. · DOI
  153. Zinda, N.: Ethics auditing framework for trustworthy AI: lessons from the IT audit literature. In: Mökander J., Ziosi, M. (eds.) The 2021 Yearbook of the Digital Ethics Lab. Springer Cham (2021). https:// doi. org/ 10. 1007/ 978-3-031-09846-8 210. · DOI
  154. Mantelero, A.: AI and Big Data: A blueprint for a human rights, social and ethical impact assessment. Comput. Law Secur. Rev. 34(4), 754-772 (2018). https:// doi. org/ 10. 1016/j. clsr. 2018. 05. 017 211. · DOI
  155. Etzioni, A., Etzioni, O.: AI assisted ethics. Ethics Inf. Technol. 18(2), 149-156 (2016). https:// doi. org/ 10. 1007/ s10676-016-9400-6 213. · DOI · Paper page
  156. Whittlestone, J., Clarke, S.: AI challenges for society and eth- ics. In: Bullock, J., Chen, Y.-C., Himmelreich, J., Hudson, V.M., Korinek, A., Young, M., Zhang, B. (eds.) The Oxford Handbook of AI Governance. Oxford University Press (2022). https:// doi. org/ 10. 1093/ oxfor dhb/ 97801 97579 329. 013.3 214 · DOI
  157. Karan, M., Šnajder, j.: Preemptive toxic language detection in Wikipedia comments using thread-level context. In: Proceed- ings of the Third Workshop on Abusive Language Online, pp. 129-134, Sep. 2019. https:// doi. org/ 10. 18653/ V1/ W19-3514 215. · DOI
  158. Gao, L., Huang, R.: Detecting online hate speech using context aware models. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pp. 260-266, Nov. 2017, https:// doi. org/ 10. 26615/ 978-954-452- 049-6_ 036 216. · DOI
  159. Delobelle, P., Tokpo, E.K., Calders, T., Berendt, B.: Measuring fairness with biased rulers: a comparative study on bias met- rics for pre-trained language models. In: NAACL 2022-2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pp. 1693-1706. (2022). https:// doi. org/ 10. 18653/ V1/ 2022. NAACL-MAIN. 122. 217. Nozza, D., Bianchi, F., Hovy, D.: HONEST: measuring hurt- ful sentence completion in language models. In: NAACL-HLT 2021-2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pp. 2398-2406 (2021). https:// doi. org/ 10. 18653/ V1/ 2021. NAACL-MAIN. 191. 218. · DOI · Paper page
  160. Costello, M., Hawdon, J., Bernatzky, C., Mendes, K.: Social group identity and perceptions of online hate. Sociol. Inq. 89(3), 427-452 (2019). https:// doi. org/ 10. 1111/ SOIN. 12274 219. · DOI
  161. Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., Smith, N. A.: Annotators with attitudes: how annotator beliefs and iden- tities bias toxic language detection. In: NAACL 2022-2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pp. 5884-5906 (2022). https:// doi. org/ 10. 18653/ V1/ 2022. NAACL-MAIN. 431 220. · DOI
  162. Kirk, H.R., Birhane, A., Vidgen, B., Derczynski, L.: Handling and Presenting Harmful Text in NLP Research. Findings of the Association for Computational Linguistics: EMNLP 2022 (2022). https:// aclan tholo gy. org/ 2022. findi ngs-emnlp. 35/ 221.
  163. Welbl, J., et al.: Challenges in detoxifying language models. In: Findings of the Association for Computational Linguistics, Find- ings of ACL: EMNLP 2021, pp. 2447-2469, Sep. 2021, https:// doi. org/ 10. 48550/ arxiv. 2109. 07445 222. · DOI
  164. Rauh, M., et al.: Characteristics of harmful text: towards rigorous benchmarking of language models. ArXiv (2022). https:// doi. org/ 10. 48550/ arxiv. 2206. 08325 223. · DOI
  165. Nangia, N., Vania, C., Bhalerao, R., Bowman, S.R.: CrowS-pairs: a challenge dataset for measuring social biases in masked lan- guage models. In: EMNLP 2020-2020 Conference on Empiri- cal Methods in Natural Language Processing, Proceedings of the Conference, pp. 1953-1967 (2020). https:// doi. org/ 10. 18653/ V1/ 2020. EMNLP-MAIN. 154 224. Rudinger, R.: GitHub -rudinger/winogender-schemas: Data for evaluating gender bias in coreference resolution systems. GitHub. https:// github. com/ rudin ger/ winog ender-schem as (2019). Accessed 25 Jan 2023 225. · DOI · Paper page
  166. Cihon, P., Kleinaltenkamp, M.J., Schuett, J., Baum, S.D.: AI Certification: advancing ethical practice by reducing information asymmetries. IEEE Trans. Technol. Soc. 2(4), 200-209 (2021). https:// doi. org/ 10. 1109/ tts. 2021. 30775 95 226. Cihon, P., Schuett, J., Baum, S.D.: Corporate governance of arti- ficial intelligence in the public interest. Information 12(7), 1-30 (2021). https:// doi. org/ 10. 3390/ info1 20702 75 227. FDA.: Artificial intelligence and machine learning in software as a medical device. In: U.S. Food & Drug Administration, 2021. https:// www. fda. gov/ medic al-devic es/ softw are-medic al- device-samd/ artifi cial-intel ligen ce-and-machi ne-learn ing-softw are-medic al-device (2021). Accessed 20 Jan 2023 228. · DOI
  167. Jacobs, A.Z., Wallach, H.: Measurement and fairness. In: FAccT 2021-Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, vol. 11, no. 21, pp. 375-385, Mar. 2021, https:// doi. org/ 10. 1145/ 34421 88. 34459 01. 229. Mökander, J., Floridi, L.: Operationalising AI governance through ethics-based auditing: an industry case study. AI and Ethics (2022). https:// doi. org/ 10. 1007/ s43681-022-00171-7 230. · DOI
  168. Mökander, J., Sheth, M., Gersbro-Sundler, M., Blomgren, P., Floridi, L.: Challenges and best practices in corporate AI gov- ernance: Lessons from the biopharmaceutical industry. Front. Comput. Sci. (2022). https:// doi. org/ 10. 3389/ fcomp. 2022. 10683 61 231 · DOI
  169. Smith, E.: Research design. In: H. Reis, H., Judd, C. (eds.) Handbook of Research Methods in Social and Personality Psy- chology, pp. 27-48 (2014) [Online]. https:// doi. org/ 10. 1017/ CBO97 80511 996481. 006 232. · DOI
  170. Sobieszek, A., Price, T.: Playing games with Ais: the limits of GPT-3 and similar large language models. Minds Mach. (Dordr) 32(2), 341-364 (2022). https:// doi. org/ 10. 1007/ s11023-022-09602-0 233. · DOI · Paper page
  171. Floridi, L.: AI as Agency without Intelligence: on ChatGPT, large language models, and other generative models. SSRN. [Online]. https:// papers. ssrn. com/ sol3/ papers. cfm? abstr act_ id= 43587 89 (2022). Accessed 13 Feb 2023 234.
  172. Reynolds, L., Ai, M., Ai, K., Mcdonell, K.: Prompt program- ming for large language models: beyond the few-shot para- digm. In: Conference on Human Factors in Computing Sys- tems-Proceedings, May 2021, https:// doi. org/ 10. 1145/ 34117 63. 34517 60 235. Hacking, I.: Representing and Intervening: Introductory Topics in the Philosophy of Natural Science. Cambridge University Press, Cambridge (1983) 236. · DOI · Paper page
  173. Rorty, R.: Pragmatism as Anti-authoritarianism. Harvard Uni- versity Press, Cambridge (2021) 237.
  174. Watson, D.S., Mökander, J.: In defense of sociotechnical prag- matism. In: Mazzi, F. (ed.) The 2022 Yearbook of the Digital Governance Research Group. Springer (2023). https:// doi. org/ 10. 1007/ 978-3-031-28678-0 239. · DOI
  175. Lee, M.S.A., Floridi, L., Singh, J.: "ormalising Trade-offs beyond algorithmic fairness: lessons from ethical philosophy and welfare economics. In: The 2021 Yearbook of the Digital Ethics Lab, pp. 157-182. Cham: Springer (2022). https:// doi. org/ 10. 1007/ 978-3-031-09846-8_ 11 240. · DOI
  176. Friedler, S.A., Scheidegger, C., Venkatasubramanian, S.: The (Im)possibility of fairness. Commun ACM 64(4), 136-143 (2021). https:// doi. org/ 10. 1145/ 34339 49 241. Islam, G., Greenwood, M.: The metrics of ethics and the eth- ics of metrics. J. Bus. Ethics (2021). https:// doi. org/ 10. 1007/ s10551-021-05004-x 242. · DOI
  177. Cugueró-Escofet, N., Rosanas, J.M.: The ethics of metrics: over- coming the dysfunctional effects of performance measurements through justice. J. Bus. Ethics 140(4), 615-631 (2017). https:// doi. org/ 10. 1007/ S10551-016-3049-2/ TABLES/2 243. Boddington, P.: Towards a code of ethics for artificial intelli- gence. In: Artificial Intelligence: Foundations, Theory, and Algo- rithms. Switzerland: Springer Cham, (2017) 244. · DOI
  178. Minkkinen, M., Zimmer, M.P., Mäntymäki, M.: Towards Ecosys- tems for Responsible AI: Expectations, Agendas and Networks in EU Documents. Springer International Publishing (2021). https:// doi. org/ 10. 1007/ 978-3-030-85447-8 245. · DOI
  179. Schöppl, N., Taddeo, M., Floridi, L.: Ethics auditing: lessons from business ethics for ethics auditing of AI. In: Mökander J., Ziosi, M. (eds.) The 2021 Yearbook of the Digital Ethics Lab, pp. 209-227. Springer, Cham (2021). https:// doi. org/ 10. 1007/ 978-3-031-09846-8 246. FDA.: Inspection classification database. U.S. Food & Drug Administration. https:// www. fda. gov/ inspe ctions-compl iance- enfor cement-and-crimi nal-inves tigat ions/ inspe ction-class ifica tion-datab ase (2022). Accessed 20 Jan 2023 247. British Safety Council.: About the British Safety Council. Web- site. https:// www. brits afe. org/ about-us/ intro ducing-the-briti sh- safety-counc il/ about-the-briti sh-safety-counc il/ (2023). Accessed 20 Jan 2023 248. Rainforest Alliance.: Our approach. Website. https:// www. rainf orest-allia nce. org/ appro ach/?_ ga=2. 13719 1288. 95390 5227. 16581 39559-11302 50530. 16581 39559 (2023). Accessed 20 Jan 2023 249. IAEA.: Quality management audits in nuclear medicine prac- tices. IAEA Human Health Series, vol. 33. [Online]. http:// www. iaea. org/ Publi catio ns/ index. html (2015). Accessed 20 Jan 2023 250. · DOI
  180. Duflo, E., Greenstone, M., Pande, R., Ryan, N.: Truth-telling by third-party auditors and the response of polluting firms: experi- mental evidence from India. Q. J. Econ. 128(4), 1499-1545 (2013). https:// doi. org/ 10. 1093/ QJE/ QJT024 251. · DOI
  181. Tutt, A.: An FDA for Algorithms. Adm. Law. Rev. 69(1), 83-123 (2017). https:// doi. org/ 10. 2139/ ssrn. 27479 94 252. Carpenter, D.: Reputation and power: organizational image and pharmaceutical regulation at the FDA. In: Reputation and Power: Organizational Image and Pharmaceutical Regulation at the FDA, pp. 1-802, (2014). https:// doi. org/ 10. 5860/ choice. 48-3548 253. · DOI
  182. Fraser, H.L., Bello y Villarino, J.-M.: Where residual risks reside: a comparative approach to Art 9(4) of the European Union's Pro- posed AI Regulation. SSRN Electron. J. (2021). https:// doi. org/ 10. 2139/ SSRN. 39604 61 254. van Merwijk, C.: An AI defense-offense symmetry thesis. Less- Wrong. https:// www. lessw rong. com/ posts/ dPe87 urYGQ PA4gD Ep/ an-ai-defen se-offen se-symme try-thesis (2022). Accessed 20 Jan 2023 255. Du Sautoy, M.: The Creativity Code : Art and Innovation in the Age of A First US edition. Fourth Estate, Cambridge (2019) 256. · DOI
  183. Floridi, L., et al.: AI4People-an ethical framework for a good AI society: opportunities, risks, principles, and recommenda- tions. Minds Mach (Dordr) 28(4), 689-707 (2018). https:// doi. org/ 10. 1007/ s11023-018-9482-5 257. · DOI · Paper page
  184. Frey, C.B.: The Technology Trap : Capital, Labor, and Power in the Age of Automation. Princeton University Press, Princeton (2019) 258.
  185. Mökander, J., Floridi, L.: From algorithmic accountability to digital governance. Nat. Mach. Intell. (2022). https:// doi. org/ 10. 1038/ s42256-022-00504-5 259. · DOI
  186. Ziegler, D. M., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., et al.: Adversarial training for high-stakes reliability. Adv. Neural Inf. Process. Syst. 35, 9274-9286 (2022). arXiv: 2205. 01663 261.
  187. Keyes, O., Durbin, M., Hutson, J.: A mulching proposal: Analys- ing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In: Conference on Human Factors in Computing Systems-Proceedings, May 2019, https:// doi. org/ 10. 1145/ 32906 07. 33104 33 262 · DOI
  188. Mökander, J., Juneja, P., Watson, D.S., Floridi, L.: The US Algo- rithmic Accountability Act of 2022 vs. The EU Artificial Intel- ligence Act: what can they learn from each other? Minds Mach (Dordr) (2022). https:// doi. org/ 10. 1007/ s11023-022-09612-y 263. · DOI
  189. Epstein, Z., et al.: Turingbox: An experimental platform for the evaluation of AI systems. In: IJCAI International Joint Confer- ence on Artificial Intelligence, vol. 2018-July, pp. 5826-5828 (2018). https:// doi. org/ 10. 24963/ ijcai. 2018/ 851 264. EPRS: A Governance Framework for Algorithmic Accountabil- ity and Transparency. European Parliamentary Research Service (2019). https:// doi. org/ 10. 2861/ 59990 265. EIU.: Staying ahead of the curve-the business case for responsi- ble AI. The Economist Intelligence Unit. https:// www. eiu. com/n/ stayi ng-ahead-of-the-curve-the-busin ess-case-for-respo nsible-ai/, (2020). Accessed 7 Oct 2020 266. · DOI
  190. Mondal, S., Das, S., Vrana, V.G.: How to bell the cat? A theo- retical review of generative artificial intelligence towards digital disruption in all walks of life. Technologies 11(2), 44 (2023). https:// doi. org/ 10. 3390/ TECHN OLOGI ES110 20044 267. · DOI
  191. Muller, M., Chilton, L.B., Kantosalo, A., Lou Maher, M., Martin, C.P., Walsh, G.: GenAICHI: generative AI and HCI. In: Con- ference on Human Factors in Computing Systems-Proceedings, Apr. 2022, https:// doi. org/ 10. 1145/ 34911 01. 35037 19 268 · DOI · Paper page
  192. Salkind, N.J.: Encyclopedia of Research Design. SAGE, Los Angeles (2010) 270.
  193. Haas, P.J., Springer, J.F.: Applied policy research: concepts and cases. In: Garland Reference Library of Social Science ; v. 1051. New York: Garland Pub (1998) 271.
  194. Grant, M.J., Booth, A.: A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 26(2), 91-108 (2009). https:// doi. org/ 10. 1111/j. 1471- 1842. 2009. 00848.x 272. Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: EASE '14 (2014). https:// doi. org/ 10. 1145/ 26012 48. 26012 68. 273. Frey, B.B.: The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation, vol. 4. SAGE Publications, Incorporated, Thousand Oaks (2018) 274. · DOI · Paper page
  195. Adams, W.C.: Conducting semi-structured interviews. In: Hand- book of Practical Program Evaluation: Fourth Edition, pp. 492- 505. (2015). https:// doi. org/ 10. 1002/ 97811 19171 386. CH19. 275. Baldwin, R., Cave, M.: Understanding Regulation: Theory, Strat- egy, and Practice. Oxford University Press, Oxford (1999) 276. · DOI
  196. Vaswani, A., et al.: Attention is all you need. Adv Neural Inf Process Syst, vol. 2017-December, pp. 5999-6009, Jun. 2017. [Online]. https:// arxiv. org/ abs/ 1706. 03762 v5. Accessed 12 Apr 2023 277.
  197. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3), 211-252 (2015). https:// doi. org/ 10. 1007/ S11263-015-0816-Y/ FIGUR ES/ 16 279. · DOI
  198. Luccioni, A., Viviano, J.D.: What's in the Box? An analysis of undesirable content in the common crawl corpus. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pp. 182-189 (2021). https:// doi. org/ 10. 18653/ V1/ 2021. ACL-SHORT. 24. 280. Han, X., et al.: Pre-trained models: past, present and future. AI Open 2, 225-250 (2021). https:// doi. org/ 10. 1016/J. AIOPEN. 2021. 08. 002 281. European Commission.: Ethics guidelines for trustworthy AI. AI HLEG, pp. 2-36, [Online]. https:// ec. europa. eu/ futur ium/ en/ ai-allia nce-consu ltati on/ guide lines# Top (2019). Accessed 10 Feb 2023 282. · DOI · Paper page
  199. Korbak, T., Elsahar, H., Kruszewski, G., Dymetman, M.: On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. ArXiv, Jun. 2022. [Online]. https:// arxiv. org/ abs/ 2206. 00761 v2. Accessed 2 Apr 2023 283.
  200. Min, S., et al.: Rethinking the role of demonstrations: what makes in-context learning work?, In: Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 11048-11064. [Online]. https:// aclan tholo gy. org/ 2022. emnlp- main. 759 (2022). Accessed 2 Apr 2023 284.

200 of 210 shown

Cited by

Sorted by citations

Metadata

Authors
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L.
Affiliations
  • Princeton University (US)
  • Goethe University Frankfurt (DE)
  • University of Oxford (GB)
Corresponding author
Mökander, Jakob
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
4
Issue
4
Pages
1085-1115
ISSN
2730-5953 (ISSN-L)
DOI
10.48550/arxiv.2302.08500
arXiv
2302.08500
OpenAlex
W4381572755
Cited by
292
References
0
Citations by year
2026: 89 · 2025: 97 · 2024: 78 · 2023: 28
FWCI
73.12
Type
journal article
Language
en
Open access
Licence
Locations
  • AI and Ethics (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • AstraZeneca
  • Economic and Social Research Council
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
large language models, artificial intelligence in auditing, AI governance, ethical risks of AI, AI regulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review