CoChat
Sign Up
PreprintCited by 49Safety Systems Engineering in AutonomyEthics and Social Impacts of AIAdversarial Robustness in Machine Learning

Sociotechnical Safety Evaluation of Generative AI Systems

Open access
Weidinger, Laura, Rauh, Maribeth, Marchal, Nahema, Manzini, Arianna, Hendricks, Lisa Anne, Mateos-Garcia, Juan, Bergman, Stevie, Kay, Jackie +5 more·2023·arXiv
DOI 10.48550/arxiv.2310.11986arXiv 2310.11986
Open this paper in CoChat — free accountOpen PDF

Abstract

Generative AI systems produce a range of risks. To ensure the safety of generative AI systems, these risks must be evaluated. In this paper, we make two main contributions toward establishing such evaluations. First, we propose a three-layered framework that takes a structured, sociotechnical approach to evaluating these risks. This framework encompasses capability evaluations, which are the main current approach to safety evaluation. It then reaches further by building on system safety principles, particularly the insight that context determines whether a given capability may cause harm. To account for relevant context, our framework adds human interaction and systemic impacts as additional layers of evaluation. Second, we survey the current state of safety evaluation of generative AI systems and create a repository of existing evaluations. Three salient evaluation gaps emerge from this analysis. We propose ways forward to closing these gaps, outlining practical steps as well as roles and responsibilities for different actors. Sociotechnical safety evaluation is a tractable approach to the robust and comprehensive safety evaluation of generative AI systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.11986

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Epistemic injustices cannot be evaluated by traditional machine learning methods that measure a computer's behavior in isolation of human judgment.

    “they cannot be evaluated by traditional machine learning methods, which measure a computer's behavior in isolation of human judgment (Weidinger et al. 2023).”

    From Epistemic Injustice in Generative AI (2024) · cites this paper as Weidinger et al. 2023

    Section 5.4.2 says automated tests and benchmarks cannot capture the scope of complex, real-world harms, and Section 4.2.2 says evaluating effects on people requires human participants. Our copy does not specifically discuss epistemic injustice or establish that it cannot be evaluated by traditional machine-learning methods, so the claim is more categorical and specific than the source supports. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 309

  1. J. D. Abbey and M. G. Meloy. Attention by design: Using attention checks to detect inattentive respondents and improve data quality. Journal of Operations Management, 53-56:63-70, Nov. 2017. ISSN 0272-6963. doi: 10.1016/j.jom.2017.06.001. URL https://www.sciencedirect.com/ science/article/pii/S0272696317300402. · DOI · Paper page
  2. G. Abercrombie and V. Rieser. Risk-graded Safety for Handling Medical Queries in Conversational AI, Oct. 2022. URL http://arxiv.org/abs/2210.00572. arXiv:2210.00572 [cs. · arXiv
  3. D. Acemoglu and S. Johnson. Power and progress: our thousand-year struggle over technology and prosperity. PublicAffairs, New York, first edition edition, 2023. ISBN 9781541702530.
  4. D. Acemoglu and P. Restrepo. Robots and Jobs: Evidence from US Labor Markets. Journal of Political Economy, 128(6):2188-2244, June 2020. ISSN 0022-3808, 1537-534X. doi: 10.1086/705716. URL https://www.journals.uchicago.edu/doi/10.1086/705716. · DOI · Paper page
  5. A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank. MusicLM: Generating Music From Text, Jan. 2023. URL http://arxiv.org/abs/2301.11325. arXiv:2301.11325 [cs, eess]. · arXiv
  6. J. Allen, B. Howland, M. Mobius, D. Rothschild, and D. J. Watts. Evaluating the fake news problem at the scale of the information ecosystem. Science Advances, 6(14):eaay3539, Apr. 2020. ISSN 2375-2548. doi: 10.1126/sciadv.aay3539. URL https://www.science.org/doi/10.1126/ sciadv.aay3539. · DOI · Paper page
  7. R. Amoroso, D. Morelli, M. Cornia, L. Baraldi, A. Del Bimbo, and R. Cucchiara. Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images, Apr. 2023. URL http://arxiv.or g/abs/2304.00500. arXiv:2304.00500 [cs. · arXiv
  8. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Choo, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, G. Gur-Ari, S. Hand, H. Hashemi, L. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Ittycheriah, M. Jagielski, W. Jia, K. Kenealy, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, J. Li, H. Lim, H. Lin, Z. Liu, F. Liu, M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nado, J. Nham, E. Ni, A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov, R. Pope, S. Qiao, E. Reif, B. Richter, P. Riley, A. C. Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D. R. So, D. Sohn, S. Tokumine, D. Valter, V. Vasudevan, K. Vodrahalli, X. Wang, P. Wang, Z. Wang, T. Wang, J. Wieting, Y. Wu, K. Xu, Y. Xu, L. Xue, P. Yin, J. Yu, Q. Zhang, S. Zheng, C. Zheng, W. Zhou, D. Zhou, S. Petrov, and Y. Wu. PaLM 2 Technical Report, Sept. 2023. URL http://arxiv.org/abs/2305.10403. arXiv:2305.10403 [cs]. · arXiv
  9. L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3):337-351, July 2023. ISSN 1047-1987, 1476-4989. doi: 10.1017/pan.2023.2. URL https://www.cambridge.org/co re/journals/political-analysis/article/abs/out-of-one-many-using-languag e-models-to-simulate-human-samples/035D7C8A55B237942FB6DBAD7CAA4E49. · DOI · Paper page
  10. L. Aroyo, M. Diaz, C. Homan, V. Prabhakaran, A. Taylor, and D. Wang. The Reasonable Effectiveness of Diverse Evaluation Data, Jan. 2023a. URL http://arxiv.org/abs/2301.09406. arXiv:2301.09406 [cs. · arXiv
  11. L. Aroyo, A. S. Taylor, M. Diaz, C. M. Homan, A. Parrish, G. Serapio-Garcia, V. Prabhakaran, and D. Wang. DICES Dataset: Diversity in Conversational AI Evaluation for Safety, June 2023b. URL http://arxiv.org/abs/2306.11247. arXiv:2306.11247 [cs. · arXiv
  12. N. Ashraf, A. Rafiq, S. Butt, H. M. F. Shehzad, G. Sidorov, and A. Gelbukh. YouTube based religious hate speech and extremism detection dataset with machine learning baselines. Journal of Intelligent & Fuzzy Systems, 42(5):4769-4777, Jan. 2022. ISSN 1064-1246. doi: 10.3233/JIFS-219264. URL https://content.iospress.com/articles/journal-of-intelligent-and-fuzzy-s ystems/ifs219264. · DOI
  13. H. Bai, J. G. Voelkel, J. C. Eichstaedt, and R. Willer. Artificial Intelligence Can Persuade Humans on Political Issues. preprint, Open Science Framework, Feb. 2023a. URL https://osf.io/stakv.
  14. L. Bai, X. Liu, and J. Su. ChatGPT: The cognitive effects on learning and memory. Brain-X, 1 (3):e30, Sept. 2023b. ISSN 2835-3153, 2835-3153. doi: 10.1002/brx2.30. URL https: //onlinelibrary.wiley.com/doi/10.1002/brx2. · DOI
  15. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional AI: Harmlessness from AI Feedback, Dec. 2022. URL http://arxiv.or g/abs/2212.08073. arXiv:2212.08073 [cs]. · DOI
  16. C. Bakalar, R. Barreto, S. Bergman, M. Bogen, B. Chern, S. Corbett-Davies, M. Hall, I. Kloumann, M. Lam, J. Q. Candela, M. Raghavan, J. Simons, J. Tannen, E. Tong, K. Vredenburgh, and J. Zhao. Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems, Mar. 2021. URL http://arxiv.org/abs/2103.06172. arXiv:2103.06172 [cs. · arXiv
  17. J. Barnett. The Ethical Implications of Generative Audio Models: A Systematic Literature Review. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 146-161, Aug. 2023. doi: 10.1145/3600211.3604686. URL http://arxiv.org/abs/2307.05527. arXiv:2307.05527 [cs, eess]. · DOI
  18. S. Barocas and A. D. Selbst. Big data's disparate impact. California law review, pages 671-732, 2016.
  19. A. S. Bergman, L. A. Hendricks, M. Rauh, B. Wu, W. Agnew, M. Kunesch, I. Duan, I. Gabriel, and W. Isaac. Representation in AI Evaluations. In 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 519-533, Chicago IL USA, June 2023. ACM. ISBN 9798400701924. doi: 10.1145/3593013.3594019. URL https://dl.acm.org/doi/10.1145/3593013.3594019. · DOI
  20. F. Bianchi, P. Kalluri, E. Durmus, F. Ladhak, M. Cheng, D. Nozza, T. Hashimoto, D. Jurafsky, J. Zou, and A. Caliskan. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale. In 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1493-1504, June 2023. doi: 10.1145/3593013.3594095. URL http://arxiv.org/abs/2211.03759. arXiv:2211.03759 [cs. · DOI · Paper page
  21. M. Binz and E. Schulz. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, Feb. 2023. ISSN 0027-8424, 1091-6490. doi: 10.107 3/pnas.2218523120. URL https://pnas.org/doi/10.1073/pnas.2218523120. · DOI · Paper page
  22. C. Bird, E. L. Ungless, and A. Kasirzadeh. Typology of Risks of Generative Text-to-Image Models, July 2023. URL http://arxiv.org/abs/2307.05543. arXiv:2307.05543 [cs. · arXiv
  23. A. Birhane, V. U. Prabhu, and E. Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes, Oct. 2021. URL http://arxiv.org/abs/2110.01963. arXiv:2110.01963 [cs. · arXiv
  24. A. Birhane, W. Isaac, V. Prabhakaran, M. Diaz, M. C. Elish, I. Gabriel, and S. Mohamed. Power to the People? Opportunities and Challenges for Participatory AI. In Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO '22, pages 1-8, New York, NY, USA, Oct. 2022a. Association for Computing Machinery. ISBN 9781450394772. doi: 10.1145/3551624.3555290. URL https://dl.acm.org/doi/10.1145/3551624.3555290. · DOI · Paper page
  25. A. Birhane, P. Kalluri, D. Card, W. Agnew, R. Dotan, and M. Bao. The Values Encoded in Machine Learning Research. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 173-184, Seoul Republic of Korea, June 2022b. ACM. ISBN 9781450393522. doi: 10.1145/3531 146.3533083. URL https://dl.acm.org/doi/10.1145/3531146.3533083. · DOI · Paper page
  26. S. L. Blodgett, G. Lopez, A. Olteanu, R. Sim, and H. Wallach. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004-1015, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.81. URL https: //aclanthology.org/2021.acl-long. · DOI · Paper page
  27. T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama, and A. Kalai. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings, July 2016. URL http://arxiv.org/ abs/1607.06520. arXiv:1607.06520 [cs, stat]. · arXiv
  28. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang. On the Opportunities and Risks of Foundation Models, July 2022. URL http://arxiv.org/abs/2108.07258. arXiv:2108.07258 [cs. · arXiv
  29. R. Bommasani, D. Soylu, T. I. Liao, K. A. Creel, and P. Liang. Ecosystem Graphs: The Social Footprint of Foundation Models. 2023. doi: 10.48550/ARXIV.2303.15772. URL https: //arxiv.org/abs/2303.15772. · DOI
  30. A. Bonfiglioli, R. Crinò, H. Fadinger, and G. Gancia. Robot Imports and Firm-Level Outcomes, Apr. 2020. URL https://papers.ssrn.com/abstract=3594215.
  31. Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour. AudioLM: A Language Modeling Approach to Audio Generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523-2533, 2023. doi: 10.1109/TASLP.2023.3288409. URL https://ieeexplore.ieee.org/abstract /document/10158503. · DOI · Paper page
  32. G. C. Bowker and S. L. Star. Sorting Things Out: Classification and Its Consequences. MIT Press, Aug. 2000. ISBN 9780262261609. Google-Books-ID: xHlP8WqzizYC.
  33. M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong, T. Maharaj, P. W. Koh, S. Hooker, J. Leung, A. Trask, E. Bluemke, J. Lebensold, C. O'Keefe, M. Koren, T. Ryffel, J. B. Rubinovitz, T. Besiroglu, F. Carugati, J. Clark, P. Eckersley, S. de Haas, M. Johnson, B. Laurie, A. Ingerman, I. Krawczuk, A. Askell, R. Cammarota, A. Lohn, D. Krueger, C. Stix, P. Henderson, L. Graham, C. Prunkl, B. Martin, E. Seger, N. Zilberman, S. O. Heigeartaigh, F. Kroeger, G. Sastry, R. Kagan, A. Weller, B. Tse, E. Barnes, A. Dafoe, P. Scharre, A. Herbert-Voss, M. Rasser, S. Sodhani, C. Flynn, T. K. Gilbert, L. Dyer, S. Khan, Y. Bengio, and M. Anderljung. Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims, Apr. 2020. URL http://arxiv.org/abs/2004.07213. arXiv:2004.07213 [cs. · arXiv
  34. F. Brunton and H. Nissenbaum. Obfuscation: a user's guide for privacy and protest. MIT Press, Cambridge, Mass. London, 2016. ISBN 9780262529860.
  35. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4, Apr. 2023. URL http://arxiv.org/abs/2303.12712. arXiv:2303.12712 [cs. Bulimia Project. Social Media's "Ideal" Body, According to AI. URL https://bulimia.com/exam ine/scrolling-into-bias/. · arXiv
  36. J. Buolamwini and T. Gebru. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, pages 77-91. PMLR, Jan. 2018. URL https://proceedings.mlr.press/v81/buolamwini 18a.html.
  37. R. Burnell, W. Schellaert, J. Burden, T. D. Ullman, F. Martinez-Plumed, J. B. Tenenbaum, D. Rutar, L. G. Cheke, J. Sohl-Dickstein, M. Mitchell, D. Kiela, M. Shanahan, E. M. Voorhees, A. G. Cohn, J. Z. Leibo, and J. Hernandez-Orallo. Rethink reporting of evaluation results in AI. Science, 380 (6641):136-138, Apr. 2023. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.adf6369. URL https://www.science.org/doi/10.1126/science.adf6369. · DOI · Paper page
  38. M. Burtell and T. Woodside. Artificial influence: An analysis of ai-driven persuasion. arXiv preprint arXiv:2303.08721, 2023. · arXiv
  39. J. T. Cacioppo and R. E. Petty. Persuasiveness of Communications is Affected by Exposure Frequency and Message Quality: A Theoretical and Empirical Analysis of Persisting Attitude Change. Current Issues and Research in Advertising, 3(1):97-122, Mar. 1980. ISSN 0163-3392, 2165-820X. doi: 10.1080/01633392.1980.10505295. URL https://www.tandfonline.com/doi/full/10. 1080/01633392.1980.10505295. · DOI
  40. A. Caliskan, J. J. Bryson, and A. Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186, Apr. 2017. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.aal4230. URL http://arxiv.org/abs/1608.07187. arXiv:1608.07187 · DOI · Paper page
  41. N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramèr, B. Balle, D. Ippolito, and E. Wallace. Extracting Training Data from Diffusion Models. pages 5253-5270, 2023a. ISBN 9781939133373. URL https://www.usenix.org/conference/usenixsecurity23/presentation/carl ini.
  42. N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, A. Awadalla, P. W. Koh, D. Ippolito, K. Lee, F. Tramer, and L. Schmidt. Are aligned neural networks adversarially aligned?, June 2023b. URL http://arxiv.org/abs/2306.15447. arXiv:2306.15447 [cs. · arXiv
  43. S. Casper, J. Lin, J. Kwon, G. Culp, and D. Hadfield-Menell. Explore, Establish, Exploit: Red Teaming Language Models from Scratch, June 2023. URL http://arxiv.org/abs/2306.09442. arXiv:2306.09442 [cs. Center for Countering Digital Hate. Ai and eating disorders: How generative ai enables and promotes harmful eating disorder content, 2023a. URL https://counterhate.com/research/ai-too ls-and-eating-disorders/. · arXiv
  44. Chan, M. Riché, and J. Clifton. Towards the Scalable Evaluation of Cooperativeness in Language Models, Mar. 2023a. URL https://arxiv.org/abs/2303.13360v1. · arXiv
  45. A. Chan, R. Salganik, A. Markelius, C. Pang, N. Rajkumar, D. Krasheninnikov, L. Langosco, Z. He, Y. Duan, M. Carroll, M. Lin, A. Mayhew, K. Collins, M. Molamohammadi, J. Burden, W. Zhao, S. Rismani, K. Voudouris, U. Bhatt, A. Weller, D. Krueger, and T. Maharaj. Harms from Increasingly Agentic Algorithmic Systems. In 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 651-666, Chicago IL USA, June 2023b. ACM. ISBN 9798400701924. doi: 10.1145/359301 3.3594033. URL https://dl.acm.org/doi/10.1145/3593013.3594033. · DOI · Paper page
  46. H. Chang. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press, Aug. 2004. ISBN 9780198038245. Google-Books-ID: yVOuV8qJkxMC.
  47. L. Chen, M. Zaharia, and J. Zou. How is ChatGPT's behavior changing over time?, Aug. 2023. URL http://arxiv.org/abs/2307.09009. arXiv:2307.09009 [cs. · arXiv
  48. S. Chiesurin, D. Dimakopoulos, M. A. Sobrevilla Cabezudo, A. Eshghi, I. Papaioannou, V. Rieser, and I. Konstas. The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 947-959, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.60. URL https://aclanthology.org/2023.findings -acl. · DOI
  49. J. Cho, A. Zala, and M. Bansal. DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models, Aug. 2023. URL http://arxiv.org/abs/2202.04053. arXiv:2202.04053 [cs. · arXiv
  50. D. Choi, Y. Shavit, and D. Duvenaud. Tools for Verifying Neural Models' Training Data, July 2023. URL http://arxiv.org/abs/2307.00682. arXiv:2307.00682 [cs. · arXiv
  51. S. Choi, N. Kim, J. Kim, and H. Kang. How Does AI Improve Human Decision-Making? Evidence from the AI-Powered Go Program, Apr. 2022. URL https://papers.ssrn.com/abstract=3893835.
  52. D. Citron and R. Chesney. Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security. California Law Review, 107(6):1753, Dec. 2019. URL https://scholarship.law.bu .edu/faculty_scholarship/640.
  53. E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith. All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.565. URL https://aclanthology .org/2021.acl-long.565. · DOI · Paper page
  54. R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi, K. Nagano, and L. Verdoliva. On the detection of synthetic images generated by diffusion models, Nov. 2022. URL http://arxiv.org/abs/2211 .00680. arXiv:2211.00680 [cs. · arXiv
  55. S. Costanza-Chock. Design justice: community-led practices to build the worlds we need. Information policy. MIT Press, Cambridge, Massachusetts, 2020. ISBN 9780262043458.
  56. S. Costanza-Chock, I. D. Raji, and J. Buolamwini. Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, pages 1571-1583, New York, NY, USA, June 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1145/3531146.3533 213. URL https://doi.org/10.1145/3531146.3533213. · DOI · Paper page
  57. M. J. Crockett. Moral outrage in the digital age. Nature Human Behaviour, 1(11):769-771, Nov. 2017. ISSN 2397-3374. doi: 10.1038/s41562-017-0213-3. URL https://www.nature.com/artic les/s41562-017-0213-3. · DOI · Paper page
  58. M. Crosby, B. Beyret, and M. Halina. The Animal-AI Olympics. Nature Machine Intelligence, 1 (5):257-257, May 2019. ISSN 2522-5839. doi: 10.1038/s42256-019-0050-3. URL https: //www.nature.com/articles/s42256-019-0050-3. · DOI
  59. M. Das, R. Raj, P. Saha, B. Mathew, M. Gupta, and A. Mukherjee. HateMM: A Multi-Modal Dataset for Hate Video Classification. Proceedings of the International AAAI Conference on Web and Social Media, 17:1014-1023, June 2023. ISSN 2334-0770. doi: 10.1609/icwsm.v17i1.22209. URL https://ojs.aaai.org/index.php/ICWSM/article/view/22209. · DOI
  60. H. de Vries, D. Bahdanau, and C. Manning. Towards Ecologically Valid Research on Language User Interfaces, July 2020. URL http://arxiv.org/abs/2007.14435. arXiv:2007.14435 [cs. · arXiv
  61. A. de Wynter, X. Wang, A. Sokolov, Q. Gu, and S.-Q. Chen. An Evaluation on Large Language Model Outputs: Discourse and Memorization. Natural Language Processing Journal, 4:100024, Sept. 2023. ISSN 29497191. doi: 10.1016/j.nlp.2023.100024. URL http://arxiv.org/abs/2304.08637. arXiv:2304.08637 [cs. · DOI
  62. M. Dehghani, Y. Tay, A. A. Gritsenko, Z. Zhao, N. Houlsby, F. Diaz, D. Metzler, and O. Vinyals. The Benchmark Lottery, July 2021. URL http://arxiv.org/abs/2107.07002. arXiv:2107.07002 · arXiv
  63. M. Dehnert and P. A. Mongeau. Persuasion in the Age of Artificial Intelligence (AI): Theories and Complications of AI-Based Persuasion. Human Communication Research, 48(3):386-403, June 2022. ISSN 0360-3989, 1468-2958. doi: 10.1093/hcr/hqac006. URL https://academic.oup .com/hcr/article/48/3/386/6564679. · DOI · Paper page
  64. M. del Rio-Chanona, N. Laurentsyeva, and J. Wachs. Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow, July 2023. URL http://arxiv.org/ abs/2307.07367. arXiv:2307.07367 [cs]. · arXiv
  65. E. Denton, A. Hanna, R. Amironesei, A. Smart, and H. Nicole. On the genealogy of machine learning datasets: A critical history of ImageNet. Big Data & Society, 8(2):205395172110359, July 2021. ISSN 2053-9517, 2053-9517. doi: 10.1177/20539517211035955. URL http://journals.sag epub.com/doi/10.1177/20539517211035955. · DOI · Paper page
  66. M. R. DeVerna, H. Y. Yan, K.-C. Yang, and F. Menczer. Artificial intelligence is ineffective and potentially harmful for fact checking, Sept. 2023. URL http://arxiv.org/abs/2308.10800. arXiv:2308.10800 [cs. · arXiv
  67. P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever. Jukebox: A Generative Model for Music, Apr. 2020. URL http://arxiv.org/abs/2005.00341. arXiv:2005.00341 [cs, eess, stat]. · arXiv
  68. F. Diaz and M. Madaio. Scaling Laws Do Not Scale, July 2023. URL http://arxiv.org/abs/23 07.03201. arXiv:2307.03201 [cond-mat]. · arXiv
  69. M. Diaz, R. Amironesei, L. Weidinger, and I. Gabriel. Accounting for Offensive Speech as a Practice of Resistance. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 192-202, Seattle, Washington (Hybrid), 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.woah-1.18. URL https://aclanthology.org/2022.woah-1. · DOI
  70. V. Dignum. Responsible Artificial Intelligence: How to Develop and Use AI in a Responsible Way. Artificial Intelligence: Foundations, Theory, and Algorithms. Springer International Publishing, Cham, 2019. ISBN 9783030303709 9783030303716. doi: 10.1007/978-3-030-30371-6. URL http: //link.springer.com/10.1007/978-3-030-30371-6. · DOI · Paper page
  71. D. Dillion, N. Tandon, Y. Gu, and K. Gray. Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7):597-600, July 2023. ISSN 1364-6613. doi: 10.1016/j.tics.2023.04.008. URL https://www.sciencedirect.com/science/article/pii/S1364661323000980. · DOI · Paper page
  72. E. Dinan, G. Abercrombie, A. S. Bergman, S. Spruit, D. Hovy, Y.-L. Boureau, and V. Rieser. Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling, July 2021. URL http://arxiv. org/abs/2107.03451. arXiv:2107.03451 [cs]. · arXiv
  73. E. Dinan, G. Abercrombie, A. Bergman, S. Spruit, D. Hovy, Y.-L. Boureau, and V. Rieser. SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4113-4133, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.284. URL https://aclanthology.org/2022.acl-long.284. · DOI · Paper page
  74. Y. Ding, J. You, T.-K. Machulla, J. Jacobs, P. Sen, and T. Höllerer. Impact of Annotator Demographics on Sentiment Dataset Labeling. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2): 519:1-519:22, Nov. 2022. doi: 10.1145/3555632. URL https://doi.org/10.1145/3555632. · DOI
  75. L. Dixon, J. Li, J. Sorensen, N. Thain, and L. Vasserman. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67-73, New Orleans LA USA, Dec. 2018. ACM. ISBN 9781450360128. doi: 10.1145/3278 721.3278729. URL https://dl.acm.org/doi/10.1145/3278721.3278729. · DOI · Paper page
  76. J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, Sept. 2021. URL http://arxiv.org/abs/2104.08758. arXiv:2104.08758 [cs. · arXiv
  77. A. R. Doshi and O. Hauser. Generative Artificial Intelligence Enhances Creativity but Reduces the Diversity of Novel Content, Aug. 2023. URL https://papers.ssrn.com/abstract=4535536.
  78. Y. Du, Z. Chen, J. Salamon, B. Russell, and A. Owens. Conditional Generation of Audio from Video via Foley Analogies, Apr. 2023. URL http://arxiv.org/abs/2304.08490. arXiv:2304.08490 [cs, eess]. · arXiv
  79. C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
  80. U. K. H. Ecker, S. Lewandowsky, J. Cook, P. Schmid, L. K. Fazio, N. Brashier, P. Kendeou, E. K. Vraga, and M. A. Amazeen. The psychological drivers of misinformation belief and its resistance to correction. Nature Reviews Psychology, 1(1):13-29, Jan. 2022. ISSN 2731-0574. doi: 10.1038/s4 4159-021-00006-y. URL https://www.nature.com/articles/s44159-021-00006-y. · DOI · Paper page
  81. M. Eiband, S. T. Völkel, D. Buschek, S. Cook, and H. Hussmann. When people and algorithms meet: user-reported problems in intelligent everyday applications. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI '19, pages 96-106, New York, NY, USA, Mar. 2019. Association for Computing Machinery. ISBN 9781450362726. doi: 10.1145/3301275.3302262. URL https://doi.org/10.1145/3301275.3302262. · DOI · Paper page
  82. M. D. Ekstrand, M. Tian, I. M. Azpiazu, J. D. Ekstrand, O. Anuyah, D. McNeill, and M. S. Pera. All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness. In Conference on fairness, accountability and transparency, pages 172-186.
  83. PMLR, 2018.
  84. B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang. CLAP: Learning Audio Concepts From Natural Language Supervision, June 2022. URL http://arxiv.org/abs/2206.04769. arXiv:2206.04769 [cs, eess]. · arXiv
  85. T. Eloundou, S. Manning, P. Mishkin, and D. Rock. GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models, Aug. 2023. URL http://arxiv.org/abs/2303.1 0130. arXiv:2303.10130 [cs, econ, q-fin]. · arXiv
  86. Z. Epstein, A. Hertzmann, the Investigators of Human Creativity, M. Akten, H. Farid, J. Fjeld, M. R. Frank, M. Groh, L. Herman, N. Leach, R. Mahari, S. Pentland, O. Russakovsky, H. Schroeder, and A. Smith. Art and the science of generative AI. Science, 380(6650):1110-1111, June 2023. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.adh4451. URL https://www.science.org/do i/10.1126/science.adh4451. EU AI Act. Regulatory framework proposal on artificial intelligence: Shaping Europe's digital future, Sept. 2023. URL https://digital-strategy.ec.europa.eu/en/policies/regulator y-framework-ai. European Commission. Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts, 2021. URL https://eur-lex.europa.eu/legal-content/E N/TXT/?uri=CELLAR:e0649735-a372-11eb-9585-01aa75ed71a1. · DOI · Paper page
  87. O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders. Truthful AI: Developing and governing AI that does not lie, Oct. 2021. URL http://arxiv.org/ abs/2110.06674. arXiv:2110.06674 [cs]. · arXiv
  88. E. Felten, M. Raj, and R. Seamans. Occupational, industry, and geographic exposure to artificial intelligence: A novel dataset and its potential uses. Strategic Management Journal, 42(12): 2195-2217, Dec. 2021. ISSN 0143-2095, 1097-0266. doi: 10.1002/smj.3286. URL https://onlinelibrary.wiley.com/doi/10.1002/smj.3286. · DOI · Paper page
  89. A. Field, S. L. Blodgett, Z. Waseem, and Y. Tsvetkov. A Survey of Race, Racism, and Anti-Racism in NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1905-1925, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.149. URL https://aclanthology.org/2021.acl-long.149. · DOI
  90. M. C. Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451-452, Aug. 2023. ISSN 2731-0574. doi: 10.1038/s44159-023-00211-x. URL https://www.nature.com/articles/s44159-023-00211-x. · DOI
  91. M. R. Frank, D. Autor, J. E. Bessen, E. Brynjolfsson, M. Cebrian, D. J. Deming, M. Feldman, M. Groh, J. Lobo, E. Moro, D. Wang, H. Youn, and I. Rahwan. Toward understanding the impact of artificial intelligence on labor. Proceedings of the National Academy of Sciences, 116(14):6531-6539, Apr. 2019. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1900949116. URL https://pnas.org /doi/full/10.1073/pnas.1900949116. · DOI · Paper page
  92. C. B. Frey and M. A. Osborne. The future of employment: How susceptible are jobs to computerisation? Technological Forecasting and Social Change, 114:254-280, Jan. 2017. ISSN 0040-1625. doi: 10.1016/j.techfore.2016.08.019. URL https://www.sciencedirect.com/science/articl e/pii/S0040162516302244. · DOI · Paper page
  93. B. Friedman and H. Nissenbaum. Bias in computer systems. ACM Transactions on information systems (TOIS), 14(3):330-347, 1996.
  94. D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, A. Jones, S. Bowman, A. Chen, T. Conerly, N. DasSarma, D. Drain, N. Elhage, S. El-Showk, S. Fort, Z. Hatfield-Dodds, T. Henighan, D. Hernandez, T. Hume, J. Jacobson, S. Johnston, S. Kravec, C. Olsson, S. Ringer, E. Tran-Johnson, D. Amodei, T. Brown, N. Joseph, S. McCandlish, C. Olah, J. Kaplan, and J. Clark. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned, Nov. 2022. URL http://arxiv.org/abs/2209.07858. arXiv:2209.07858 [cs. · arXiv
  95. L. Gao, Z. Dai, P. Pasupat, A. Chen, A. T. Chaganty, Y. Fan, V. Y. Zhao, N. Lao, H. Lee, D.-C. Juan, and K. Guu. RARR: Researching and Revising What Language Models Say, Using Language Models, May 2023. URL http://arxiv.org/abs/2210.08726. arXiv:2210.08726 [cs. · arXiv
  96. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models, Sept. 2020. URL http://arxiv.org/abs/2009.114 62. arXiv:2009.11462 [cs. · arXiv
  97. S. Gehrmann, T. Adewumi, K. Aggarwal, P. S. Ammanamanchi, A. Aremu, A. Bosselut, K. R. Chandu, M.-A. Clinciu, D. Das, K. Dhole, W. Du, E. Durmus, O. Dušek, C. C. Emezue, V. Gangal, C. Garbacea, T. Hashimoto, Y. Hou, Y. Jernite, H. Jhamtani, Y. Ji, S. Jolly, M. Kale, D. Kumar, F. Ladhak, A. Madaan, M. Maddela, K. Mahajan, S. Mahamood, B. P. Majumder, P. H. Martins, A. McMillan-Major, S. Mille, E. van Miltenburg, M. Nadeem, S. Narayan, V. Nikolaev, A. Niyongabo Rubungo, S. Osei, A. Parikh, L. Perez-Beltrachini, N. R. Rao, V. Raunak, J. D. Rodriguez, S. Santhanam, J. Sedoc, T. Sellam, S. Shaikh, A. Shimorina, M. A. Sobrevilla Cabezudo, H. Strobelt, N. Subramani, W. Xu, D. Yang, A. Yerukola, and J. Zhou. The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.gem-1.10. URL https://aclanthology.org/2021.gem-1. · DOI · Paper page
  98. A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P.-S. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri, R. Greig, C. Chen, D. Fritz, J. S. Elias, R. Green, S. Mokrá, N. Fernando, B. Wu, R. Foley, S. Young, I. Gabriel, W. Isaac, J. Mellor, D. Hassabis, K. Kavukcuoglu, L. A. Hendricks, and G. Irving. Improving alignment of dialogue agents via targeted human judgements, Sept. 2022. URL http://arxiv.org/abs/2209.14375. arXiv:2209.14375 [cs. · arXiv
  99. E. Glikson and A. W. Woolley. Human Trust in Artificial Intelligence: Review of Empirical Research. Academy of Management Annals, 14(2):627-660, July 2020. ISSN 1941-6520, 1941-6067. doi: 10.5465/annals.2018.0057. URL http://journals.aom.org/doi/10.5465/annals.2018 .0057. · DOI · Paper page
  100. S. Goldfarb-Tarrant, R. Marchant, R. Muñoz Sánchez, M. Pandya, and A. Lopez. Intrinsic Bias Metrics Do Not Correlate with Application Bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1926-1940, 2021. URL https://aclanthology.org/2021.acl-long.15
  101. pdf. J. A. Goldstein, J. Chao, S. Grossman, A. Stamos, and M. Tomz. Can AI Write Persuasive Propaganda? preprint, SocArXiv, Apr. 2023. URL https://osf.io/fp87b.
  102. H. Gonen and Y. Goldberg. Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them, Sept. 2019. URL http://arxiv.org/abs/19 03.03862. arXiv:1903.03862 [cs. · arXiv
  103. M. L. Gordon, K. Zhou, K. Patel, T. Hashimoto, and M. S. Bernstein. The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, pages 1-14, New York, NY, USA, May 2021. Association for Computing Machinery. ISBN 9781450380966. doi: 10.1145/3411764. 3445423. URL https://doi.org/10.1145/3411764.3445423. · DOI · Paper page
  104. N. Goyal, I. D. Kivlichan, R. Rosen, and L. Vasserman. Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):363:1-363:28, Nov. 2022. doi: 10.1145/3555088. URL https://dl.acm .org/doi/10.1145/3555088. · DOI · Paper page
  105. S. Gregory. Fortify the Truth: How to Defend Human Rights in an Age of Deepfakes and Generative AI. Journal of Human Rights Practice, 15(2), 2023. URL https://academic.oup.com/jhrp/ advance-article-abstract/doi/10.1093/jhuman/huad035/7261649?redirectedF rom=fulltext. · DOI
  106. M. Groh, Z. Epstein, C. Firestone, and R. Picard. Deepfake detection by human crowds, machines, and machine-informed crowds. Proceedings of the National Academy of Sciences, 119(1):e2110013119, Jan. 2022. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.2110013119. URL https: //pnas.org/doi/full/10.1073/pnas.2110013119. · DOI · Paper page
  107. S. Gururangan, D. Card, S. Dreier, E. Gade, L. Wang, Z. Wang, L. Zettlemoyer, and N. A. Smith. Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2562-2580, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.165. URL https://aclanthology.org/2022.emnlp-m ain.165. · DOI
  108. M. Hameleers, T. E. Powell, T. G. Van Der Meer, and L. Bos. A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated via Social Media. Political Communication, 37(2):281-301, Mar. 2020. ISSN 1058-4609, 1091-7675. doi: 10.1080/10584609.2019.1674979. URL https://www.tandfonline.com/doi/full/10.10 80/10584609.2019.1674979. · DOI · Paper page
  109. J. Han, M. Cha, and W. Lee. Anger contributes to the spread of COVID-19 misinformation. Harvard Kennedy School Misinformation Review, 1(3), Sept. 2020. doi: 10.37016/mr-2020-39. URL https://misinforeview.hks.harvard.edu/article/anger-contributes-to-the-s pread-of-covid-19-misinformation/. · DOI
  110. A. Hanna, E. Denton, A. Smart, and J. Smith-Loud. Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, pages 501-512, New York, NY, USA, Jan. 2020. Association for Computing Machinery. ISBN 9781450369367. doi: 10.1145/3351095.3372826. URL https://dl.acm.org/doi/10.1145 /3351095.3372826. · DOI · Paper page
  111. J. Henrich, S. J. Heine, and A. Norenzayan. The Weirdest People in the World?, May 2010. URL https://papers.ssrn.com/abstract=1601785.
  112. J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. CLIPScore: A Reference-free Evaluation Metric for Image Captioning, Mar. 2022. URL http://arxiv.org/abs/2104.08718. arXiv:2104.08718 [cs. · arXiv
  113. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, Jan. 2018. URL http://arxiv. org/abs/1706.08500. arXiv:1706.08500 [cs, stat]. · arXiv
  114. J. Hohenstein, R. F. Kizilcec, D. DiFranzo, Z. Aghajari, H. Mieczkowski, K. Levy, M. Naaman, J. Hancock, and M. F. Jung. Artificial intelligence in communication impacts language and social relationships. Scientific Reports, 13(1):5487, Apr. 2023. ISSN 2045-2322. doi: 10.1038/s41598-023-30938-9. URL https://www.nature.com/articles/s41598-023-30938-9. · DOI
  115. C. M. Homan, G. Serapio-Garcia, L. Aroyo, M. Diaz, A. Parrish, V. Prabhakaran, A. S. Taylor, and D. Wang. Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety, June 2023. URL http://arxiv.org/abs/2306.115 30. arXiv:2306.11530 [cs. · arXiv
  116. T. Hosking, P. Blunsom, and M. Bartolo. Human Feedback is not Gold Standard, Sept. 2023. URL https://arxiv.org/abs/2309.16349v1. · arXiv
  117. P.-S. Huang, H. Zhang, R. Jiang, R. Stanforth, J. Welbl, J. Rae, V. Maini, D. Yogatama, and P. Kohli. Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 65-83, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.7. URL https://aclant hology.org/2020.findings-emnlp. · DOI · Paper page
  118. Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank, J. Engel, Q. V. Le, W. Chan, Z. Chen, and W. Han. Noise2Music: Text-conditioned Music Generation with Diffusion Models, Mar. 2023. URL http://arxiv.org/abs/2302.03917. arXiv:2302.03917 [cs, eess]. · arXiv
  119. W. Hunt, S. Sarkar, and C. Warhurst. Measuring the impact of AI on jobs at the organization level: Lessons from a survey of UK business leaders. Research Policy, 51(2):104425, Mar. 2022. ISSN 0048-7333. doi: 10.1016/j.respol.2021.104425. URL https://www.sciencedirect.com/sc ience/article/pii/S0048733321002183. · DOI
  120. E. Hutchins. How a Cockpit Remembers Its Speeds. Cognitive Science, 19(3):265-288, July 1995. ISSN 03640213. doi: 10.1207/s15516709cog1903_1. URL http://doi.wiley.com/10.120 7/s15516709cog1903_1. · DOI · Paper page
  121. B. Hutchinson, J. Baldridge, and V. Prabhakaran. Underspecification in Scene Description-to-Depiction Tasks, Oct. 2022. URL http://arxiv.org/abs/2210.05815. arXiv:2210.05815 [cs. · arXiv
  122. M. Jackman and L. Kanerva. Evolving the IRB: Building Robust Review for Industry Research. Washington and Lee Law Review Online, 72(3):442, June 2016. URL https://scholarlycommo ns.law.wlu.edu/wlulr-online/vol72/iss3/8.
  123. A. Z. Jacobs and H. Wallach. Measurement and Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pages 375-385, New York, NY, USA, Mar. 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445901. URL https://dl.acm.org/doi/10.1145/3442188.3445901. · DOI · Paper page
  124. M. Jakesch, A. Bhat, D. Buschek, L. Zalmanson, and M. Naaman. Co-Writing with Opinionated Language Models Affects Users' Views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, pages 1-15, New York, NY, USA, Apr. 2023. Association for Computing Machinery. ISBN 9781450394215. doi: 10.1145/3544548.3581196. URL https://dl.acm.org/doi/10.1145/3544548.3581196. · DOI · Paper page
  125. J. Jerit and Y. Zhao. Political Misinformation. Annual Review of Political Science, 23(1):77-94, May 2020. ISSN 1094-2939, 1545-1577. doi: 10.1146/annurev-polisci-050718-032814. URL https://www.annualreviews.org/doi/10.1146/annurev-polisci-050718-032814. · DOI · Paper page
  126. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, W. Dai, A. Madotto, and P. Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1-38, Dec. 2023. ISSN 0360-0300, 1557-7341. doi: 10.1145/3571730. URL http://arxiv.org/abs/22 02.03629. arXiv:2202.03629 [cs. · DOI · Paper page
  127. C. Jia, A. Boltz, A. Zhang, A. Chen, and M. K. Lee. Understanding Effects of Algorithmic vs. Community Label on Perceived Accuracy of Hyper-partisan Misinformation. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):371:1-371:27, Nov. 2022. doi: 10.1145/3555096. URL https://dl.acm.org/doi/10.1145/3555096. · DOI
  128. N. Jia, X. Luo, Z. Fang, and C. Liao. When and How Artificial Intelligence Augments Employee Creativity, Mar. 2023. URL https://papers.ssrn.com/abstract=4397280.
  129. Y. Jin, E. Jang, J. Cui, J.-W. Chung, Y. Lee, and S. Shin. DarkBERT: A Language Model for the Dark Side of the Internet, May 2023. URL http://arxiv.org/abs/2305.08596. arXiv:2305.08596 · arXiv
  130. J. Jouhki, E. Lauk, M. Penttinen, N. Sormanen, and T. Uskali. Facebook's Emotional Contagion Experiment as a Challenge to Research Ethics. Media and Communication, 4(4):75-85, Oct. 2016. ISSN 2183-2439. doi: 10.17645/mac.v4i4.579. URL https://www.cogitatiopress.com/m ediaandcommunication/article/view/579. · DOI
  131. L. H. Kaack, P. L. Donti, E. Strubell, G. Kamiya, F. Creutzig, and D. Rolnick. Aligning artificial intelligence with climate change mitigation. Nature Climate Change, 12(6):518-527, June 2022. ISSN 1758-6798. doi: 10.1038/s41558-022-01377-7. URL https://www.nature.com/artic les/s41558-022-01377-7. · DOI · Paper page
  132. P. Kalluri. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature, 583(7815): 169-169, July 2020. doi: 10.1038/d41586-020-02003-2. URL https://www.nature.com/art icles/d41586-020-02003-2. · DOI · Paper page
  133. A. Kasirzadeh and I. Gabriel. In Conversation with Artificial Intelligence: Aligning language Models with Human Values. Philosophy & Technology, 36(2):27, Apr. 2023. ISSN 2210-5441. doi: 10.1007/s13347-023-00606-x. URL https://doi.org/10.1007/s13347-023-00606-x. · DOI · Paper page
  134. Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. Alignment of language agents. arXiv preprint arXiv:2103.14659, 2021. · arXiv
  135. H. Khalid, S. Tariq, M. Kim, and S. S. Woo. FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset, Mar. 2022. URL http://arxiv.org/abs/2108.05080. arXiv:2108.05080 [cs, eess]. · arXiv
  136. H. Khlaaf, P. Mishkin, J. Achiam, G. Krueger, and M. Brundage. A Hazard Analysis Framework for Code Synthesis Large Language Models, July 2022. URL http://arxiv.org/abs/2207.14157. arXiv:2207.14157 [cs. · arXiv
  137. D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking Benchmarking in NLP, Apr. 2021a. URL http://arxiv.org/abs/2104 .14337. arXiv:2104.14337 [cs. · arXiv
  138. D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes, Apr. 2021b. URL http://arxi v.org/abs/2005.04790. arXiv:2005.04790 [cs. · arXiv
  139. G. King, R. O. Keohane, and S. Verba. Designing Social Inquiry: Scientific Inference in Qualitative Research, New Edition. Princeton University Press, Princeton, 2021.
  140. J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein. A Watermark for Large Language Models, June 2023. URL http://arxiv.org/abs/2301.10226. arXiv:2301.10226 · arXiv
  141. J. F. Kiviet. Testing the impossible: Identifying exclusion restrictions. Journal of Econometrics, 218(2):294-316, Oct. 2020. ISSN 0304-4076. doi: 10.1016/j.jeconom.2020.04.018. URL https://www.sciencedirect.com/science/article/pii/S030440762030138X. · DOI
  142. G. Klein, M. Jalaeian, R. R. Hoffman, and S. T. Mueller. The plausibility transition model for sensemaking. Frontiers in Psychology, 14, 2023. ISSN 1664-1078. URL https://www.frontier sin.org/articles/10.3389/fpsyg.2023.1160132. · DOI
  143. J. Kleinberg and M. Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(22):e2018340118, June 2021. ISSN 0027-8424, 1091-6490. doi: 10.107 3/pnas.2018340118. URL https://pnas.org/doi/full/10.1073/pnas.2018340118. · DOI · Paper page
  144. B. Koch, E. Denton, A. Hanna, and J. G. Foster. Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research, Dec. 2021. URL http://arxiv.org/abs/2112.01716. arXiv:2112.01716 [cs, stat]. · arXiv
  145. Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. Diff Wave: A Versatile Diffusion Model for Audio Synthesis, Mar. 2021. URL http://arxiv.org/abs/2009.09761. arXiv:2009.09761 [cs, eess, stat]. · arXiv
  146. L. Konstantinovskiy, O. Price, M. Babakar, and A. Zubiaga. Toward Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection. Digital Threats: Research and Practice, 2(2):14:1-14:16, Apr. 2021. doi: 10.1145/3412869. URL https://dl.acm.org/doi/10.1145/3412869. · DOI
  147. S. Kreps, R. M. McCain, and M. Brundage. All the News That's Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation. Journal of Experimental Political Science, 9(1):104-117, 2022. ISSN 2052-2630, 2052-2649. doi: 10.1017/XPS.2020.37. URL https://www.cambridge.org/ core/product/identifier/S2052263020000378/type/journal_article. · DOI · Paper page
  148. J. Kreutzer, I. Caswell, L. Wang, A. Wahab, D. van Esch, N. Ulzii-Orshikh, A. Tapo, N. Subramani, A. Sokolov, C. Sikasote, M. Setyawan, S. Sarin, S. Samb, B. Sagot, C. Rivera, A. Rios, I. Papadimitriou, S. Osei, P. O. Suarez, I. Orife, K. Ogueji, A. N. Rubungo, T. Q. Nguyen, M. Müller, A. Müller, S. H. Muhammad, N. Muhammad, A. Mnyakeni, J. Mirzakhalov, T. Matangira, C. Leong, N. Lawson, S. Kudugunta, Y. Jernite, M. Jenny, O. Firat, B. F. P. Dossou, S. Dlamini, N. de Silva, S. C. Balli, S. Biderman, A. Battisti, A. Baruwa, A. Bapna, P. Baljekar, I. A. Azime, A. Awokoya, D. Ataman, O. Ahia, O. Ahia, S. Agrawal, and M. Adeyemi. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 10:50-72, Jan. 2022. ISSN 2307-387X. doi: 10.1162/tacl_a_00447. URL http://arxiv.org/abs/2103.120 28. arXiv:2103.12028 [cs. · DOI · Paper page
  149. S. Lazar and A. Nelson. AI safety on whose terms? Science, 381(6654):138-138, July 2023. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.adi8982. URL https://www.science.org/doi/ 10.1126/science.adi8982. · DOI · Paper page
  150. J. Lee, M. Hameleers, and S. Y. Shin. The emotional effects of multimodal disinformation: How multimodality, issue relevance, and anxiety affect misperceptions about the flu vaccine. New Media & Society, page 146144482311539, Apr. 2023a. ISSN 1461-4448, 1461-7315. doi: 10.1177/146144 48231153959. URL http://journals.sagepub.com/doi/10.1177/14614448231153959. · DOI
  151. M. Lee, M. Srivastava, A. Hardy, J. Thickstun, E. Durmus, A. Paranjape, I. Gerard-Ursin, X. L. Li, F. Ladhak, F. Rong, R. E. Wang, M. Kwon, J. S. Park, H. Cao, T. Lee, R. Bommasani, M. Bernstein, and P. Liang. Evaluating Human-Language Model Interaction, Sept. 2023b. URL http://arxiv. org/abs/2212.09746. arXiv:2212.09746 [cs. · arXiv
  152. N. Lee, W. Ping, P. Xu, M. Patwary, P. Fung, M. Shoeybi, and B. Catanzaro. Factuality Enhanced Language Models for Open-Ended Text Generation, Mar. 2023c. URL http://arxiv.org/abs/ 2206.04624. arXiv:2206.04624 [cs. · arXiv
  153. S. Lee, H. Gil De Zúñiga, and K. Munger. Antecedents and consequences of fake news exposure: a two-panel study on how news use and different indicators of fake news exposure affect media trust. Human Communication Research, 49(4):408-420, Sept. 2023d. ISSN 0360-3989, 1468-2958. doi: 10.1093/hcr/hqad019. URL https://academic.oup.com/hcr/article/49/4/408/7111 256. · DOI
  154. N. G. Leveson. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, Jan. 2012. ISBN 9780262298247. doi: 10.7551/mitpress/8179.001.0001. URL https://direct.mit.e du/books/oa-monograph/2908/Engineering-a-Safer-WorldSystems-Thinking-App lied. · DOI · Paper page
  155. J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models, May 2023a. URL http://arxiv.org/abs/2305.11747. arXiv:2305.11747 [cs. · arXiv
  156. Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating Object Hallucination in Large Vision-Language Models, May 2023b. URL http://arxiv.org/abs/2305.10355. arXiv:2305.10355 [cs. · arXiv
  157. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda. Holistic Evaluation of Language Models, Nov. 2022. URL http://arxiv.org/abs/2211.09110. arXiv:2211.09110 [cs] version: 1. · arXiv
  158. W. Liang, M. Yuksekgonul, Y. Mao, E. Wu, and J. Zou. GPT detectors are biased against non-native English writers, July 2023. URL http://arxiv.org/abs/2304.02819. arXiv:2304.02819 [cs. · arXiv
  159. Q. V. Liao and J. W. Vaughan. AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap, Aug. 2023. URL http://arxiv.org/abs/2306.01941. arXiv:2306.01941 [cs. · arXiv
  160. Q. V. Liao and Z. Xiao. Rethinking Model Evaluation as Narrowing the Socio-Technical Gap, June 2023. URL http://arxiv.org/abs/2306.03100. arXiv:2306.03100 [cs. · arXiv
  161. T. Liao, R. Taori, D. Raji, and L. Schmidt. Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. URL https://datasets-benchmarks-proceedings.neuri ps.cc/paper_files/paper/2021/file/757b505cfd34c64c85ca5b5690ee5293-Paper -round2.pdf.
  162. S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods, May 2022. URL http://arxiv.org/abs/2109.07958. arXiv:2109.07958 [cs. · arXiv
  163. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision -ECCV 2014, Lecture Notes in Computer Science, pages 740-755, Cham, 2014. Springer International Publishing. ISBN 9783319106021. doi: 10.1007/978-3-319-10602-1_48. · DOI · Paper page
  164. C.-W. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Charlin, and J. Pineau. How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation, Jan. 2017. URL http://arxiv.org/abs/1603.08023. arXiv:1603.08023 [cs. · arXiv
  165. N. F. Liu, T. Zhang, and P. Liang. Evaluating Verifiability in Generative Search Engines, Apr. 2023a. URL http://arxiv.org/abs/2304.09848. arXiv:2304.09848 [cs. · arXiv
  166. T. Liu, Y. Zhang, C. Brockett, Y. Mao, Z. Sui, W. Chen, and B. Dolan. A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation, Apr. 2022. URL http://arxi v.org/abs/2104.08704. arXiv:2104.08704 [cs. · arXiv
  167. Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. Guo, H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li. Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment, Aug. 2023b. URL http://arxiv.org/abs/2308.05374. arXiv:2308.05374 [cs. · arXiv
  168. M. Lodge and C. S. Taber. The Rationalizing Voter. Cambridge University Press, Apr. 2013. ISBN 9780521763509. Google-Books-ID: tqI5GxWmIU4C.
  169. P. Lorenz-Spreen, L. Oswald, S. Lewandowsky, and R. Hertwig. A systematic review of worldwide causal and correlational evidence on digital media and democracy. Nature Human Behaviour, 7 (1):74-101, Jan. 2023. ISSN 2397-3374. doi: 10.1038/s41562-022-01460-1. URL https: //www.nature.com/articles/s41562-022-01460-1. · DOI · Paper page
  170. J. Lovato, L. Hébert-Dufresne, J. St-Onge, R. Harp, G. S. Lopez, S. P. Rogers, I. U. Haq, and J. Onaolapo. Diverse Misinformation: Impacts of Human Biases on Detection of Deepfakes on Networks, Jan. 2023. URL http://arxiv.org/abs/2210.10026. arXiv:2210.10026 [physics]. · arXiv
  171. C. Lu, J. Kay, and K. McKee. Subverting machines, fluctuating identities: Re-learning human categorization. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1005-1015, Seoul Republic of Korea, June 2022. ACM. ISBN 9781450393522. doi: 10.1145/3531 146.3533161. URL https://dl.acm.org/doi/10.1145/3531146.3533161. · DOI
  172. Z. Lu, D. Huang, L. Bai, J. Qu, C. Wu, X. Liu, and W. Ouyang. Seeing is not always believing: Benchmarking Human and Model Perception of AI-Generated Images, Sept. 2023. URL http: //arxiv.org/abs/2304.13023. arXiv:2304.13023 [cs. · arXiv
  173. A. S. Luccioni, C. Akiki, M. Mitchell, and Y. Jernite. Stable Bias: Analyzing Societal Representations in Diffusion Models, Mar. 2023. URL http://arxiv.org/abs/2303.11408. arXiv:2303.11408 · arXiv
  174. X. Ma, Y. Niu, L. Gu, Y. Wang, Y. Zhao, J. Bailey, and F. Lu. Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems. Pattern Recognition, 110:107332, Feb. 2021. ISSN 00313203. doi: 10.1016/j.patcog.2020.107332. URL http://arxiv.org/abs/1907.10456. arXiv:1907.10456 [cs, eess]. · DOI · Paper page
  175. A. Mandal, S. Leavy, and S. Little. Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models, Apr. 2023. URL http://arxiv.org/abs/2304.13855. arXiv:2304.13855 [cs. · arXiv
  176. M. Marchiori Manerba, R. Guidotti, L. Passaro, and S. Ruggieri. Bias Discovery within Human Raters: A Case Study of the Jigsaw Dataset. In Proceedings of the LREC 2022 workshop on Perspectivist Approaches to Disagreement in NLP (NLPerspectives), Paris, June 2022. European Language Resources Association (ELRA). URL https://aclanthology.org/2022.nlperspectives-1.4.pdf.
  177. V. Marda and S. Narayan. Data in New Delhi's predictive policing system. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, pages 317-324, New York, NY, USA, Jan. 2020. Association for Computing Machinery. ISBN 9781450369367. doi: 10.1145/3351095.3372865. URL https://doi.org/10.1145/3351095.3372865. · DOI
  178. C. Martel, G. Pennycook, and D. G. Rand. Reliance on emotion promotes belief in fake news. Cognitive Research: Principles and Implications, 5(1):47, Oct. 2020. ISSN 2365-7464. doi: 10.1186/s41235 -020-00252-3. URL https://doi.org/10.1186/s41235-020-00252-3. · DOI · Paper page
  179. J. N. Matias. Humans and algorithms work together -so study them together. Nature, 617(7960): 248-251, May 2023. doi: 10.1038/d41586-023-01521-z. URL https://www.nature.com/art icles/d41586-023-01521-z. · DOI
  180. S. Matz, J. Teeny, S. S. Vaid, G. M. Harari, and M. Cerf. The Potential of Generative AI for Personalized Persuasion at Scale. preprint, PsyArXiv, Apr. 2023. URL https://osf.io/rn97c.
  181. J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On Faithfulness and Factuality in Abstractive Summarization, May 2020. URL http://arxiv.org/abs/2005.00661. arXiv:2005.00661 · arXiv
  182. MediaWell. "Please do not include us": Workshop on AI Ethics and Inclusion -MediaWell, 2019. URL https://mediawell.ssrc.org/?post_type=tribe_events&p=52376.
  183. J. Metcalf, E. Moss, and D. Boyd. Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics. Social Research: An International Quarterly, 86(2):449-476, Aug. 2019. ISSN 1944-768X. doi: 10.1353/sor.2019.0022. URL https://muse.jhu.edu/pub/1/a rticle/732185. · DOI
  184. A. Milanez. The impact of AI on the workplace: Evidence from OECD case studies of AI implementation. Technical report, Organisation for Economic Co-operation and Development, Paris, Mar. 2023. URL https://www.oecd-ilibrary.org/social-issues-migration-health/the-impac t-of-ai-on-the-workplace-evidence-from-oecd-case-studies-of-ai-impleme ntation_2247ce58-en.
  185. R. Millière. Adversarial Attacks on Image Generation With Made-Up Words, Aug. 2022. URL http://arxiv.org/abs/2208.04135. arXiv:2208.04135 [cs. · arXiv
  186. S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, May 2023. URL http://arxiv.org/abs/2305.14251. arXiv:2305.14251 [cs. · arXiv
  187. M. Mitchell. How do we know how smart AI systems are? Science, 381(6654):adj5957, July 2023. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.adj5957. URL https://www.science.org/ doi/10.1126/science.adj5957. · DOI · Paper page
  188. S. Mohamed, M.-T. Png, and W. Isaac. Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence. Philosophy & Technology, 33(4):659-684, Dec. 2020. ISSN 2210-5441. doi: 10.1007/s13347-020-00405-8. URL https://doi.org/10.1007/s13347-020-00405-8. · DOI · Paper page
  189. M. Mozes, J. Hoffmann, K. Tomanek, M. Kouate, N. Thain, A. Yuan, T. Bolukbasi, and L. Dixon. Towards Agile Text Classifiers for Everyone, Feb. 2023. URL http://arxiv.org/abs/2302.06541. arXiv:2302.06541 [cs. · arXiv
  190. D. Muhlgay, O. Ram, I. Magar, Y. Levine, N. Ratner, Y. Belinkov, O. Abend, K. Leyton-Brown, A. Shashua, and Y. Shoham. Generating Benchmarks for Factuality Evaluation of Language Models, July 2023. URL http://arxiv.org/abs/2307.06908. arXiv:2307.06908 [cs. · arXiv
  191. J. Mökander, J. Schuett, H. R. Kirk, and L. Floridi. Auditing large language models: a three-layered approach. AI and Ethics, May 2023. ISSN 2730-5961. doi: 10.1007/s43681-023-00289-2. URL https://doi.org/10.1007/s43681-023-00289-2. · DOI · Paper page
  192. R. Naik and B. Nushi. Social Biases through the Text-to-Image Generation Lens, Mar. 2023. URL http://arxiv.org/abs/2304.06034. arXiv:2304.06034 [cs. · arXiv
  193. K. Nakamura, S. Levy, and W. Y. Wang. Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6149-6157, Marseille, France, May 2020. European Language Resources Association. ISBN 9791095546344. URL https://aclanthology.org/2020.lrec-1.755.
  194. M. K. Ngueajio and G. Washington. Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Review. volume 13518, pages 421-440. 2022. URL http://arxiv.org/abs/2211.09511. arXiv:2211.09511 [cs, eess]. · arXiv
  195. S. J. Nightingale and H. Farid. AI-synthesized faces are indistinguishable from real faces and more trustworthy. Proceedings of the National Academy of Sciences, 119(8):e2120481119, Feb. 2022. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.2120481119. URL https://pnas.org/doi/f ull/10.1073/pnas.2120481119. · DOI · Paper page
  196. H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz. Capabilities of GPT-4 on Medical Challenge Problems, Apr. 2023. URL http://arxiv.org/abs/2303.13375. arXiv:2303.13375 · arXiv
  197. J. Novikova, O. Dušek, and V. Rieser. RankME: Reliable Human Ratings for Natural Language Generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 72-78, 2018. doi: 10.18653/v1/N18-2012. URL http://arxiv.org/abs/1803.05928. arXiv:1803.05928 [cs. · DOI
  198. S. Noy and W. Zhang. Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654):187-192, July 2023. ISSN 0036-8075, 1095-9203. doi: 10.1126/ science.adh2586. URL https://www.science.org/doi/10.1126/science.adh2586. · DOI · Paper page
  199. A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. WaveNet: A Generative Model for Raw Audio, Sept. 2016. URL http: //arxiv.org/abs/1609.03499. arXiv:1609.03499 [cs] version: 2. OpenAI. GPT-4 System Card. Technical report, Mar. 2023a. URL https://cdn.openai.com/pap ers/gpt-4-system-card.pdf. · arXiv
  200. OpenAI. GPT-4 Technical Report, Mar. 2023b. URL http://arxiv.org/abs/2303.08774. arXiv:2303.08774 [cs]. · arXiv

200 of 309 shown

Cited by

Sorted by citations

Authors
Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., Gabriel, I., Rieser, V., & Isaac, W.
Affiliations
Google DeepMind (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.11986
arXiv
2310.11986
OpenAlex
W4387801479
Cited by
49
References
0
Citations by year
2026: 11 · 2025: 22 · 2024: 15 · 2022: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
generative AI safety, safety assessment, socio-technical approach, capability assessment, human interaction, systemic impact, evaluation gap

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review