Visibility into AI Agents
Open accessThe DOI matches its Crossref record; 126 of the 126 references it cites were found in a registry or an index.
Abstract
Increased delegation of commercial, scientific, governmental, and personal activities to AI agents—systems capable of pursuing complex goals with limited supervision—may exacerbate existing societal risks and introduce new risks. Understanding and mitigating these risks involves critically evaluating existing governance structures, revising and adapting these structures where needed, and ensuring accountability of key stakeholders. Information about where, why, how, and by whom certain AI agents are used, which we refer to as visibility, is critical to these objectives. In this paper, we assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging. For each, we outline potential implementations that vary in intrusiveness and informativeness. We analyze how the measures apply across a spectrum of centralized through decentralized deployment contexts, accounting for various actors in the supply chain including hardware and software service providers. Finally, we discuss the implications of our measures for privacy and concentration of power. Further work into understanding the measures and mitigating their negative impacts can help to build a foundation for the governance of AI agents.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“When coupled with validation protocols (e.g., response verification or schema checks in Waitgpt [219]) and observation mechanisms (e.g., feedback loops or audit logs [220, 221]), these boundaries are reinforced, ensuring that each agent not only performs its assigned task but does so transparently and accountably.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [220]
Citations
Cites
Theory of the firm: Managerial behavior, agency costs and ownership structure
Jensen, M. C., & Meckling, W. H. (1976) · Journal of Financial Economics · doi:10.1016/0304-405x(76)90026-x · Cited by 71,491
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“…struggle to perform even simple tasks [98, 106, 108, 130, 163, 164], but given increasing investments in AI research [61], scaling laws [15, 83, 94], pressures to develop autonomous capabilities for military use [86, 102, 142], economic applications [35], and scientific prestige [35, 67], we…”
BackgroundReinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Moral Hazard and Observability
Holmstrom, B. (1979) · The Bell Journal of Economics · doi:10.2307/3003320 · Cited by 8,311
Multitask principal – agent analyses: Incentive contracts, asset ownership, and job design
Holmstrom, B., & Milgrom, P. (1991) · The Journal of Law, Economics, and Organization · doi:10.1093/jleo/7.special_issue.24 · Cited by 6,046
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Agency Problems and Residual Claims
Fama, E. F., & Jensen, M. C. (1983) · The Journal of Law and Economics · doi:10.1086/467038 · Cited by 4,827
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“We refer to the set of such additional information as an agent card , drawing inspiration from previous work on documenting AI systems [24, 68, 69, 112].”
BackgroundMethodAutomation and New Tasks: How Technology Displaces and Reinstates Labor
Acemoglu, D., & Restrepo, P. (2019) · Journal of Economic Perspectives · doi:10.1257/jep.33.2.3 · Cited by 2,612
A 61-million-person experiment in social influence and political mobilization
Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D. I., Marlow, C., Settle, J. E., & Fowler, J. H. (2012) · Nature · doi:10.1038/nature11421 · Cited by 2,575
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 2×
“We refer to the set of such additional information as an agent card , drawing inspiration from previous work on documenting AI systems [24, 68, 69, 112].”
BackgroundMethodGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“Since such deployment may exacerbate existing risks and introduce new ones [35, 147], it is imperative to understand how to govern AI agents.”
BackgroundGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
The Theory of Incentives: The Principal-Agent Model
Laffont, J.-J., & Martimort, D. (2002) · Princeton University Press eBooks · doi:10.1515/9781400829453 · Cited by 2,106
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Ananny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · cited 2×
“Even with a comprehensive understanding of agent activities, those harmed by them may not have the power to intervene and reduce risks [8].”
BackgroundThe Out-of-the-Loop Performance Problem and Level of Control in Automation
Endsley, M. R., & Kiris, E. O. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779064555 · Cited by 1,406
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Board of Governors of the Federal Reserve System
(2001) · Choice Reviews Online · doi:10.5860/choice.38sup-465 · Cited by 1,130
ENDSLEY, M. R., & KABER, D. B. (1999) · Ergonomics · doi:10.1080/001401399185595 · Cited by 1,088
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 1×
“…struggle to perform even simple tasks [98, 106, 108, 130, 163, 164], but given increasing investments in AI research [61], scaling laws [15, 83, 94], pressures to develop autonomous capabilities for military use [86, 102, 142], economic applications [35], and scientific prestige [35, 67],…”
BackgroundTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 798 · cited 1×
“Many AI developers are creating systems with greater autonomy, access to external tools or services, and an increased ability to reliably adapt, plan, and act open-endedly over long time-horizons to achieve goals [35, 97, 106, 133, 137, 146, 158].”
BackgroundAugmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
Magnetic control of tokamak plasmas through deep reinforcement learning
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de las Casas, D., et al. (2022) · Nature · doi:10.1038/s41586-021-04301-9 · Cited by 751 · cited 1×
“As AI agents improve in capabilities, speed, and cost, 2 it may be easier and more competitive to delegate tasks currently done by humans to AI agents instead.”
BackgroundComplex Networks: Structure, Robustness and Function
(2010) · Cited by 729
Complex Networks: Structure, Robustness and Function
Cohen, R., & Havlin, S. (2010) · Cambridge University Press eBooks · doi:10.1017/cbo9780511780356 · Cited by 696
Workplace surveillance: an overview
Ball, K. (2010) · Labor History · doi:10.1080/00236561003654776 · Cited by 488
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
Burrell, J., & Fourcade, M. (2021) · Annual Review of Sociology · doi:10.1146/annurev-soc-090820-020800 · Cited by 430
Lieberman, H. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258592 · Cited by 371
Accountability in a computerized society
Nissenbaum, H. (1996) · Science and Engineering Ethics · doi:10.1007/bf02639315 · Cited by 346
Dual use of artificial-intelligence-powered drug discovery
Urbina, F., Lentzos, F., Invernizzi, C., & Ekins, S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00465-9 · Cited by 346
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · cited 1×
“We focus in this section on agents run by (agent) deployers — entities that deploy agents, or important subcomponents like a foundation model, as a service to users.”
BackgroundPredictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.07785 · Cited by 324
Monitoring approaches for health-care workers during the COVID-19 pandemic
Bielicki, J. A., Duval, X., Gobat, N., Goossens, H., Koopmans, M., Tacconelli, E., & van der Werf, S. (2020) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(20)30458-8 · Cited by 315
A Survey of Text Watermarking in the Era of Large Language Models
Liu, A., Pan, L., Lu, Y., Li, J., Hu, X., Zhang, X., Wen, L., King, I., Xiong, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3691626 · Cited by 299
Auditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
Algorithmic recourse: From counterfactual explanations to interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445899 · Cited by 274
Levels of AGI for Operationalizing Progress on the Path to AGI
Morris, M. R., Sohl-Dickstein, J., Fiedel, N., Warkentin, T., Dafoe, A., Faust, A., Farabet, C., & Legg, S. (2023) · arXiv · doi:10.48550/arxiv.2311.02462 · Cited by 274
Automation Bias in Intelligent Time Critical Decision Support Systems
Cummings, M. (2004) · AIAA 1st Intelligent Systems Technical Conference · doi:10.2514/6.2004-6313 · Cited by 269
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265 · cited 1×
“Since such deployment may exacerbate existing risks and introduce new ones [35, 147], it is imperative to understand how to govern AI agents.”
BackgroundThe flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263 · cited 1×
“Since real-time monitoring requires an immediate decision to flag a behaviour, deployers would only be able to find problems which they can identify in real-time.”
BackgroundThe Flaws of Policies Requiring Human Oversight of Government Algorithms
Green, B. (2021) · doi:10.2139/ssrn.3921216 · Cited by 259
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534181 · Cited by 209
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · cited 1×
“…163, 164], but given increasing investments in AI research [61], scaling laws [15, 83, 94], pressures to develop autonomous capabilities for military use [86, 102, 142], economic applications [35], and scientific prestige [35, 67], we should not discount continued improvements in capabilities [26].”
BackgroundMilli, S., Carroll, M., Wang, Y., Pandey, S., & Dragan, A. D. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16941 · Cited by 200
Application of LLM Agents in Recruitment: A Novel Framework for Automated Resume Screening
Gan, C., Zhang, Q., & Mori, T. (2024) · Journal of Information Processing · doi:10.2197/ipsjjip.32.881 · Cited by 184
Bottom-up data Trusts: disturbing the ‘one size fits all’ approach to data governance
Delacroix, S., & Lawrence, N. D. (2019) · International Data Privacy Law · doi:10.1093/idpl/ipz014 · Cited by 179
Karinshak, E., Liu, S. X., Park, J. S., & Hancock, J. T. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579592 · Cited by 167 · cited 1×
“Since such deployment may exacerbate existing risks and introduce new ones [35, 147], it is imperative to understand how to govern AI agents.”
BackgroundHarms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149 · cited 4×
“More comprehensive information collection may be justified for agents deemed to be high-risk, potentially based on the results of evaluations [9, 98, 108, 135, 137, 152] or deployment in high-risk domains [2, 45, 100].”
BackgroundInfluentialSelf-citationCooper, A. F., Moss, E., Laufer, B., & Nissenbaum, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533150 · Cited by 129
ChemCrow: Augmenting large-language models with chemistry tools
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2023) · arXiv · doi:10.48550/arxiv.2304.05376 · Cited by 122
The philosophical basis of algorithmic recourse
Venkatasubramanian, S., & Alfano, M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372876 · Cited by 114 · cited 1×
“We leave details about the implementation challenges of automation and human review to other work [72, 95, 157, 166].”
MethodNatural Selection Favors AIs over HumansPreprint
Hendrycks, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.16200 · Cited by 100
Can Large Language Models Really Improve by Self-critiquing Their Own Plans?
Valmeekam, K., Marquez, M., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2310.08118 · Cited by 90 · cited 1×
“Current AI agents sometimes struggle to perform even simple tasks [98, 106, 108, 130, 163, 164], but given increasing investments in AI research [61], scaling laws [15, 83, 94], pressures to develop autonomous capabilities for military use [86, 102, 142], economic applications [35], and scientific…”
BackgroundThe Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90
Minkkinen, M., Laine, J., & Mäntymäki, M. (2022) · Digital Society · doi:10.1007/s44206-022-00022-2 · Cited by 88
Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI
Widder, D. G., West, S., & Whittaker, M. (2023) · doi:10.2139/ssrn.4543807 · Cited by 87
TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AIPreprint
Critch, A., & Russell, S. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.06924 · Cited by 86
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79 · cited 2×
“Many AI developers are creating systems with greater autonomy, access to external tools or services, and an increased ability to reliably adapt, plan, and act open-endedly over long time-horizons to achieve goals [35, 97, 106, 133, 137, 146, 158].”
BackgroundFrontier AI Regulation: Managing Emerging Risks to Public Safety
Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O'Keefe, C., Whittlestone, J., Avin, S., Brundage, M., Bullock, J., Cass-Beggs, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03718 · Cited by 77
Self-citationAlgorithmic Recourse: from Counterfactual Explanations to Interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2020) · arXiv · doi:10.48550/arxiv.2002.06278 · Cited by 76
Democratising AI: Multiple Meanings, Goals, and Methods
Seger, E., Ovadya, A., Siddarth, D., Garfinkel, B., & Dafoe, A. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604693 · Cited by 73
Emergent autonomous scientific research capabilities of large language models
Boiko, D. A., MacKnight, R., & Gomes, G. (2023) · arXiv · doi:10.48550/arxiv.2304.05332 · Cited by 72
AI capabilities can be significantly improved without expensive retrainingPreprint
Shimomoto, T., Denain, J.-S., Villalobos, P., & Bas, G. (2023) · arXiv (Cornell University) · doi:10.2139/ssrn.6823878 · Cited by 71
Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., & Kambhampati, S. (2022) · arXiv · doi:10.48550/arxiv.2206.10498 · Cited by 70
Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., & Kambhampati, S. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1693 · Cited by 70
Koessler, L., & Schuett, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08823 · Cited by 66
Model evaluation for extreme risks
Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.15324 · Cited by 66 · cited 3×
“More broadly, profit and efficiency motives may lead to collective dependence on agents for essential societal functions, such as the provision of government services [50, 179] or the operation of essential infrastructure [17, 51].”
BackgroundSelf-citationReward Reports for Reinforcement Learning
Gilbert, T. K., Lambert, N., Dean, S., Zick, T., Snoswell, A., & Mehta, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604698 · Cited by 63
Preventing Language Models From Hiding Their ReasoningPreprint
Roger, F., & Greenblatt, R. (2023) · arXiv.org · doi:10.48550/arxiv.2310.18512 · Cited by 63
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
Bommasani, R., Creel, K. A., Kumar, A., Jurafsky, D., & Liang, P. (2022) · arXiv · doi:10.48550/arxiv.2211.13972 · Cited by 60 · cited 1×
“Since such deployment may exacerbate existing risks and introduce new ones [35, 147], it is imperative to understand how to govern AI agents.”
BackgroundEcosystem Graphs: The Social Footprint of Foundation ModelsPreprint
Bommasani, R., Soylu, D., Liao, T., Creel, K. A., & Liang, P. (2023) · arXiv.org · doi:10.48550/arxiv.2303.15772 · Cited by 54
Picking on the Same Person: Does Algorithmic Monoculture Lead to Outcome Homogenization?
Bommasani, R., Creel, K., Kumar, A., Jurafsky, D., & Liang, P. (2022) · neural information processing systems · doi:10.52202/068431-0265 · Cited by 53
Self-critiquing models for assisting human evaluators
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., & Leike, J. (2022) · arXiv · doi:10.48550/arxiv.2206.05802 · Cited by 50 · cited 1×
“Different categories of logs could exist, such as for tool use, internal reasoning [172], self-critique [141], memory management, etc, each of which may be useful to different actors.”
BackgroundPowerGridworld: a framework for multi-agent reinforcement learning in power systems
Biagioni, D., Zhang, X., Wald, D., Vaidhynathan, D., Chintala, R. H., King, J., & Zamzam, A. S. (2022) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3538637.3539616 · Cited by 50
Sociotechnical Safety Evaluation of Generative AI Systems
Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11986 · Cited by 49
Deployment Corrections: An incident response framework for frontier AI modelsPreprint
O'Brien, J., Ee, S., & Williams, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.00328 · Cited by 45
Runtime Monitoring of Dynamic Fairness Properties
Henzinger, T. A., Karimi, M., Kueffner, K., & Mallik, K. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594028 · Cited by 43
Reclaiming the Digital Commons: A Public Data Trust for Training Data
Chan, A., Bradley, H., & Rajkumar, N. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604658 · Cited by 43
Self-citationOversight for Frontier AI through a Know-Your-Customer Scheme for Compute ProvidersPreprint
Egan, J., & Heim, L. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.35246.48969 · Cited by 35
Self-citationRismani, S., Shelby, R., Smart, A., Jatho, E., Kroll, J., Moon, A., & Rostamzadeh, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 33
Algorithmic collusion: A critical reviewPreprint
Dorner, F. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04740 · Cited by 33
Preparing for the (Non-Existent?) Future of Work
Korinek, A., & JUELFS, M. E. (2022) · National Bureau of Economic Research · doi:10.3386/w30172 · Cited by 32
Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2304.03279 · Cited by 32 · cited 1×
“Real-time monitoring could also flag when an agent exerts significant influence in its environment [127].”
BackgroundA Human-Centric Perspective on Model Monitoring
Shergadwala, M., Lakkaraju, H., & Kenthapadi, K. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21997 · Cited by 30
Monitoring Misuse for Accountable 'Artificial Intelligence as a Service'
Javadi, S. A., Cloete, R., Cobbe, J., Lee, M. S. A., & Singh, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375873 · Cited by 29
Zilka, M., Sargeant, H., & Weller, A. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534200 · Cited by 28
Spear Phishing With Large Language Models
Hazell, J. (2023) · arXiv · doi:10.48550/arxiv.2305.06972 · Cited by 28 · cited 1×
“Extremely persuasive AI agents may also enable and enhance influence campaigns [10, 78, 96].”
Intelligent Autonomous Agents are Key to Cyber Defense of the Future Army Networks
Kott, A. (2018) · The Cyber Defense Review · doi:10.55682/cdr/szep-ay0y · Cited by 27
Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
Bhatt, M., Chennabasappa, S., Nikolaidis, C., Wan, S., Evtimov, I., Gabi, D., Song, D., Ahmad, F., Aschermann, C., Fontana, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.04724 · Cited by 27
Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use CasesPreprint
Bluemke, E., Collins, T., Garfinkel, B., & Trask, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.08956 · Cited by 26
Self-citationCan large language models democratize access to dual-use biotechnology?
Soice, E. H., Rocha, R., Cordova, K., Specter, M., & Esvelt, K. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.03809 · Cited by 25
Emergence in complex networks of simple agents
Green, D. G. (2023) · Journal of Economic Interaction and Coordination · doi:10.1007/s11403-023-00385-w · Cited by 24
From Explanation to Recommendation: Ethical Standards for Algorithmic Recourse
Sullivan, E., & Verreault-Julien, P. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534185 · Cited by 23
'Generative CI' through Collective Response SystemsPreprint
Ovadya, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00672 · Cited by 23
Monitoring AI Services for Misuse
Javadi, S. A., Norval, C., Cloete, R., & Singh, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462566 · Cited by 21 · cited 2×
“While activity logs are useful for understanding risks from AI systems in general [91, 92] and are common practice in many domains [3, 4, 41, 42, 89], they can be especially helpful for addressing the risks we identified in Section 1.1.”
BackgroundMethodInfluentialHuman-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19 · cited 2×
“Examples of agents include reinforcement learning systems [132, 160] that interact extensively with the real world 6 or more capable versions of language models with tool access [27, 122, 137, 144].”
BackgroundIdentifying the Risks of LM Agents with an LM-Emulated Sandbox
Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2309.15817 · Cited by 19
Oversight for Frontier AI through a Know-Your-Customer Scheme for Compute ProvidersPreprint
Egan, J., & Heim, L. (2023) · arXiv.org · doi:10.48550/arxiv.2310.13625 · Cited by 18
Self-citationKenton, Z., Kumar, R., Farquhar, S., Richens, J., MacDermott, M., & Everitt, T. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103963 · Cited by 17
The Gradient of Generative AI Release: Methods and ConsiderationsPreprint
Solaiman, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.04844 · Cited by 16
Self-preferencing by platforms: A literature review
Kittaka, Y., Sato, S., & Zennyo, Y. (2023) · Japan and The World Economy · doi:10.1016/j.japwor.2023.101191 · Cited by 16
Ecosystem Graphs: The Social Footprint of Foundation Models
Bommasani, R., Soylu, D., Liao, T., Creel, K., & Liang, P. (2023) · doi:10.21203/rs.3.rs-2961271/v1 · Cited by 14
Debunking the AI Arms Race Theory (Summer 2021)
Scharre, P. (2021) · doi:10.26153/tsw/13985 · Cited by 13
Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B. (2023) · arXiv · doi:10.48550/arxiv.2311.04378 · Cited by 12 · cited 1×
“In Section 3.1.2, we discuss additional, useful information that could be attached to an identifier.”
BackgroundInfluentialGAIA: a benchmark for General AI Assistants
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2311.12983 · Cited by 12
Regulatory Monitors: Policing Firms in the Compliance Era
Loo, R. V. (2018) · bepress Legal Repository · Cited by 11
Democratising the Digital Revolution: The Role of Data Governance
Delacroix, S., Pineau, J., & Montgomery, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69128-8_3 · Cited by 10
Application of LLM Agents in Recruitment: A Novel Framework for Resume ScreeningPreprint
Gan, C., Zhang, Q., & Mori, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.08315 · Cited by 10
Sandbrink, J. B., Hobbs, H., Swett, J. L., Dafoe, A., & Sandberg, A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4213670 · Cited by 7
Algorithmic progress in computer vision
Erdil, E., & Besiroglu, T. (2022) · arXiv · doi:10.48550/arxiv.2212.05153 · Cited by 7
Kenton, Z., Kumar, R., Farquhar, S., Richens, J., MacDermott, M., & Everitt, T. (2022) · arXiv · doi:10.48550/arxiv.2208.08345 · Cited by 7
Testing Language Model Agents Safely in the Wild
Naihin, S., Atkinson, D., Green, M., Hamadi, M., Swift, C., Schonholtz, D., Kalai, A. T., & Bau, D. (2023) · arXiv · doi:10.48550/arxiv.2311.10538 · Cited by 4 · cited 1×
“Such monitoring is likely to be automated so as to be able to cope with the speed and scale of agent activities [74, 147]; see Naihin et al. [116] for a preliminary implementation of a real-time monitoring system for agents.”
BackgroundA Survey of Text Watermarking in the Era of Large Language ModelsPreprint
Liu, A., Pan, L., Lu, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.07913 · Cited by 4
Research works printed but not linked: 29
- Daron Acemoglu and Pascual Restrepo. 2018. Artificial Intelligence, Automation, and Work. In The Economics of Artificial Intelligence: An Agenda. University of Chicago Press, 197–236. https://www.nber.org/books-and-chapters/economics-artificial-intelligence-agenda/artificial-intelligence-automation-and-work
- 10.31219/osf.io · doi:10.31219/osf.io
- Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. 2023. Image Hijacks: Adversarial Images can Control Generative Models at Runtime. https://arxiv.org/abs/2309.00236v2 · arXiv
- Patrick Birkinshaw. 2006. Freedom of Information and Openness: Fundamental Human Rights?Administrative Law Review 58, 1 (2006), 177–218. https://www.jstor.org/stable/40712007 Publisher: American Bar Association.
- 10.18653/v1 · doi:10.18653/v1
- Ryan Carey and Tom Everitt. 2023. Human Control: Definitions and Algorithms. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence. PMLR, 271–281. https://proceedings.mlr.press/v216/carey23a.html ISSN: 2640-3498.
- Securities and Exchange Commission. 2022. 17 CFR § 240.17a-3 - Records to be made by certain exchange members, brokers and dealers.Federal Register (April 2022).
- Securities and Exchange Commission. 2022. 17 CFR § 240.17a-4 - Records to be preserved by certain exchange members, brokers and dealers.Federal Register (April 2022).
- Ernesto Dal Bó. 2006. Regulatory capture: A review. Oxford review of economic policy 22, 2 (2006), 203–225. Publisher: Oxford University Press.
- 10.1093/idpl · doi:10.1093/idpl
- Jeffrey C. Ely and Balazs Szentes. 2023. Natural selection of artificial intelligence. (2023).
- 10.1093/idpl · doi:10.1093/idpl
- Chloe Goodwin. 2018. Cooperation or resistance?: The role of tech companies in government surveillance. 131 (2018), 1722–1722.
- Ryan Greenblatt Buck Shlegeris Kshitij Sachan and Fabien Roger. 2024. AI Control: Improving Safety Despite Intentional Subversion. http://arxiv.org/abs/2312.06942arXiv:2312.06942 [cs].
- Valentina Guarnieri, Maria Moriondo, Mattia Giovannini, Lorenzo Lodi, Silvia Ricci, Laura Pisano, Paola Barbacci, Costanza Bini, Giuseppe Indolfi, Alberto Zanobini, and Chiara Azzari. 2021. Surveillance on Healthcare Workers During the First Wave of SARS-CoV-2 Pandemic in Italy: The Experience of a Tertiary Care Pediatric Hospital. Frontiers in Public Health 9 (2021). https://www.frontiersin.org/articles/10.3389/fpubh.2021.644702 · DOI
- Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R. Lin, Hjalmar Wijk, Joel Burget, Aaron Ho, Elizabeth Barnes, and Paul Christiano. 2023. Evaluating Language-Model Agents on Realistic Autonomous Tasks. https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.pdf
- 10.2307/j.ctv7h0rwr · doi:10.2307/j.ctv7h0rwr
- Richard Ngo Lawrence Chan and Sören Mindermann. 2022. The alignment problem from a deep learning perspective. https://arxiv.org/abs/2209.00626v5 · arXiv
- Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. 2022. A Generalist Agent. Transactions on Machine Learning Research (2022). https://openreview.net/forum?id=1ikK0kHjvj
- Felix Richter. 2023. Infographic: Amazon Maintains Lead in the Cloud Market. https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers
- Stuart J. Russell and Peter Norvig. 2021. Artificial Intelligence: A Modern Approach (4 ed.).
- Jonas B. Sandbrink. 2023. Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools. http://arxiv.org/abs/2306.13952arXiv:2306.13952 [cs].
- Lee Sharkey, Clíodhna Ní Ghuidhir, Dan Braun, Jérémy Scheurer, Mikita Balesni, Lucius Bushnaq, Charlotte Stix, and Marius Hobbhahn. 2023. A Causal Framework for AI Regulation and Auditing. https://static1.squarespace.com/static/6461e2a5c6399341bcfc84a5/t/654bc268049d687cecac24d8/1699463818729/auditing_framework_web.pdf
- Alexander F Siegenfeld and Yaneer Bar-Yam. 2020. An introduction to complex systems science and its applications. Complexity 2020 (2020), 1–16. Publisher: Hindawi Limited.
- Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. 2015. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence.
- Zihan Wang Olivia Byrnes Hu Wang Ruoxi Sun Congbo Ma Huaming Chen Qi Wu and Minhui Xue. 2021. Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography. https://arxiv.org/abs/2107.09287v3 · arXiv
- Zelai Xu Chao Yu Fei Fang Yu Wang and Yi Wu. 2023. Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game. https://arxiv.org/abs/2310.18940v2 · arXiv
- Andy Zou Zifan Wang Nicholas Carlini Milad Nasr J. Zico Kolter and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. https://arxiv.org/abs/2307.15043v2 · arXiv
- Remco Zwetsloot and Allan Dafoe. 2019. Thinking about risks from AI: Accidents, misuse and structure. Lawfare. February 11 (2019), 2019.
Cited by
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“Systems like ReAct [136] exemplify this architecture, combining reasoning and action in an iterative framework where agents simulate internal dialogue before selecting external actions.”
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
Anwar, U., Saparov, A., Ramírez, J. R., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., et al. (2024) · arXiv (Cornell University) · doi:10.3929/ethz-b-000706962 · Cited by 17
Self-citationAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
The role of agentic AI in shaping a smart future: A systematic review
Hosseini, S., & Seilani, H. (2025) · Array · doi:10.1016/j.array.2025.100399 · Cited by 280
Metadata
- Authors
- Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., Heim, L., & Anderljung, M.
- Affiliations
- Mila - Quebec Artificial Intelligence Institute (CA)
- Harvard University (US)
- Independent, United Kingdom
- University of Oxford (GB)
- University of Cambridge (GB)
- Centre for the Governance of AI (GB)
- University of Toronto (CA)
- ORCID
- Venue
- The 2024 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2024-06-03
- Pages
- 958-973
- OpenAlex
- W4399364360
- Cited by
- 84
- References
- 103
- Citations by year
- 2026: 42 · 2025: 38 · 2024: 4
- FWCI
- 38.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- AI agents, visibility, real-time monitoring, activity logging, AI governance, accountability, privacy, concentration of power, supply chain, societal risk
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.