CoChat
Sign Up
Cited by 53Psychiatry, Mental Health, NeurosciencePhilosophy and Theoretical ScienceAI-based Problem Solving and Planning

Reflective artificial intelligence

Open access
Lewis, Peter R., Sarkadi, Ştefan·2024·Minds and Machines
DOI 10.1007/s11023-024-09664-2
Open this paper in CoChat — free account

Abstract

As artificial intelligence (AI) technology advances, we increasingly delegate mental tasks to machines. However, today’s AI systems usually do these tasks with an unusual imbalance of insight and understanding: new, deeper insights are present, yet many important qualities that a human mind would have previously brought to the activity are utterly absent. Therefore, it is crucial to ask which features of minds have we replicated, which are missing, and if that matters. One core feature that humans bring to tasks, when dealing with the ambiguity, emergent knowledge, and social context presented by the world, is reflection. Yet this capability is completely missing from current mainstream AI. In this paper we ask what reflective AI might look like. Then, drawing on notions of reflection in complex systems, cognitive science, and agents, we sketch an architecture for reflective AI agents, and highlight ways forward.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11023-024-09664-2

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Reflective reasoning and memory systems allow agents to store context across multiple interactions, evaluate past decisions, and iteratively refine their strategies.

    “Furthermore, reflective reasoning and memory systems allow agents to store context across multiple interactions, evaluate past decisions, and iteratively refine their strategies [154].”

    From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [154]

    Section 6 (p. 7) describes agents incorporating feedback from new experiences into reflective models incrementally, and the Tier 3 loops (p. 11) describe using the outcome of an intervention to adapt future behavior. These support learning from experience and refining strategies. The cited passages do not establish a memory system that stores context across multiple interactions, and the paper presents these capabilities as features of a proposed architecture rather than demonstrated agent abilities. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 69

  1. [1][1] Margaret A Boden. AI: Its Nature and Future . Oxford University Press, 2016. -
  2. [2][2] Adrienne Mayor. Gods and Robots: Myths, Machines, and Ancient Dreams of Technology . Princeton University Press, 2018. -
  3. [3][3] Virginia Dignum and Frank Dignum. Agents are dead. Long live agents! In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 1701-1705, 2020. -
  4. [4][4] Ron Sun. Cognitive science meets multi-agent systems: A prolegomenon. Philosophical psychology , 14(1):5-28, 2001. -
  5. [5][5] Antonio Lieto. Cognitive design for artificial minds . Routledge, 2021. -
  6. [6][6] Jeremy Pitt, editor. The Computer After Me . Imperial College Press / World Scientific, 2014. -
  7. [7][7] MTine. Uncovering a differentiated Theory of Mind in children with autism and Asperger syndrome . PhD thesis, Boston College, USA, 2009. -
  8. [9][9] Dagmar Monett, Colin W. P. Lewis, Kristinn R. Thórisson, Joscha Bach, Gianluca Baldassarre, Giovanni Granato, Istvan S. N. Berkeley, François Chollet, Matthew Crosby, Henry Shevlin, John Fox, John E. Laird, Shane Legg, Peter Lindes, Tomáš Mikolov, William J. Rapaport, Raúl Rojas, Marek Rosa, Peter Stone, Richard S. Sutton, Roman V. Yampolskiy, Pei Wang, Roger Schank, Aaron Sloman, and Alan Winfield. Special issue 'on defining artificial intelligence' - commentaries and author's response. Journal of Artificial General Intelligence , 11:1-100, 2020. -
  9. [13][13] Plato (translated by Paul Shorey). Plato in Twelve Volumes , volume 5 & 6. Harvard University Press, Cambridge, MA, USA, 1969. -
  10. [15][15] Germund Hesslow. Conscious thought as simulation of behaviour and perception. Trends in cognitive sciences , 6(6):242-247, 2002. -
  11. [18][18] Alvin M Weinberg. Science and trans-science. Science , 177(4045):211-211, 1972. -
  12. [20][20] Stuart Russell and Peter Norvig. Artificial intelligence: A modern approach, global edition 4th. Foundations , 19:23, 2021. -
  13. [21][21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020. -
  14. [22][22] Michael P Georgeff and Amy L Lansky. Reactive reasoning and planning. In AAAI , volume 87, pages 677-682, 1987. -
  15. [23][23] John R Anderson, Michael Matessa, and Christian Lebiere. ACT-R: A theory of higher level cognition and its relation to visual attention. Human-Computer Interaction , 12(4):439-462, 1997. -
  16. [24][24] Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics , 5(4):115-133, 1943. -
  17. [25][25] Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review , 65(6):386, 1958. -
  18. [26][26] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015. -
  19. [28][28] Aaron Sloman and Ron Chrisley. Virtual machines and consciousness. Journal of Consciousness Studies , 10:133-172, 2003. -
  20. [31][31] Daniel C Dennett. Why the law of effect will not go away. Journal for the Theory of Social Behaviour , 5:169-187, 1975. -
  21. [32][32] Brian Cantwell Smith. Reflection and semantics in lisp. In Proceedings of the 11th ACM SIGACT-SIGPLAN symposium on Principles of programming languages , pages 23-35, 1984. -
  22. [33][33] Brian Cantwell Smith. Procedural reflection in programming languages . PhD thesis, Massachusetts Institute of Technology, 1982. -
  23. [34][34] Anand S Rao, Michael P Georgeff, et al. BDI agents: From theory to practice. In ICMAS , volume 95, pages 312-319, 1995. -
  24. [35][35] Lavindra De Silva, Felipe Meneguzzi, and Brian Logan. Bdi agent architectures: A survey. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI), 2020, Japão. International Joint Conferences on Artificial Intelligence, 2020. -
  25. [36][36] Sam Leask and Brian Logan. Programming agent deliberation using procedural reflection. Fundamenta Informaticae , 158(1-3):93-120, 2018. -
  26. [37][37] Christopher Landauer and Kirstie L. Bellman. Wrappings for software development. In Proceedings of the Thirty-First Hawaii International Conference on System Sciences , volume 3, pages 420-429 vol.3, 1998. -
  27. [38][38] Frances Brazier and Jan Treur. Formal specification of reflective agents. In IJCAI '95 Workshop on Reflection, M. Ibrahim, ed. Montreal , pages 103-112, 1995. -
  28. [40][40] Peter R. Lewis, Arjun Chandra, Funmilade Faniyi, Kyrre Glette, Tao Chen, Rami Bahsoon, J im Torresen, and Xin Yao. Architectural aspects of self-aware and self-expressive computing systems. IEEE Computer , 48:62-70, 2015. -
  29. [41][41] Samuel Kounev, Peter Lewis, Kirstie Bellman, Nelly Bencomo, Javier Camara, Ada Diaconescu, Lukas Esterle, Kurt Geihs, Holger Giese, Sebastian Göetz, Paola Inverardi, Jeffrey Kephart, and Andrea Zisman. The notion of self-aware computing. In Samuel Kounev, Jeffrey O. Kephart, Aleksandar Milenkoski, and Xiaoyun Zhu, editors, Self-Aware Computing Systems , pages 3-16. Springer, 2017. -
  30. [43][43] John McCarthy. Making robots conscious of their mental states. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995] , page 3-17. Oxford University, 1999. -
  31. [44][44] Melanie Mitchell. Self-awareness and control in decentralized systems. In Metacognition in Computation , pages 80-85. AAAI Spring Symposium, 2005. -
  32. [46][46] Peter R. Lewis, Arjun Chandra, Shaun Parsons, Edward Robinson, Kyrre Glette, Rami Bahsoon, Jim Torresen, and Xin Yao. A Survey of Self-Awareness and Its Application in Computing Systems. In Proceedings of the International Conference on Self-Adaptive and Self-Organizing Systems Workshops (SASOW) , pages 102-107. IEEE Computer Society, 2011. -
  33. [47][47] Peter R. Lewis, Arjun Chandra, Funmilade Faniyi, Kyrre Glette, Tao Chen, Rami Bahsoon, Jim Torresen, and Xin Yao. Architectural aspects of self-aware and self-expressive computing systems. IEEE Computer , 48:62-70, 2015. -
  34. [48][48] Peter R. Lewis, Marco Platzner, Bernhard Rinner, Jim Tørresen, and Xin Yao, editors. Self-Aware Computing Systems: An Engineering Approach . Springer, 2016. -
  35. [49][49] Ulric Neisser. The roots of self-knowledge: Perceiving self, it, and thou. Annals of the New York Academy of Science , 818:19-33, 1997. -
  36. [50][50] Peter R. Lewis, Kirstie L. Bellman, Christopher Landauer, Lukas Esterle, Kyrre Glette, Ada Diaconescu, and Holger Giese. Towards a framework for the levels and aspects of self-aware computing systems. In Samuel Kounev, Jeffrey O. Kephart, Aleksandar Milenkoski, and Xiaoyun Zhu, editors, Self-Aware Computing Systems , pages 3-16. Springer, 2017. -
  37. [51][51] K. Bellman, C. Landauer, N. Dutt, L. Esterle, A. Herkersdorf, A. Jantsch, N. TaheriNejad, P. R. Lewis, M. Platzner, and K. Tammemäe. Self-aware cyber-physical systems. ACM Transactions on Cyber-Physical Systems , 4(4), 2020. -
  38. [52][52] Christopher Landauer and Kirstie L. Bellman. Reflective systems need models at run time. In Sebastian Götz, Nelly Bencomo, Kirstie L. Bellman, and Gordon S. Blair, editors, Proceedings of the 11th International Workshop on Models@run.time co-located with 19th International Conference on Model Driven Engineering Languages and Systems (MODELS 2016), Saint Malo, France, October 4, 2016 , volume 1742 of CEUR Workshop Proceedings , pages 52-59. CEUR-WS.org, 2016. -
  39. [53][53] Kirstie L. Bellman, Christopher Landauer, Phyllis Nelson, Nelly Bencomo, Sebastian Götz, Peter R. Lewis, and Lukas Esterle. Self-modeling and self-awareness. In Samuel Kounev, Jeffrey O. Kephart, Aleksandar Milenkoski, and Xiaoyun Zhu, editors, Self-Aware Computing Systems , pages 3-16. Springer, 2017. -
  40. [54][54] Abdessalam Elhabbash, Rami Bahsoon, Peter Tino, Peter R Lewis, and Yehia Elkhatib. Attaining meta-selfawareness through assessment of quality-of-knowledge. In 2021 IEEE International Conference on Web Services (ICWS) , pages 712-723. IEEE Computer Society, 2021. -
  41. [55][55] Danny Weyns, M Usman Iftikhar, and Joakim Söderlund. Do external feedback loops improve the design of self-adaptive systems? a controlled experiment. In 2013 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) , pages 3-12. IEEE, 2013. -
  42. [56][56] Roberta Calegari, Giovanni Ciatto, and Andrea Omicini. On the integration of symbolic and sub-symbolic techniques for XAI: A survey. Intelligenza Artificiale , 14(1):7-32, 2020. -
  43. [58][58] Mitchell A Potter and Kenneth A De Jong. A cooperative coevolutionary approach to function optimization. In International Conference on Parallel Problem Solving from Nature , pages 249-257. Springer, 1994. -
  44. [61][61] Shuo Wang, Georg Nebehay, Lukas Esterle, Kristian Nymoen, , and Leandro L. Minku. Common techniques for self-awareness and self-expression. In Peter R. Lewis, Marco Platzner, Bernhard Rinner, Jim Tørresen, and Xin Yao, editors, Self-Aware Computing Systems: An Engineering Approach , pages 113-142. Springer, 2016. -
  45. [62][62] Frances MT Brazier and Jan Treur. Compositional modelling of reflective agents. International Journal of Human-Computer Studies , 50(5):407-431, 1999. -
  46. [65][65] Alan F. Winfield, Katina Michael, Jeremy Pitt, and Vanessa Evers. Machine ethics: The design and governance of ethical ai and autonomous systems [scanning the issue]. Proceedings of the IEEE , 107(3):509-517, 2019. -
  47. [67][67] Alan FT Winfield and Marina Jirotka. Ethical governance is essential to building trust in robotics and artificial intelligence systems. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 376(2133):20180085, 2018. -
  48. [68][68] Dieter Vanderelst and Alan F. T. Winfield. The dark side of ethical robots. In AAAI/ACM Conference on AI Ethics and Society , pages 317-322, 2018. -
  49. [70][70] Katie Atkinson, Trevor Bench-Capon, and Peter McBurney. Multi-Agent Argumentation for eDemocracy. In EUMAS , pages 35-46, 2005. -
  50. [71][71] Katie Atkinson and Trevor Bench-Capon. Practical reasoning as presumptive argumentation using action based alternating transition systems. Artificial Intelligence , 171(10-15):855-874, 2007. -
  51. [72][72] Katie Atkinson and Trevor Bench-Capon. States, goals and values: Revisiting practical reasoning. Argument & Computation , 7(2-3):135-154, 2016. -
  52. [73][73] Katie Atkinson and Trevor Bench-Capon. Value-based argumentation. Journal of Applied Logics , 8(6):1543-1588, 2021. -
  53. [74][74] Elizabeth I Sklar, Mohammad Q Azhar, Simon Parsons, and Todd Flyr. A case for argumentation to enable human-robot collaboration. Proceedings of Autonomous Agents and Multiagent Systems (AAMAS), St Paul, MN, USA , 2013. -
  54. [75][75] Douglas Walton, Christopher Reed, and Fabrizio Macagno. Argumentation schemes . Cambridge University Press, 2008. -
  55. [76][76] ¸ Stefan Sarkadi, Alison R Panisson, Rafael H Bordini, Peter McBurney, Simon Parsons, and Martin Chapman. Modelling deception using theory of mind in multi-agent systems. AI Communications , 32(4):287-302, 2019. -
  56. [77][77] Francesca Mosca, ¸ Stefan Sarkadi, Jose M Such, and Peter McBurney. Agent EXPRI: Licence to explain. In International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems , pages 21-38. Springer, 2020. -
  57. [78][78] Francesca Mosca and Jose Such. Elvira: an explainable agent for value and utility-driven multiuser privacy. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , 2021. -
  58. [79][79] ¸ Stefan Sarkadi, Alison R Panisson, Rafael H Bordini, Peter McBurney, and Simon Parsons. Towards an approach for modelling uncertain theory of mind in multi-agent systems. In International Conference on Agreement Technologies , pages 3-17. Springer, 2018. -
  59. [80][80] Peter McBurney and Michael Luck. The agents are all busy doing stuff! IEEE Intelligent Systems , 22(4):6-7, 2007. -
  60. [81][81] Louise A Dennis and Nir Oren. Explaining bdi agent behaviour through dialogue. In Proc. of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2021) . International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2021. -
  61. [82][82] Natalia Criado, Estefania Argente, and V Botti. Open issues for normative multi-agent systems. AI communications , 24(3):233-264, 2011. -
  62. [83][83] Stephen Cranefield and Bastin Tony Roy Savarimuthu. Normative multi-agent systems and human-robot interaction. In Workshop on Robot Behavior Adaptation to Human Social Norms (TSAR) , pages 1-3, 2021. -
  63. [84][84] Sevan Gregory Ficici. Solution Concepts in Coevolutionary Algorithms . PhD thesis, Brandeis University, 2004. -
  64. [85][85] Carles Sierra, Nick R Jennings, Pablo Noriega, and Simon Parsons. A framework for argumentation-based negotiation. In International Workshop on Agent Theories, Architectures, and Languages , pages 177-192. Springer, 1997. -
  65. [86][86] Carles Sierra, Juan Antonio Rodriguez-Aguilar, Pablo Noriega, Marc Esteva, and Josep Lluis Arcos. Engineering multi-agent systems as electronic institutions. European Journal for the Informatics Professional , 4(4):33-39, 2004. -
  66. [87][87] Jeremy Pitt, Julia Schaumeier, and Alexander Artikis. Axiomatization of socio-economic principles for selforganizing institutions: Concepts, experiments and challenges. ACM Transactions on Autonomous and Adaptive Systems (TAAS) , 7(4):1-39, 2012. -
  67. [88][88] Natalia Criado, Estefania Argente, and V Botti. A bdi architecture for normative decision making. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1 , pages 1383-1384, 2010. -
  68. [89][89] Natalia Criado. Using norms to control open multi-agent systems. AI Communications , 26(3):317-318, 2013. -
  69. [91][91] Max Oelschlaeger. The myth of the technological fix. The Southwestern Journal of Philosophy , 10(1):43-53, 1979.

Cited by

Sorted by citations

Authors
Lewis, P. R., & Sarkadi, Ş.
Affiliations
  • Ontario Tech University (CA)
  • King's College London (GB)
Corresponding author
  • Lewis, Peter R.
  • Sarkadi, Ştefan
ORCID
Venue
Minds and Machines
Publisher
Springer Science and Business Media LLC
Published
2024-05-18
Volume
34
Issue
2
ISSN
  • 0924-6495 (ISSN-L)
  • 1572-8641
DOI
10.1007/s11023-024-09664-2
OpenAlex
W4397047033
Cited by
53
References
5
Citations by year
2026: 19 · 2025: 23 · 2024: 8 · 2023: 3
FWCI
7.38
Type
journal article
Language
en
Open access
Licence
Locations
  • Minds and Machines (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Research Portal (King's College London) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
Royal Academy of Engineering
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Philosophy and Theoretical Science
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
reflection, AI agents, AI architecture, human cognition, complex systems, cognitive science, ambiguity, emergent knowledge, social environment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review