The DOI matches its DataCite record; 2 of the 3 references it cites were found in a registry or an index, and 1 are not checked yet.
I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench
Open accessAbstract
Do large language models (LLMs) exhibit any forms of awareness similar to humans? In this paper, we introduce AwareBench, a benchmark designed to evaluate awareness in LLMs. Drawing from theories in psychology and philosophy, we define awareness in LLMs as the ability to understand themselves as AI models and to exhibit social intelligence. Subsequently, we categorize awareness in LLMs into five dimensions, including capability, mission, emotion, culture, and perspective. Based on this taxonomy, we create a dataset called AwareEval, which contains binary, multiple-choice, and open-ended questions to assess LLMs' understandings of specific awareness dimensions. Our experiments, conducted on 13 LLMs, reveal that the majority of them struggle to fully recognize their capabilities and missions while demonstrating decent social intelligence. We conclude by connecting awareness of LLMs with AI alignment and safety, emphasizing its significance to the trustworthy and ethical development of LLMs. Our dataset and code are available at https://github.com/HowieHwong/Awareness-in-LLM.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Li et al. [48] revealed that while models exhibit robust mission awareness (understanding their purpose), capability awareness (grasping functional limits) remains the weakest cognitive dimension.”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [48]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite these capabilities, this self-modeling remains architecturally asymmetric.”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Li et al. [48] revealed that while models exhibit robust mission awareness (understanding their purpose), capability awareness (grasping functional limits) remains the weakest cognitive dimension.”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 45
- Michael V Antony. 2001. Is 'consciousness' ambigu- ous? Journal of Consciousness Studies, 8(2):19-44.
- Marta Arguedas, Athanasios Daradoumis, and Fatos Xhafa Xhafa. 2016. Analyzing how emotion aware- ness influences students' motivation, engagement, self-regulation and learning outcome. Educational technology and society, 19(2):87-103.
- Janet Wilde Astington and Jennifer M Jenkins. 1995. Theory of mind development and social understand- ing. Cognition & Emotion, 9(2-3):151-165.
- Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. 2023. Taken out of context: On measuring situational awareness in llms. arXiv preprint arXiv:2309.00667. · arXiv
- Olaf Blanke. 2012. Multisensory brain mechanisms of bodily self-consciousness. Nature Reviews Neuro- science, 13(8):556-571.
- Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202.05144. · arXiv
- Heather Browning and Walter Veit. 2023. Studying introspection in animals and ais. Journal of Con- sciousness Studies, 30(9-10):63-74.
- Stephanie M Carlson, Melissa A Koenig, and Made- line B Harms. 2013. Theory of mind. Wiley Interdis- ciplinary Reviews: Cognitive Science, 4(4):391-402.
- Stacy L. Carter and John J. Wheeler. 2019. Chapter 9 - social validity and cultural competence. In Stacy L. Carter and John J. Wheeler, editors, The Social Valid- ity Manual (Second Edition), second edition edition, pages 217-228. Academic Press. · DOI
- Frank EX Dance. 1970. The "concept" of communica- tion. Journal of communication, 20(2):201-210.
- David DeGrazia. 2009. Self-awareness in animals. The philosophy of animal minds. Cambridge, England: Cambridge University Press.
- Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman. 2023. Solving math word prob- lems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102. · arXiv
- Geert Hofstede, Gert Jan Hofstede, and Michael Minkov. 2010. Cultures et organisations: Nos programma- tions mentales. Pearson Education France.
- Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023. Metatool bench- mark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128. · arXiv
- Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398. · arXiv
- Bärbel Inhelder. 1967. The child's conception of space. London: Routledge & K. Paul.
- Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, and Xin Zhou. 2023. Better zero-shot reasoning with role-play prompting. arXiv preprint arXiv:2308.07702. · arXiv
- Justin Kruger and David Dunning. 1999. Unskilled and unaware of it: how difficulties in recogniz- ing one's own incompetence lead to inflated self- assessments. Journal of personality and social psy- chology, 77(6):1121.
- Dorothée Legrand. 2006. The bodily self: The sensori- motor roots of pre-reflective self-consciousness. Phe- nomenology and the cognitive sciences, 5(1):89-118.
- Alan M Leslie, Ori Friedman, and Tim P German. 2004. Core mechanisms in 'theory of mind'. Trends in cognitive sciences, 8(12):528-533.
- Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023b. Halueval: A large- scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6449-6464.
- Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yi- fan Xu, Weng Lam Tam, et al. 2023a. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743. · arXiv
- Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. Gpte- val: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. · arXiv
- Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023c. Trust- worthy llms: a survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05374. · arXiv
- Karl Marx. 1845. Theses on feuerbach. Newcomb Livraria Press.
- George Herbert Mead et al. 1934. Mind, self, and society, volume 111. University of Chicago press Chicago.
- Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1-40.
- Alain Morin. 2011. Self-awareness part 1: Definition, measures, effects, functions, and antecedents. Social and personality psychology compass, 5(10):807-823.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
- Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022a. Red team- ing language models with language models. arXiv preprint arXiv:2202.03286. · arXiv
- Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kada- vath, et al. 2022b. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251. · arXiv
- Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open- domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207. · arXiv
- Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6943- 6951.
- Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role play with large language models. Nature, pages 1-6.
- Stephane M Shepherd, Cynthia Willis-Esqueda, Danielle Newton, Diane Sivasubramaniam, and Yin Paradies. 2019. The challenge of cultural compe- tence in the workplace: perspectives of healthcare providers. BMC Health Services Research, 19(1):1- 11.
- Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Gerben A Van Kleef. 2009. How emotions regulate social life: The emotions as social information (easi) model. Current directions in psychological science, 18(3):184-188.
- Wendell Wallach and Colin Allen. 2008. Moral ma- chines: Teaching robots right from wrong. Oxford University Press.
- Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neu- big, and Pengfei Liu. 2023. Alignment for honesty. arXiv preprint arXiv:2312.07000. · arXiv
- Jing Yao, Xiaoyuan Yi, Xiting Wang, Yifan Gong, and Xing Xie. 2023. Value fulcra: Mapping large language models to the multidimensional spectrum of basic human values. arXiv preprint arXiv:2311.10766. · arXiv
- Emily S Zhan, María D Molina, Minjin Rheu, and Wei Peng. 2023. What is there to fear? understand- ing multi-dimensional fear of ai from a technologi- cal affordance perspective. International Journal of Human-Computer Interaction, pages 1-18.
- Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023a. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219. · arXiv
- Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023b. Safety- bench: Evaluating the safety of large language mod- els with multiple choice questions. arXiv preprint arXiv:2309.07045. · arXiv
- Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964. · arXiv
Cited by
Sorted by citations
From Imitation to Introspection: Probing Self-Consciousness in Language Models
Chen, S., Yu, S., Zhao, S., & Lu, C. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.392 · Cited by 4
Zhang, J., Yuan, B., & Zhang, Q. (2026) · Entropy · doi:10.3390/e28090951 · Cited by 2
Quantifying UncertAInty: Testing the Accuracy of LLMs’ Confidence JudgmentsPreprint
Cash, T. N., Oppenheimer, D. M., & Christie, S. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/47df5 · Cited by 1
Metadata
- Authors
- Li, Y., Huang, Y., Lin, Y., Wu, S., Wan, Y., & Sun, L.
- Affiliations
- Lehigh University (US)
- University of Notre Dame (US)
- Huazhong University of Science and Technology (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2401.17882
- OpenAlex
- W4391462974
- Cited by
- 1
- References
- 0
- Citations by year
- 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Arts and Humanities › Philosophy › Philosophy and Theoretical Science
- Keywords
- large language models, AI awareness, benchmarking, self-understanding, social intelligence, AI alignment, AI safety, AI ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
