Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs
Open accessAbstract
AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model". This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness. To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the $\textbf{Situational Awareness Dataset (SAD)}$, a benchmark comprising 7 task categories and over 13,000 questions. The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge. We evaluate 16 LLMs on SAD, including both base (pretrained) and chat models. While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU). Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks. The purpose of SAD is to facilitate scientific understanding of situational awareness in LLMs by breaking it down into quantitative abilities. Situational awareness is important because it enhances a model's capacity for autonomous planning and action. While this has potential benefits for automation, it also introduces novel risks related to AI safety and control. Code and latest results available at https://situational-awareness-dataset.org .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SAD is a benchmark for measuring a model's situational awareness across seven task categories.
“SAD (Laine et al., 2024), a benchmark for measuring a model's situational awareness across seven task categories.”
From From Imitation to Introspection: Probing Self-Consciousness in Language Models (2025) · cites this paper as Laine et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 15
- Lin, S., Hilton, J., and Evans, O. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022a. · arXiv
- Long, R. Introspective capabilities in large language models. Journal of Consciousness Studies, 30 (9-10):143-153, September 2023. doi: 10.53765/20512201.30.9.143. · DOI · Paper page
- Meinke, A. and Evans, O. Tell, don't show: Declarative facts influence how llms generalize. December 2023. URL https://arxiv.org/abs/2312.07779. · arXiv
- Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., Zhang, H., Emmons, S., and Hendrycks, D. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, pp. 26837-26867.
- PMLR, 2023.
- Perez, E. and Long, R. Towards evaluating ai systems for moral status using self-reports. arXiv preprint arXiv:2311.08576, 2023. · arXiv
- Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., Jones, A., Chen, A., Mann, B., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Khundadze, G., Kernion, J., Landis, J., Kerr, J., Mueller, J., Hyun, J., Landau, J., Ndousse, K., Goldberg, L., Lovitt, L., Lucas, M., Sellitto, M., Zhang, M., Kingsland, N., Elhage, N., Joseph, N., Mercado, N., DasSarma, N., Rausch, O., Larson, R., McCandlish, S., Johnston, S., Kravec, S., El Showk, S., Lanham, T., Telleen-Lawton, T., Brown, T., Henighan, T., Hume, T., Bai, Y., Hatfield-Dodds, Z., Clark, J., Bowman, S. R., Askell, A., Grosse, R., Hernandez, D., Ganguli, D., Hubinger, E., Schiefer, N., and Kaplan, J. Discovering language model behaviors with model-written evaluations, 2022. URL https://arxiv.org/abs/2212.09251. · arXiv
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. · arXiv
- Thombre, S., Zhao, Z., Ramm-Schmidt, H., Vallet García, J. M., Malkamäki, T., Nikolskiy, S., Hammarberg, T., Nuortie, H., H. Bhuiyan, M. Z., Särkkä, S., and Lehtola, V. V. Sensors and AI techniques for situational awareness in autonomous ships: A review. IEEE Transactions on Intelligent Transportation Systems, 23(1):64-83, 2022. doi: 10.1109/TITS.2020.3023957. · DOI
- Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
- Treutlein, J., Choi, D., Betley, J., Anil, C., Marks, S., Grosse, R. B., and Evans, O. Connecting the dots: Llms can infer and verbalize latent structure from disparate training data, 2024. URL https://arxiv.org/abs/2406.14546. · arXiv
- van Duijn, M., van Dijk, B., Kouwenhoven, T., de Valk, W., Spruit, M., and van der Putten, P. Theory of mind in large language models: Examining performance of 11 state-of-the-art models vs. children aged 7-10 on advanced tests. In Jiang, J., Reitter, D., and Deng, S. (eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pp. 389-402, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. conll-1.25. URL https://aclanthology.org/2023.conll-1. · DOI · Paper page
- Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. · arXiv
- Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM.
Cited by
Sorted by citations
From Imitation to Introspection: Probing Self-Consciousness in Language Models
Chen, S., Yu, S., Zhao, S., & Lu, C. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.392 · Cited by 4 · see citing works
Does It Make Sense to Speak of Introspection in Large Language Models?
Comsa, I. M., & Shanahan, M. (2025) · arXiv · doi:10.48550/arxiv.2506.05068 · Cited by 3 · see citing works
“…al. (2024) explored the concept of self-consciousness in existing LLMs through a curated list of relevant datasets, including situational awareness (Laine et al., 2024), self-reflection (Kim et al., 2023) and known unknowns (Yin et al., 2023), and analysed model activations to describe internal…”
Sabotage Evaluations for Frontier Models
Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., Durmus, E., Ganguli, D., Kravec, S., Shlegeris, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.21514 · Cited by 2 · see citing works
Natural Emergent Misalignment from Reward Hacking in Production RL
MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., Bouscal, N., Bowman, S., Bricken, T., Cloud, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.18397 · Cited by 1 · see citing works
Search-Time Data Contamination
Han, Z., Mankikar, M., Michael, J., & Wang, Z. (2025) · arXiv · doi:10.48550/arxiv.2508.13180
“It would also be interesting to explore if models know they are being evaluated (situational awareness [7]), cheating on the evaluation, or even encourage models to cheat (or encourage them to not cheat) on the evaluation, and whether this affects their ability to retrieve contaminated sources.”
- Authors
- Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., Hobbhahn, M., Meinke, A., & Evans, O.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 64010-64118
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2407.04694
- OpenAlex
- W4415797398
- Cited by
- 8
- References
- 0
- Citations by year
- 2026: 6 · 2025: 2
- FWCI
- 2.31
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- arxiv.org/licenses/nonexclusive-distrib/1.0/
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Keywords
- situational awareness, large language models, human-AI interaction, datasets, human-AI collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
