Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model". This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness. To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the $\textbf{Situational Awareness Dataset (SAD)}$, a benchmark comprising 7 task categories and over 13,000 questions. The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge. We evaluate 16 LLMs on SAD, including both base (pretrained) and chat models. While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU). Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks. The purpose of SAD is to facilitate scientific understanding of situational awareness in LLMs by breaking it down into quantitative abilities. Situational awareness is important because it enhances a model's capacity for autonomous planning and action. While this has potential benefits for automation, it also introduces novel risks related to AI safety and control. Code and latest results available at https://situational-awareness-dataset.org .
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the Situational Awareness Dataset (SAD), Laine et al. [56] observed that even state-of-the-art models perform well below human baselines in recognizing their own outputs or deployment contexts.”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [56]
In the citing paper
“a juxtaposition that underscores the fragmented nature of current LLM introspection.”
From Self-Referential Introspection in Large Language Models: The Critical Threshold for Recursive Self-Improvement (2026) · cites this paper as [56]
Citations
Cites
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899
Self-citationThe WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.03218 · Cited by 90
Teaching Models to Express Their Uncertainty in Words
Lin, S., Hilton, J., & Evans, O. (2022) · arXiv · doi:10.48550/arxiv.2205.14334 · Cited by 59
Self-citationGAIA: a benchmark for General AI Assistants
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2311.12983 · Cited by 12
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents
Liu, Z., Yao, W., Zhang, J., Xue, L., Heinecke, S., Murthy, R., Feng, Y., Chen, Z., Niebles, J. C., Arpit, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.05960 · Cited by 11
WildChat: 1M ChatGPT Interaction Logs in the Wild
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., & Deng, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.01470 · Cited by 10
Tell, don't show: Declarative facts influence how LLMs generalize
Meinke, A., & Evans, O. (2023) · arXiv · doi:10.48550/arxiv.2312.07779
Self-citation
Cited by
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Self-citationThe alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68
From Imitation to Introspection: Probing Self-Consciousness in Language Models
Chen, S., Yu, S., Zhao, S., & Lu, C. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.392 · Cited by 29
Zhang, J., Yuan, B., & Zhang, Q. (2026) · Entropy · doi:10.3390/e28090951 · Cited by 2
Metadata
- Authors
- Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., Hobbhahn, M., Meinke, A., & Evans, O.
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 64010-64118
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415797398
- Cited by
- 8
- References
- 8
- Citations by year
- 2026: 6 · 2025: 2
- FWCI
- 2.31
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- situational awareness, large language models, human-AI interaction, datasets, human-AI collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs. https://app.cochat.ai/papers/10.52202/079017-2044
- The paper
- Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., Hobbhahn, M., Meinke, A., & Evans, O. (2024). Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs. Advances in Neural Information Processing Systems 37. https://doi.org/10.52202/079017-2044
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.