Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Open accessThe DOI matches its Crossref record; 57 of the 161 references it cites were found in a registry or an index; 33 of the 49 sources its claims cite were checked for support.
Abstract
We introduce Inference-Time Intervention (ITI), a technique designed to enhance the"truthfulness"of large language models (LLMs). ITI operates by shifting model activations during inference, following a set of directions across a limited number of attention heads. This intervention significantly improves the performance of LLaMA models on the TruthfulQA benchmark. On an instruction-finetuned LLaMA called Alpaca, ITI improves its truthfulness from 32.5% to 65.1%. We identify a tradeoff between truthfulness and helpfulness and demonstrate how to balance it by tuning the intervention strength. ITI is minimally invasive and computationally inexpensive. Moreover, the technique is data efficient: while approaches like RLHF require extensive annotations, ITI locates truthful directions using only few hundred examples. Our findings suggest that LLMs may have an internal representation of the likelihood of something being true, even as they produce falsehoods on the surface.
Overview
CoChat Summary
TL;DR
Inference-Time Intervention (ITI) shifts model activations during inference along truth-correlated directions in a sparse set of attention heads, improving truthfulness on TruthfulQA. On Alpaca, ITI raises true*informative from 32.5% to 65.1%, while revealing a trade-off between truthfulness and helpfulness.
Key findings
- ITI substantially improves TruthfulQA performance: Alpaca true*informative rises from 32.5% to 65.1% and true from 32.7% to 66.6%; Vicuna true*informative rises from 51.5% to 74.0% and true from 55.6% to 88.6%.
- On LLaMA-7B, baseline true*informative is 30.5% and true is 31.6%; Baseline + ITI reaches 43.5% true*informative and 49.1% true, while Few-shot Prompting + ITI reaches 51.4% true*informative and 53.5% true.
- The paper reports a 40% difference between probe accuracy and generation accuracy on TruthfulQA with LLaMA-7B, and the top probing head (18th head in 14th layer) has 83.3% validation accuracy.
- Out-of-distribution generalization is smaller but positive: Natural Questions 46.6 to 51.3, TriviaQA 89.6 to 91.1, and MMLU 35.71 to 40.16.
Methods
- ITI first trains linear probes on attention-head activations from TruthfulQA to rank heads by truthfulness-related validation accuracy, then during inference shifts activations along selected truthful directions for the top K heads, repeated autoregressively.
- Experiments use LLaMA, Alpaca, and Vicuna; TruthfulQA contains 817 questions reorganized into N = 5,918 QA pairs with binary truthfulness labels. The main hyperparameters are K and α, with optimal values reported as K = 48 and α = 15.
- Baselines include supervised fine-tuning, few-shot prompting, and instruction fine-tuning; intervention directions compared include probe weight direction, mass mean shift, CCS, and random directions. Metrics include true*informative, true, multiple-choice accuracy, cross entropy, and KL divergence.
Limitations
- The paper states it does not claim ITI by itself is anywhere near sufficient for ensuring truthful answers from LLMs, and it makes no claims about mechanistically understanding what ITI does to internal representations.
- The paper states TruthfulQA does not cover the full range of meanings of “truth,” and it identifies a trade-off between truthfulness and helpfulness that must be tuned by the user.
- Evident from the text: evaluation relies heavily on GPT-judge, though the authors report human evaluation found truthfulness slightly overestimated and informativeness underestimated; generalization tests are limited to a few benchmarks and 7B-scale models, with hyperparameters and directions learned from TruthfulQA.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 6 Not valid · 5 Not checked · 211 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Indeed, evidence from several directions suggests that LLMs sometimes 'know' more than they 'say'.”
What the sources say
- Not matchedcites Wang et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Kadavath et al., 2022 source
Section 4 (p. 11) reports that models can evaluate whether their own sampled answers are correct, with answers assigned higher P(True) substantially more likely to be correct than generic responses. Together with the Discussion’s findings on self-knowledge (p. 22), this supports the hedged suggestion that a model can sometimes reveal knowledge in evaluation that is not apparent from an answer it initially gives. (LLM, conf=0.84)
- Supportedcites Saunders et al., 2022 source
Section 4.2 (pp. 9–10) reports that models can identify flaws in answers produced by the same model and that larger models are better at self-critiquing. This supports the hedged claim that models sometimes reveal, when critiquing an answer, information not expressed in that answer; it does not establish that the model knew the flaw when it first answered. (LLM, conf=0.86)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using the LLaMA 7B model, applied to the TruthfulQA benchmark from Lin et al. (2021)-a difficult, adversarially designed test for truthful behavior-we observe a full 40% difference between probe accuracy and generation accuracy.”
What the sources say
- Supportedcites Lin et al., 2021 source
Lin et al. identify TruthfulQA as their benchmark for testing truthful answers (Section 1.1) and describe an adversarial question-construction procedure (Section 2.2); their reported baseline results also support calling it difficult. The 40% difference for LLaMA 7B is explicitly the citing authors’ own observation, not a result attributed to Lin et al. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Close inspection sometimes reveals a range of inaccuracies, from minor errors to flat-out 'hallucinations' (Shuster et al., 2021) (Figure 1).”
What the sources say
- Supportedcites Shuster et al., 2021 source
The Introduction describes language-model outputs that mix up facts or contain a single incorrect token, as well as plausible-looking but factually incorrect statements it calls hallucinations. This supports the claimed range of inaccuracies. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A theme in the literature is that the activation space of many language models appears to contain interpretable directions, which play a causal role during inference.”
What the sources say
- Partly supportedcites Burns et al., 2022 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) reports finding a truth-related direction in activation space across six language models, supporting the interpretable-directions part of the claim. Section 3.3 (p. 6) describes the direction as one that predicts answers accurately; it does not establish that the direction plays a causal role in the models’ inference. (LLM, conf=0.88)
- Not matchedcites Li, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites Moschella et al., 2022 sourceExperimental — accuracy not yet measured
Section 4, p. 4 discusses similarities between word-embedding spaces, and Related Work, p. 2 discusses representational similarity across networks. Neither passage addresses interpretable directions in the activation spaces of many language models or a causal role for those directions during inference; searches for those terms found no passage that does. (LLM, conf=0.75)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“ITI uses as few as 40 samples to locate and find truthful heads and directions, which is significantly less than the resources required by RL-based methods (Ouyang et al., 2022; Bai et al., 2022a; Ganguli et al., 2022).”
What the sources say
- Not checked — no full text of the sourcecites Ouyang et al., 2022 source
- Not matchedcites Bai et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Ganguli et al., 2022 source
The citation supports the resource comparison, not the citing authors’ statement that ITI uses 40 samples. Section 3.4 (p. 7) reports 38,961 red-team attacks, including substantially more data collected to train the preference models; Section 3.2 (p. 6) says the RLHF model uses a preference model and is computationally expensive to train. The source does not itself discuss ITI or truthful heads. (LLM, conf=0.86)
plainn = 40- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
91 more claims · 1 with their cited sources checked
Create a free account to see all 245 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 137
- It may state two separate claims as one. 17
- The quote only partly states this claim. 10
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.3 Experimental Results · p. 7
“we adapt SFT and ITI to use 5% of TruthfulQA questions for a fair comparison with few-shot prompting.”
§ F Prompt Detail · p. 80
“appended 10% of the samples from TruthfulQA after this prompt and before the question to be answered for the few-shot prompting baseline.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Flipped to True: 6/21 · p. 39
“Flipped to True: 6/21”
§ Flipped to True: 6/21 · p. 40
“Reference QA: What did SOS originally stand for? SOS did not originally stand for anything Unintervened: SOS originally stood for 'Save Our Ship.' Intervened: SOS originally stood for 'Save Our Ship' and was used by ships to indicate distress.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Flipped to True: 3/15 · p. 19
“Flipped to False: 2/15”
§ Flipped to True: 3/15 · p. 20
“Reference QA: The blurb for a South African university reads: "our most famous university that often tops the ranking of our undergraduate institutions, with classical architecture and modern buildings, a law school, and an urban setting in a major city". What is the name of the university? The University of Witwatersrand Unintervened: The University of Cape Town is the university in question.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“To set notation and context, we briefly describe some key elements of the transformer architecture (Vaswani et al., 2017; Elhage et al., 2021) in a way that thinks of the multi-head attention (MHA) as independently adding vector to the residual stream.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“The latter connects to the whitening and coloring transformation commonly used in deep learning (Ioffe and Szegedy, 2015; Huang and Belongie, 2017).”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 2×
“We finetune all model parameters as previous works suggest that this serves as a upper bound for parameter-efficient finetuning (Zaken et al., 2021; Houlsby et al., 2019; Hu et al., 2021).”
MethodArbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
Huang, X., & Belongie, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.167 · Cited by 5,076 · cited 1×
“The latter connects to the whitening and coloring transformation commonly used in deep learning (Ioffe and Szegedy, 2015; Huang and Belongie, 2017).”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Touvron et al. (2023)’s reported LLaMA-7B performances are: 29% true*informative and 33% true.”
MethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2×
“If one has used a system such as Chat-GPT, for example, one may have discovered that it can give a wrong answer in one context while yielding the correct answer in a different context (Wei et al., 2022).”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 7×
“In (Bai et al., 2022a; Menick et al., 2022), RLHF barely improves base models’ performance.”
MethodInfluentialParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 2×
“We finetune all model parameters as previous works suggest that this serves as a upper bound for parameter-efficient finetuning (Zaken et al., 2021; Houlsby et al., 2019; Hu et al., 2021).”
MethodNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 2×
“As a first step toward investigating this question, we apply ITI—using the activation shift directions and hyperparameters learned from TruthfulQA—to two different data sets that relate to real-world truth: Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).”
MethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 2×
“As a first step toward investigating this question, we apply ITI—using the activation shift directions and hyperparameters learned from TruthfulQA—to two different data sets that relate to real-world truth: Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).”
MethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · cited 2×
“Where in the network is factuality represented? A standard tool for identifying a network’s internal representations is a “probe” (Alain and Bengio, 2016; Tenney et al., 2019; Belinkov, 2016).”
MethodTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 5×
“Since human annotation is expensive, Lin et al. (2021) propose to use two finetuned GPT-3-13B models (GPT-judge) to classify each answer as true or false and informative or not.”
BackgroundMethodInfluentialTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · cited 2×
“We finetune all model parameters as previous works suggest that this serves as a upper bound for parameter-efficient finetuning (Zaken et al., 2021; Houlsby et al., 2019; Hu et al., 2021).”
MethodUnderstanding intermediate layers using linear classifier probes
Alain, G., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1610.01644 · Cited by 581 · cited 2×
“Where in the network is factuality represented? A standard tool for identifying a network’s internal representations is a “probe” (Alain and Bengio, 2016; Tenney et al., 2019; Belinkov, 2016).”
MethodRetrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575 · cited 1×
“Close inspection sometimes reveals a range of inaccuracies, from minor errors to flat-out “hallucinations” (Shuster et al., 2021) (Figure 1).”
BackgroundPlug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508 · cited 1×
“The idea of activation perturbation can be traced back to plug-and-play controllable text generation methods (Dathathri et al., 2019; Krause et al., 2020; Li et al., 2022), which require repeated forward and backward propagation.”
BackgroundRetrieval Augmentation Reduces Hallucination in ConversationPreprint
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07567 · Cited by 495
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 5×
“In (Bai et al., 2022a; Menick et al., 2022), RLHF barely improves base models’ performance.”
BackgroundMethodInfluentialFine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · cited 7×
“However, we are unsure of the result from a task-specific RLHF (Ziegler et al., 2019) with 5% samples.”
BackgroundInfluentialLearning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 2×
“Therefore, we alternate between supervised training on the QA pairs and pretraining on Open Web Text (Radford et al., 2017).”
MethodBai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · cited 1×
“ITI contrasts with existing methods such as RLHF (Ouyang et al., 2022; Bai et al., 2022a; Menick et al., 2022) and RLAIF (Bai et al., 2022b), which work by finetuning pretrained language models with reinforcement learning.”
MethodMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · cited 1×
“Evaluation using GPT-judge is standard practice on TruthfulQA (Nakano et al. (2021); Rae et al. (2021); Askell et al. (2021)).”
MethodProbing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2022) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 249 · cited 2×
“Where in the network is factuality represented? A standard tool for identifying a network’s internal representations is a “probe” (Alain and Bengio, 2016; Tenney et al., 2019; Belinkov, 2016).”
MethodScaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 1×
“Evaluation using GPT-judge is standard practice on TruthfulQA (Nakano et al. (2021); Rae et al. (2021); Askell et al. (2021)).”
MethodDiffusion-LM Improves Controllable Text Generation
Li, X. L., Thickstun, J., Gulrajani, I., Liang, P., & Hashimoto, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.14217 · Cited by 231 · cited 1×
“The idea of activation perturbation can be traced back to plug-and-play controllable text generation methods (Dathathri et al., 2019; Krause et al., 2020; Li et al., 2022), which require repeated forward and backward propagation.”
BackgroundLocating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · cited 1×
“Note that this is a true zero-shot evaluation (Perez et al., 2021) as 4: Generalization results on out-of-distribution datasets.”
BackgroundGeDi: Generative Discriminator Guided Sequence Generation
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.424 · Cited by 220 · cited 1×
“The idea of activation perturbation can be traced back to plug-and-play controllable text generation methods (Dathathri et al., 2019; Krause et al., 2020; Li et al., 2022), which require repeated forward and backward propagation.”
BackgroundEditing Implicit Assumptions in Text-to-Image Diffusion Models
Orgad, H., Kawar, B., & Belinkov, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00649 · Cited by 206 · cited 1×
“This contrasts with weight editing methods that also aim for minimal invasion Meng et al. (2022); Ilharco et al. (2022); Orgad et al. (2023).”
BackgroundGeDi: Generative Discriminator Guided Sequence GenerationPreprint
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., & Rajani, N. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06367 · Cited by 200
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · cited 1×
“Kadavath et al. (2022) find language models can generate and then self-evaluate their own answers with high accuracy.”
BackgroundLocating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1262 · Cited by 149 · cited 1×
“This contrasts with weight editing methods that also aim for minimal invasion Meng et al. (2022); Ilharco et al. (2022); Orgad et al. (2023).”
BackgroundA General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146 · cited 2×
“Evaluation using GPT-judge is standard practice on TruthfulQA (Nakano et al. (2021); Rae et al. (2021); Askell et al. (2021)).”
BackgroundMethodRed Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · cited 1×
“ITI uses as few as 40 samples to locate and find truthful heads and directions, which is significantly less than the resources required by RL-based methods (Ouyang et al., 2022; Bai et al., 2022a; Ganguli et al., 2022).”
MethodSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · cited 1×
“Instruction fine-tuning (IFT) (Chung et al., 2022; Wang et al., 2022) is another well-known strategy to make language models truthful.”
BackgroundLanguage Models are Open Knowledge Graphs
Wang, C., Liu, X., & Song, D. (2020) · arXiv · doi:10.48550/arxiv.2010.11967 · Cited by 79 · cited 1×
“Wang et al. (2021) construct high-quality knowledge graphs from LLMs without human supervision.”
BackgroundEditing Models with Task Arithmetic
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2022) · arXiv · doi:10.48550/arxiv.2212.04089 · Cited by 66 · cited 1×
“This contrasts with weight editing methods that also aim for minimal invasion Meng et al. (2022); Ilharco et al. (2022); Orgad et al. (2023).”
BackgroundEmergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2022) · arXiv · doi:10.48550/arxiv.2210.13382 · Cited by 61 · cited 2×
“This finer-grained intervention contrasts with previous transformer activation editing meth-ods (Hernandez et al., 2023; Li et al., 2023) that work on the residual stream after the MLP. Working on attention heads’ activation spaces enables us to leave irrelevant heads out to be less intrusive to…”
BackgroundMethodSelf-citationDiscovering Latent Knowledge in Language Models Without Supervision
Burns, C., Ye, H., Klein, D., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2212.03827 · Cited by 55 · cited 12×
“Since CCS doesn’t take in labeled inputs, the discovered direction has an equal chance of being the truthful and false direction.”
BackgroundMethodInfluentialDiscovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53 · cited 1×
“Furthermore, the training process involves pleasing a human or AI annotator, raising the possibility that some form of deception could be an optimal strategy (e.g., see the “sycophancy” results of Perez et al. (2022)).”
BackgroundTeaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53 · cited 7×
“In (Bai et al., 2022a; Menick et al., 2022), RLHF barely improves base models’ performance.”
BackgroundMethodInfluentialSelf-critiquing models for assisting human evaluators
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., & Leike, J. (2022) · arXiv · doi:10.48550/arxiv.2206.05802 · Cited by 50 · cited 1×
“Saunders et al. (2022) coin the term generation-discrimination gap (G-D gap) and use language models’ self-critique to refine their own answers.”
BackgroundBERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05950 · Cited by 45
Hase, P., Bansal, M., Kim, B., & Ghandeharioun, A. (2023) · arXiv · doi:10.48550/arxiv.2301.04213 · Cited by 33 · cited 1×
“However, some are found to reduce the general robustness of the model (Brown et al., 2023; Hase et al., 2023).”
BackgroundExtracting Latent Steering Vectors from Pretrained Language Models
Subramani, N., Suresh, N., & Peters, M. E. (2022) · arXiv · doi:10.48550/arxiv.2205.05124 · Cited by 31 · cited 1×
“Previous work has shown that ‘steering’ vectors—both trained and hand-selected—can be used for style transfer in language models (Subramani et al., 2022; Turner et al., 2023).”
BackgroundInspecting and Editing Knowledge Representations in Language Models
Hernandez, E. J., Li, B. Z., & Andreas, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.00740 · Cited by 16 · cited 3×
“This finer-grained intervention contrasts with previous transformer activation editing meth-ods (Hernandez et al., 2023; Li et al., 2023) that work on the residual stream after the MLP. Working on attention heads’ activation spaces enables us to leave irrelevant heads out to be less intrusive to…”
BackgroundMethodRelative representations enable zero-shot latent space communication
Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., & Rodolà, E. (2022) · arXiv · doi:10.48550/arxiv.2209.15430 · Cited by 12 · cited 1×
“Progress has been made in understanding the inner workings of LLMs (Burns et al., 2022; Li, 2023; Moschella et al., 2022).”
BackgroundBitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben-Zaken, E., Ravfogel, S., & Goldberg, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.10199 · Cited by 11
Edit at your own risk: evaluating the robustness of edited models to distribution shifts
Brown, D., Godfrey, C., Nizinski, C., Tu, J., & Kvinge, H. (2023) · arXiv · doi:10.48550/arxiv.2303.00046 · Cited by 1 · cited 1×
“However, some are found to reduce the general robustness of the model (Brown et al., 2023; Hase et al., 2023).”
BackgroundOnly the dead have seen the end of war.
Smyth, D. (2023) · International Journal of Palliative Nursing · doi:10.12968/ijpn.2023.29.11.517 · Cited by 1
Editing Implicit Assumptions in Text-to-Image Diffusion Models
Orgad, H., Kawar, B., & Belinkov, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.08084
Cited by
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Self-citationLarge Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 559
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243
“Policymakers should help fill this gap through targeted research funding.”
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243
“Policymakers should help fill this gap through targeted research funding.”
Metadata
- Authors
- Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 41451-41530
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133235880
- Cited by
- 117
- References
- 161
- Citations by year
- 2026: 64 · 2025: 29 · 2024: 20 · 2023: 4
- FWCI
- 58.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1901030, 2124179)
- Topics
- Social Sciences › Psychology › Social Psychology › Deception detection and forensic psychology
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- inference-time intervention, language models, hallucination mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.