CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Does Transformer Interpretability Transfer to RNNs?

Paulo, Gonçalo, Marshall, Thomas, Belrose, Nora·2024·arXiv
DOI 10.48550/arxiv.2404.05971arXiv 2404.05971
Open this paper in CoChat — free account

Abstract

Recent advances in recurrent neural network architectures, such as Mamba and RWKV, have enabled RNNs to match or exceed the performance of equal-size transformers in terms of language modeling perplexity and downstream evaluations, suggesting that future systems may be built on completely new architectures. In this paper, we examine if selected interpretability methods originally designed for transformer language models will transfer to these up-and-coming recurrent architectures. Specifically, we focus on steering model outputs via contrastive activation addition, on eliciting latent predictions via the tuned lens, and eliciting latent knowledge from models fine-tuned to produce false outputs under certain conditions. Our results show that most of these techniques are effective when applied to RNNs, and we show that it is possible to improve some of them by taking advantage of RNNs' compressed state.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.05971

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 22

  1. Belrose, N.; Furman, Z.; Smith, L.; Halawi, D.; Ostrovsky, I.; McKinney, L.; Biderman, S.; and Steinhardt, J. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112. · arXiv
  2. Burns, C.; Ye, H.; Klein, D.; and Steinhardt, J. 2022. Dis- covering latent knowledge in language models without su- pervision. arXiv preprint arXiv:2212.03827. · arXiv
  3. Dey, N.; Soboleva, D.; Al-Khateeb, F.; Yang, B.; Pathria, R.; Khachane, H.; Muhammad, S.; Myers, R.; Steeves, J. R.; Vassilieva, N.; et al. 2023. Btlm-3b-8k: 7b parame- ter performance in a 3b parameter model. arXiv preprint arXiv:2309.11568. · arXiv
  4. Fisher, R. A. 1936. THE USE OF MULTIPLE MEASURE- MENTS IN TAXONOMIC PROBLEMS. Annals of Eugen- ics, 7(2): 179-188.
  5. Gao, L.; Biderman, S.; Black, S.; Golding, L.; Hoppe, T.; Foster, C.; Phang, J.; He, H.; Thite, A.; Nabeshima, N.; et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.00027. · arXiv
  6. Ghandeharioun, A.; Caciularu, A.; Pearce, A.; Dixon, L.; and Geva, M. 2024. Patchscope: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102. · arXiv
  7. Gu, A.; and Dao, T. 2023. Mamba: Linear-time se- quence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. · arXiv
  8. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. · arXiv
  9. Jastrzebski, S.; Arpit, D.; Ballas, N.; Verma, V.; Che, T.; and Bengio, Y. 2017. Residual connections encourage iterative inference. arXiv preprint arXiv:1710.04773. · arXiv
  10. Mallen, A.; and Belrose, N. 2023. Eliciting Latent Knowl- edge from Quirky Language Models. arXiv preprint arXiv:2312.01037. · arXiv
  11. Marks, S.; and Tegmark, M. 2023. The Geometry of Truth: Emergent Linear Structure in Large Language Model Rep- resentations of True/False Datasets. arXiv:2310.06824. · arXiv
  12. Martin, E.; and Cundy, C. 2017. Parallelizing linear re- current neural nets over sequence length. arXiv preprint arXiv:1709.04057. · arXiv
  13. nostalgebraist. 2020. interpreting GPT: the logit lens. Less- Wrong. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774. · arXiv
  14. Pal, K.; Sun, J.; Yuan, A.; Wallace, B. C.; and Bau, D. 2023. Future lens: Anticipating subsequent tokens from a single hidden state. arXiv preprint arXiv:2311.04897. · arXiv
  15. Peng, B.; Alcaide, E.; Anthony, Q.; Albalak, A.; Arcadinho, S.; Cao, H.; Cheng, X.; Chung, M.; Grella, M.; GV, K. K.; et al. 2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048. · arXiv
  16. Peng, B.; Goldstein, D.; Anthony, Q. G.; Albalak, A.; Al- caide, E.; Biderman, S.; Cheah, E.; Ferdinan, T.; GV, K. K.; Hou, H.; Krishna, S.; Jr., R. M.; Muennighoff, N.; Obeid, F.; Saito, A.; Song, G.; Tu, H.; Zhang, R.; Zhao, B.; Zhao, Q.; Zhu, J.; and Zhu, R.-J. 2024. Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence. In First Conference on Language Modeling.
  17. Perez, E.; Ringer, S.; Lukošiūtė, K.; Nguyen, K.; Chen, E.; Heiner, S.; Pettit, C.; Olsson, C.; Kundu, S.; Kada- vath, S.; et al. 2022. Discovering language model be- haviors with model-written evaluations. arXiv preprint arXiv:2212.09251. · arXiv
  18. Rimsky, N.; Gabrieli, N.; Schulz, J.; Tong, M.; Hubinger, E.; and Turner, A. M. 2023. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681. · arXiv
  19. Turner, A.; Thiergart, L.; Udell, D.; Leech, G.; Mini, U.; and MacDiarmid, M. 2023. Activation addition: Steer- ing language models without optimization. arXiv preprint arXiv:2308.10248. · arXiv
  20. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. At- tention is all you need. Advances in neural information pro- cessing systems, 30.
  21. Wang, K.; Variengien, A.; Conmy, A.; Shlegeris, B.; and Steinhardt, J. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593. · arXiv
  22. Zou, A.; Phan, L.; Chen, S.; Campbell, J.; Guo, P.; Ren, R.; Pan, A.; Yin, X.; Mazeika, M.; Dombrowski, A.-K.; et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. · arXiv

Cited by

Sorted by citations

See all 1

Metadata

Authors
Paulo, G., Marshall, T., & Belrose, N.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2404.05971
arXiv
2404.05971
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Does Transformer Interpretability Transfer to RNNs? https://app.cochat.ai/papers/10.48550/arxiv.2404.05971
The paper
Paulo, G., Marshall, T., & Belrose, N. (2024). Does Transformer Interpretability Transfer to RNNs? arXiv. https://doi.org/10.48550/arxiv.2404.05971