Understanding Learning Dynamics Of Language Models with SVCCA
Open accessAbstract
Naomi Saphra, Adam Lopez. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
With an expressive-enough probe, there is a danger of the probe computing or memorizing the desired feature rather than extracting it.
“With an expressive-enough probe, there is a danger of the probe computing or memorizing the desired feature that one is looking for, rather than extracting (Pimentel et al., 2020a;Saphra and Lopez, 2019).”
From Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023) · cites this paper as Saphra and Lopez, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 19
- Afra Alishahi, Marie Barking, and Grzegorz Chrupała. 2017. Encoding of phonology in a recurrent neural model of grounded speech. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) , pages 368-378.
- Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do Neural Machine Translation Models Learn about Morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 861-872, Vancouver, Canada. Association for Computational Linguistics.
- Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. 2018. Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks. CoRR , abs/1801.07772. · arXiv
- Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic Tagging with Deep Residual Networks. In COLING .
- Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs Encode Soft Hierarchical Syntax. arXiv:1805.04218 [cs] . ArXiv: 1805.04218. · arXiv
- Johan Bos, Valerio Basile, Kilian Evang, Noortje J Venhuizen, and Johannes Bjerva. 2017. The groningen meaning bank. In Handbook of linguistic annotation , pages 463-496. Springer.
- Remi Tachet des Combes, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. 2018. On the Learning Dynamics of Deep Neural Networks. arXiv:1809.06848 [cs, stat] . ArXiv: 1809.06848. · arXiv
- Jonathan Frankle and Michael Carbin. 2018. The Lottery Ticket Hypothesis: Training Pruned Neural Networks. arXiv:1803.03635 [cs] . ArXiv: 1803.03635. · arXiv
- Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information. arXiv:1808.08079 [cs] . ArXiv: 1808.08079. · arXiv
- Yova Kementchedjhieva and Adam Lopez. 2018. Indicatements that character language models learn English morpho-syntactic units and regularities. In Proc. of Workshop on Analyzing and interpreting neural networks for NLP .
- Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context. arXiv:1805.04623 [cs] . ArXiv: 1805.04623. · arXiv
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. arXiv:1611.01368 [cs] . ArXiv: 1611.01368. · arXiv
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems 26 , pages 3111-3119. Curran Associates, Inc.
- Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. arXiv:1806.05759 [cs, stat] . ArXiv: 1806.05759. · arXiv
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365 . · arXiv
- Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. arXiv:1706.05806 [cs, stat] . ArXiv: 1706.05806. · arXiv
- Ravid Shwartz-Ziv and Naftali Tishby. 2017. Opening the Black Box of Deep Neural Networks via Information. arXiv:1703.00810 [cs] . ArXiv: 1703.00810. · arXiv
- Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 . · arXiv
- Kelly W. Zhang and Samuel R. Bowman. 2018. Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis. arXiv:1809.10040 [cs] . ArXiv: 1809.10040. · arXiv
Cited by
Sorted by citations
Designing and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1275 · Cited by 393 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
Emergent Linear Representations in World Models of Self-Supervised Sequence Models
Nanda, N., Lee, A., & Wattenberg, M. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.2 · Cited by 55 · see citing works
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
- Authors
- Saphra, N., & Lopez, A.
- Venue
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Published
- 2019
- Pages
- 3257-3267
- OpenAlex
- W2950976310
- Cited by
- 31
- References
- 0
- Citations by year
- 2026: 1 · 2024: 5 · 2023: 4 · 2022: 2 · 2021: 7 · 2020: 5 · 2019: 7
- FWCI
- 2.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- language models, learning dynamics, causal discovery, latent representations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
