Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Open accessAbstract
Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods. Despite impressive recent progress, we demonstrate that Transformer-XL language models perform worse in the realistic setup of predicting future utterances from beyond their training period, and that model performance becomes increasingly worse with time. We find that, while increasing model size alone -- a key driver behind recent progress -- does not solve this problem, having models that continually update their knowledge with new information can indeed mitigate this performance degradation over time. Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world. We publicly release our dynamic, streaming language modelling benchmarks for WMT and arXiv to facilitate language model evaluation that takes temporal dynamics into account.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Knowledge within dynamic environments is typically conditional rather than eternally valid.
“Knowledge within environments of a dynamic nature is typically conditional rather than eternally valid (Lazaridou et al., 2021; Jang et al., 2022; Ko et al., 2024).”
From From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026) · cites this paper as Lazaridou et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,649 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 4× · see citing works
“For instance, recently there has been renewed public support and interest for social justice movements in 2020, such as the #BlackLivesMatter movement (Cohn and Quealy, 2020).”
BackgroundInfluentialRecurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · see citing works
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 2× · see citing works
“4 We find that increasing model size alone—a key driver behind recent LM progress (Kaplan et al., 2020)—is not a solution for the temporal generalization problem (§4): Larger models suffer from the same performance degradation with time, and a smaller model trained on more recent data can outperform…”
BackgroundTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Domain adaptation with structural correspondence learning
Blitzer, J., McDonald, R., & Pereira, F. (2006) · Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06 · doi:10.3115/1610075.1610094 · Cited by 1,543 · see citing works
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · see citing works
Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1141 · Cited by 958 · see citing works
Detecting Change in Data Streams
Kifer, D., Ben-David, S., & Gehrke, J. E. (2004) · Elsevier eBooks · doi:10.1016/b978-012088469-8.50019-x · Cited by 809 · see citing works
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · see citing works
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 2× · see citing works
“To illustrate this point, we perform a reading comprehension experiment using NewsQA (Trischler et al., 2017), where the evidence documents are presented together with the questions into the prefix of the model.”
MethodAdversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
Thrun, S., & Mitchell, T. M. (1995) · Robotics and Autonomous Systems · doi:10.1016/0921-8890(95)00004-y · Cited by 523 · see citing works
Carbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486 · see citing works
Embracing Change: Continual Learning in Deep Neural Networks
Hadsell, R., Rao, D., Rusu, A. A., & Pascanu, R. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.004 · Cited by 467 · see citing works
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1704.05179 · Cited by 359 · see citing works
Pretrained Transformers Improve Out-of-Distribution Robustness
Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., & Song, D. X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.244 · Cited by 312 · see citing works
Adaptively Sparse Transformers
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1223 · Cited by 211 · see citing works
MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims
Augenstein, I., Lioma, C., Wang, D., Lima, L. C., Hansen, C., Hansen, C., & Simonsen, J. G. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1475 · Cited by 198 · see citing works
Editing Factual Knowledge in Language Models
De Cao, N., Aziz, W., & Titov, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.522 · Cited by 180 · see citing works
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Lewis, P., Stenetorp, P., & Riedel, S. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.86 · Cited by 145 · see citing works
Adaptive concept drift detection
Dries, A., & Rückert, U. (2009) · Statistical Analysis and Data Mining The ASA Data Science Journal · doi:10.1002/sam.10054 · Cited by 126 · see citing works
Mitchell, T. M., Cohen, W. W., Junior, E. R. H., Talukdar, P. P., Betteridge, J., Carlson, A., Dalvi Mishra, B., Gardner, M., Kisiel, B., Krishnamurthy, J., et al. (2015) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v29i1.9498 · Cited by 105 · see citing works
Video Action Transformer Network
Girdhar, R., Carreira, J., Doersch, C., & Zisserman, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00033 · Cited by 93 · see citing works
Deep Neural Models of Semantic Shift
Rosenfeld, A., & Erk, K. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1044 · Cited by 93 · see citing works
(2019) · doi:10.18653/v1/d19-1432 · Cited by 88 · see citing works
The LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87 · see citing works
Self-citationWe Need To Talk About Random Splits
Søgaard, A., Ebert, S., Bastings, J., & Filippova, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.156 · Cited by 81 · see citing works
What Kind of Language Is Hard to Language-Model?
Mielke, S. J., Cotterell, R., Gorman, K., et al. (2019) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p19-1491 · Cited by 69 · see citing works
Adaptive Semiparametric Language Models
Yogatama, D., d’Autume, C. D. M., & Kong, L. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00371 · Cited by 58 · see citing works
Self-citationDynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.186 · Cited by 56 · see citing works
Editable Neural NetworksPreprint
Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S., & Babenko, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00345 · Cited by 49 · see citing works
Modifying Memories in Transformer Models
Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., & Kumar, S. (2020) · arXiv · doi:10.48550/arxiv.2012.00363 · Cited by 48 · see citing works
Temporally-Informed Analysis of Named Entity Recognition
Rijhwani, S., & Preoțiuc-Pietro, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.680 · Cited by 48 · see citing works
Cross-Domain Generalization of Neural Constituency Parsers
Fried, D., Kitaev, N., & Klein, D. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1031 · Cited by 41 · see citing works
Dynamic Evaluation of Transformer Language Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2019) · arXiv · doi:10.48550/arxiv.1904.08378 · Cited by 30 · see citing works
Exponential Reservoir Sampling for Streaming Language Models
Osborne, M., Lall, A., & Durme, B. V. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-2112 · Cited by 21 · see citing works
Back to the Future – Temporal Adaptation of Text Representations
Bjerva, J., Kouw, W. M., & Augenstein, I. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6240 · Cited by 16 · see citing works
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, C., Mikolov, T., Schuster, M., et al. (2013) · arXiv · doi:10.48550/arxiv.1312.3005
Editing Factual Knowledge in Language Models
Cao, N. D., Aziz, W., & Titov, I. (2021) · arXiv · doi:10.48550/arxiv.2104.08164
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Lewis, P., Stenetorp, P., & Riedel, S. (2020) · arXiv · doi:10.48550/arxiv.2008.02637
Crowdsourcing and annotating NER for Twitter #drift
Fromreide, H., Hovy, D., & Søgaard, A. (2014) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/3mepts35xng8
Streaming for large scale NLP: language modeling
Goyal, A., Daumé, H., & Venkatasubramanian, S. (2009) · Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics on - NAACL '09 · doi:10.3115/1620754.1620829
A dynamic language model for speech recognition
Jelinek, F., Merialdo, B., Roukos, S., & Strauss, M. (1991) · Proceedings of the workshop on Speech and Natural Language - HLT '91 · doi:10.3115/112405.112464
DynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15349
Sentiment analysis under temporal shift
Lukes, J., & Søgaard, A. (2018) · Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w18-6210
Dynamic Language Models for Streaming Text
Yogatama, D., Wang, C., Routledge, B. R., Smith, N. A., & Xing, E. P. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00175
Self-citation
Flagged first, then by citations
Research works printed but not linked: 27
- Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In Proc. of ICLR , 2018.
- Amittai Axelrod, Xiaodong He, and Jianfeng Gao. Domain adaptation via pseudo in-domain data selection. In Proc. of EMNLP , 2011.
- Manuel Baena-Garcıa, José del Campo-Ávila, Raúl Fidalgo, Albert Bifet, R Gavalda, and R MoralesBueno. Early drift detection method. In Fourth international workshop on knowledge discovery from data streams , volume 6, pages 77-86, 2006.
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR , 2015.
- Johannes Bjerva, Wouter M Kouw, and Isabelle Augenstein. Back to the future-sequential alignment of text representations. In 34rd AAAI Conference on Artificial Intelligence , pages 1909-03464. Association for the Advancement of Artificial Intelligence, 2019.
- Kai-Wei Chang, Vinod Prabhakaran, and Vicente Ordonez. Bias and fairness in natural language processing. In Proc. of EMNLP-IJCNLP: Tutorial Abstracts , 2019.
- Hal Daumé III. Frustratingly easy domain adaptation. In Proc. of ACL , 2007.
- Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic Memory in Lifelong Language Learning. In Proc. of NeurIPS , 2019.
- Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. Deep learning for event-driven stock prediction. In Proc. of IJCAI , 2015.
- Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences , 3 (4), 1999.
- Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval-augmented language model pre-training. In Proc. of ICML , 2020.
- Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. of ACL , 2017b.
- Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In Proc. of ICLR , 2020.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In Proc. of ICLR , 2020.
- Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In Proc. of ICML , 2018.
- Abby Levenberg, Chris Callison-Burch, and Miles Osborne. Stream-based translation models for statistical machine translation. In Proc. of NAACL-HLT , 2010.
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Proc. of NeurIPS , 2020b.
- David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Proc. of NeurIPS , 2017.
- Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering , 31(12), 2019.
- Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. LAMOL: LAnguage MOdeling for Lifelong Language Learning. In Proc.of ICLR , 2020.
- Terrence Szymanski. Temporal Word Analogies: Identifying Lexical Replacement with Diachronic Word Embeddings. In Proc. of ACL , 2017.
- James Thorne and Andreas Vlachos. Automated fact checking: Task formulations, methods and future directions. In Proc. of ICCL , 2018.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurIPS , 2017.
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Proc. of NeurIPS , 2015.
- Chong Wang, David Blei, and David Heckerman. Continuous time dynamic topic models. In Proc. of UAI , 2008.
- Zi Yin, Vin Sachidananda, and Balaji Prabhakar. The global anchor method for quantifying linguistic shifts and domain adaptation. Proc. of NeurIPS , 2018.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Proc. of NeurIPS , 2019.
Cited by
Sorted by citations
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“In addition to increasing representation ofmarginalised groups and addressing specific needs in downstream applications [159], approaches to expanding or updating LMs in real-time also include modes that continue to learn online, as changes occur to a training corpus [8, 43, 106, 112, 179].”
BackgroundTowards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366 · see citing works
Internet-Augmented Dialogue Generation
Komeili, M., Shuster, K., & Weston, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.579 · Cited by 192 · see citing works
Retrieval-based Language Models and Applications
Asai, A., Min, S., Zhong, Z., & Chen, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) · doi:10.18653/v1/2023.acl-tutorials.6 · Cited by 81 · see citing works
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71 · see citing works
Towards Continual Knowledge Learning of Language Models
Jang, J., Ye, S., Yang, S., Shin, J., Han, J., Kim, G., Choi, S. J., & Seo, M. (2021) · arXiv · doi:10.48550/arxiv.2110.03215 · Cited by 49 · see citing works
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Wang, P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang, F., & Chen, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14768 · Cited by 19 · see citing works
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3 · see citing works
“Knowledge within environments of a dynamic nature is typically conditional rather than eternally valid (Lazaridou et al., 2021; Jang et al., 2022; Ko et al., 2024).”
Do as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1 · see citing works
“Furthermore, the parameterized knowledge can become inaccurate or outdated [66, 67], potentially leading to errors and hallucinations [68–70].”
- Authors
- Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., d'Autume, C. D. M., Kocisky, T., Ruder, S., Yogatama, D., Cao, K., Young, S., & Blunsom, P.
- Affiliations
- Google (United States) (US)
- University of Oxford (GB)
- Italian Institute of Technology (IT)
- Imperial College London (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 34
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.01951
- OpenAlex
- W3209540659
- Cited by
- 90
- References
- 64
- Citations by year
- 2026: 1 · 2025: 11 · 2024: 20 · 2023: 38 · 2022: 20
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- temporal generalization, language model evaluation, Transformer-XL, continual learning, model size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
