The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Time-Aware Language Models as Temporal Knowledge Bases
Open accessAbstract
Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently “refreshed” as new data arrives, without the need for retraining from scratch.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LMs have an unawareness of the progression of time.
“and an unawareness of the progression of time (Dhingra et al., 2022).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Dhingra et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614
Editing Factual Knowledge in Language Models
De Cao, N., Aziz, W., & Titov, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.522 · Cited by 180
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Lewis, P., Stenetorp, P., & Riedel, S. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.86 · Cited by 145
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., d'Autume, C. D. M., Kocisky, T., Ruder, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01951 · Cited by 90
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Jiang, Z., Anastasopoulos, A., Araki, J., Ding, H., & Neubig, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.479 · Cited by 88
Flagged first, then by citations
Research works printed but not linked: 34
- Bamler, 2017, Dynamic word embeddings, 380
- Delasalles, 2019, Learning dynamic author representations with temporal language models, 120, 10.1109/ICDM.2019.00022 · DOI
- Dubossarsky, 2019, Time-out: Temporal referencing for robust modeling of lexical semantic change, 457, 10.18653/v1/P19-1044 · DOI
- Florio, 2020, Time of your hate: The challenge of time in hate speech detection on social media, Applied Sciences, 10, 10.3390/app10124180 · DOI
- Frermann, 2016, A Bayesian model of diachronic meaning change, Transactions of the Association for Computational Linguistics, 4, 31, 10.1162/tacl_a_00081 · DOI
- Giulianelli, 2020, Analysing lexical semantic change with contextualised word representations, 3960, 10.18653/v1/2020.acl-main.365 · DOI
- Guu, 2020, Retrieval augmented language model pre-training, 3929
- Hamilton, 2016, Diachronic word embeddings reveal statistical laws of semantic change, 1489, 10.18653/v1/P16-1141 · DOI
- Hombaiah, 2021, Dynamic language models for continuously evolving content, 10.1145/3447548.3467162 · DOI
- Huang, 2018, Examining temporality in document classification, 694, 10.18653/v1/P18-2110 · DOI
- Huang, 2019, Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models, 4113, 10.18653/v1/P19-1403 · DOI
- Logan, 2021, Cutting down on prompts and parameters: Simple few-shot learning with language models, CoRR
- Jaidka, 2018, Diachronic degradation of language models: Insights from social media, 195, 10.18653/v1/P18-2032 · DOI
- Jia, 2018, Tempquestions: A benchmark for temporal question answering, 1057, 10.1145/3184558.3191536 · DOI
- Kassner, 2021, Multilingual LAMA: Investigating knowledge in multilingual pretrained language models, 3250, 10.18653/v1/2021.eacl-main.284 · DOI
- Krause, 2018, Dynamic evaluation of neural sequence models, 2766
- Lazaridou, 2020, Discovering biased news articles leveraging multiple human annotations, 1268
- Lee, 2020, Language models as fact checkers?, 36, 10.18653/v1/2020.fever-1.5 · DOI
- Liu, 2021, Newsembed: Modeling news through pre-trained document representations, 1076, 10.1145/3447548.3467392 · DOI
- Lukes, 2018, Sentiment analysis under temporal shift, 65, 10.18653/v1/W18-6210 · DOI
- Ning, 2020, TORQUE: A reading comprehension dataset of temporal ordering questions, 1158, 10.18653/v1/2020.emnlp-main.88 · DOI
- Petroni, 2019, Language models as knowledge bases?, 2463, 10.18653/v1/D19-1250 · DOI · Paper page
- Ringgaard, 2017, SLING: A framework for frame semantic parsing, CoRR
- Roberts, 2020, How much knowledge can you pack into the parameters of a language model?, 5418, 10.18653/v1/2020.emnlp-main.437 · DOI · Paper page
- Röttger, 2021, Temporal adaptation of BERT and performance on downstream document classification: Insights from social media, 2400, 10.18653/v1/2021.findings-emnlp.206 · DOI
- Saxena, 2021, Question answering over temporal knowledge graphs, 6663, 10.18653/v1/2021.acl-long.520 · DOI · Paper page
- Sinitsin, 2020, Editable neural networks
- Strubell, 2019, Energy and policy considerations for deep learning in NLP, 3645, 10.18653/v1/P19-1355 · DOI · Paper page
- Vaswani, 2017, Attention is all you need
- Vrandečić, 2014, Wikidata: A free collaborative knowledgebase, Commun. ACM, 57, 78, 10.1145/2629489 · DOI
- Wijaya, 2011, Understanding semantic change of words over centuries, 35, 10.1145/2064448.2064475 · DOI
- Zhou, 2019, “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding, 3363, 10.18653/v1/D19-1332 · DOI · Paper page
- Zhou, 2021, Temporal reasoning on implicit events from distant supervision, 1361, 10.18653/v1/2021.naacl-main.107 · DOI
- Zhu, 2020, Modifying memories in transformer models, CoRR
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
TimeLMs: Diachronic Language Models from Twitter
Loureiro, D., Barbieri, F., Neves, L., Espinosa Anke, L., & Camacho-collados, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.25 · Cited by 263
Can We Edit Factual Knowledge by In-Context Learning?
Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., & Chang, B. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.296 · Cited by 54
Towards Continual Knowledge Learning of Language Models
Jang, J., Ye, S., Yang, S., Shin, J., Han, J., Kim, G., Choi, S. J., & Seo, M. (2021) · arXiv · doi:10.48550/arxiv.2110.03215 · Cited by 49
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H., Yao, Z., Zhang, X., Li, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.09296 · Cited by 29
“We believe the ideal approach is to do evaluations on newly emerging data and maintain a continuously evolving benchmark, like the attempts that include time-sensitive evolving data (Kasai et al., 2022; Dhingra et al., 2022).”
BackgroundTRAM: Benchmarking Temporal Reasoning for Large Language Models
Wang, Y., & Zhao, Y. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.382 · Cited by 24
DSI++: Updating Transformer Memory with New Documents
Mehta, S., Gupta, J., Tay, Y., Dehghani, M., Tran, V., Rao, J., Najork, M., Strubell, E., & Metzler, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.510 · Cited by 21
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
RET-LLM: Towards a General Read-Write Memory for Large Language Models
Modarressi, A., Imani, A., Fayyaz, M., & Schütze, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14322 · Cited by 8
Metadata
- Authors
- Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W.
- Affiliations
- Google (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2022
- Volume
- 10
- Pages
- 257-273
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3174266714
- Cited by
- 212
- References
- 13
- Citations by year
- 2026: 57 · 2025: 38 · 2024: 29 · 2023: 46 · 2022: 36
- FWCI
- 9.97
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- temporal knowledge bases, diagnostic dataset, forecast calibration, model updating
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
