The DOI matches its DataCite record.
Time Machine GPT
Open accessAbstract
Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora, reflecting the lack of datasets with temporal metadata. This approach is not aligned with the evolving nature of language. Conventional methods for creating temporally adapted language models often depend on further pre-training static models on time-specific data. This paper presents a new approach: a series of point-in-time LLMs called Time Machine GPT (TiMaGPT), specifically designed to be nonprognosticative. This ensures they remain uninformed about future factual information and linguistic changes. This strategy is beneficial for understanding language evolution and is of critical importance when applying models in dynamic contexts, such as time-series forecasting, where foresight of future information can prove problematic. We provide access to both the models and training datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“leakage effects scale with model size unless explicitly mitigated (Kong et al., 2025; Drinkall et al., 2024).”
From Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance (2026) · cites this paper as Drinkall et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Their approach demonstrates that explicit temporal constraints can prevent future information leakage.”
From Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance (2026) · cites this paper as Drinkall et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 30
- Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
- Joan Bybee. 2015. Language Change. Cambridge Text- books in Linguistics. Cambridge University Press. · DOI
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
- Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257- 273. · DOI · Paper page
- Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
- Felix Drinkall, Stefan Zohren, and Janet Pierrehumbert. 2022. Forecasting COVID-19 caseloads using unsu- pervised embedding clusters of social media posts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1471-1484, Seattle, United States. Association for Computational Linguistics. · DOI
- Jonathan Dunn. 2023. Variation and instability in dialect-based embedding spaces. In Tenth Work- shop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 67-77, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cyril Goutte, Yunli Wang, Fangming Liao, Zachary Zanussi, Samuel Larkin, and Yuri Grinberg. 2018. EuroGames16: Evaluating change detection in on- line conversation. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
- Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics. · DOI · Paper page
- William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany. Association for Com- putational Linguistics. · DOI
- Bastian Herre. 2023. Identifying ideologues: A global dataset on political leaders, 1945-2020. British Jour- nal of Political Science, 53(2):740-748. · DOI
- Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021a. Dynamic contextualized word em- beddings. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 6970-6984, Online. Association for Computa- tional Linguistics. · DOI
- Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021b. Superbizarre is not superb: Deriva- tional morphology improves BERT's interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3594-3608, Online. Association for Computational Linguistics. · DOI
- Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computa- tional Linguistics.
- Jona Kräenbring, Tika Monzon Penza, Joanna Gutmann, Susanne Mühlich, Oliver Zolk, Leszek Wojnowski, Renke Maas, Stefan Engelhardt, and Antonio Sarikas. 2014. Accuracy and completeness of drug informa- tion in wikipedia: A comparison with standard text- books of pharmacology. PLoS ONE, 9:e106930. · DOI
- Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 1384-1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2017. Tracing armed conflicts with diachronic word embed- ding models. In Proceedings of the Events and Sto- ries in the News Workshop, pages 31-36, Vancouver, Canada. Association for Computational Linguistics. · DOI
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019a. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240. · DOI · Paper page
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019b. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240. · DOI · Paper page
- Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
- Alexandra Luccioni and Joseph Viviano. 2021. What's in the box? an analysis of undesirable content in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 182-189, Online. Association for Computational Linguistics. · DOI · Paper page
- Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2020. Leveraging contextual embeddings for detect- ing diachronic semantic shift. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 4811-4819, Marseille, France. European Language Resources Association.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- tems, volume 26.
- Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics. · DOI · Paper page
- Janet B Pierrehumbert. 2012. The dynamic lexicon. Handbook of laboratory phonology, pages 173-183.
- Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400-2412, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. · DOI
- Wenjun Qiu and Yang Xu. 2022a. Histbert: A pre- trained language model for diachronic lexical seman- tic analysis. · DOI
- Wenjun Qiu and Yang Xu. 2022b. Histbert: A pre- trained language model for diachronic lexical seman- tic analysis. · DOI
- Qiongkai Xu, Lizhen Qu, Zeyu Gao, and Gholamreza Haffari. 2020. Personal information leakage detec- tion in conversations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6567-6580, On- line. Association for Computational Linguistics. · DOI
- Ziqian Zeng, Xin Liu, and Yangqiu Song. 2018. Biased random walk based social regularization for word embeddings. In International Joint Conference on Artificial Intelligence (IJCAI), volume 27.
Cited by
Sorted by citations
Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance
Benhenda, M. (2026) · arXiv
Metadata
- Authors
- Drinkall, F., Rahimikia, E., Pierrehumbert, J. B., & Zohren, S.
- Affiliations
- University of Oxford (GB)
- University of Manchester (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 3281-3292
- arXiv
- 2404.18543
- OpenAlex
- W4401043421
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 1
- FWCI
- 1.73
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Research Explorer (The University of Manchester) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/T023333/1)
- Economic and Social Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- temporal metadata, language evolution, time series forecasting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Time Machine GPT. https://app.cochat.ai/papers/10.48550/arxiv.2404.18543
- The paper
- Drinkall, F., Rahimikia, E., Pierrehumbert, J. B., & Zohren, S. (2024). Time Machine GPT. arXiv. https://doi.org/10.48550/arxiv.2404.18543
