Lookahead Bias in Pretrained Language Models
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
Empirical analysis that uses outputs from pretrained language models can be subject to a form of temporal lookahead bias . This bias arises when a language model’s pre-training data contains information about the future, which then leaks into analysis that should only use information from the past. In this paper we develop direct tests for lookahead bias, based on the assumption that some events are unpredictable given a prespecified information set. Using these tests, we find evidence of lookahead bias in two applications of language models to social science: Predicting risk factors from corporate earnings calls and predicting election winners from candidate biographies. We additionally discuss the limitations of prompting-based approaches to counteract this bias. The issues we raise can be addressed by using models whose pretraining data is free of survivorship bias and contains only language produced prior to the analysis period of interest
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This sample period is selected to avoid look-ahead bias and ensure an out-of-sample evaluation of LLMs like ChatGPT, as the version we evaluate was trained on data up until September 2021.”
From Can ChatGPT forecast stock price movements? Return predictability and large language models (2026) · cites this paper as Sarkar and Vafa (2024)
In the citing paper
“This sample period ensures that our evaluation is out-of-sample as the training data of the ChatGPT models stops in September 2021 (e.g., Sarkar and Vafa (2024) and Lopez-Lira, Tang, and Zhu (2025)).”
From Can ChatGPT forecast stock price movements? Return predictability and large language models (2026) · cites this paper as Sarkar and Vafa (2024)
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Gentzkow, M., Kelly, B., & Taddy, M. (2019) · Journal of Economic Literature · doi:10.1257/jel.20181020 · Cited by 1,236
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629
AI recognition of patient race in medical imaging: a modelling study
Gichoya, J. W., Banerjee, I., Bhimireddy, A. R., Burns, J. L., Celi, L. A., Chen, L.-C., Correa, R., Dullerud, N., Ghassemi, M., Huang, S.-C., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00063-2 · Cited by 611 · cited 1×
“This evidence is consistent with evidence in other domains that shows that machine learning models can infer information from “degraded” data, even if humans cannot (e.g. Gichoya et al., 2022).”
ResultSurvivor Bias and Mutual Fund Performance
Elton, E. J., Gruber, M. J., & Blake, C. R. (1996) · Review of Financial Studies · doi:10.1093/rfs/9.4.1097 · Cited by 595
Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015) · American Economic Review · doi:10.1257/aer.p20151023 · Cited by 584 · cited 1×
“Many questions in social science involve a form of prediction (e.g. Kleinberg et al., 2015): A researcher may be interested in a firm’s risk factors, a politician’s electoral prospects, or a country’s growth opportunities.”
Background
Research works printed but not linked: 4
- Enrique Manjavacas, 1920, MacBERTh: Development and evaluation of a historically pre-trained language model for English (1450-1950)," in "Proceedings of the Workshop on Natural Language Processing for Digital Humanities, 23
- Alec Radford, 2018, Improving language understanding by generative pre-training
- Suproteem K Sarkar, 1920, StoriesLM: A Family of Language Models With Sequentially-Expanding Pretraining Windows
- Van Strien, 1920, headline-similarity
Cited by
Can ChatGPT forecast stock price movements? Return predictability and large language models
Lopez-Lira, A., & Tang, Y. (2026) · Journal of Financial Economics · doi:10.1016/j.jfineco.2026.104335 · Cited by 425
Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy
Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., & Tetlock, P. E. (2024) · Science Advances · doi:10.1126/sciadv.adp1528 · Cited by 191
Is Your LLM Outdated? A Deep Look at Temporal Generalization
Zhu, C., Chen, N., Gao, Y., Zhang, Y., Tiwari, P., & Wang, B. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.381 · Cited by 33
Stock portfolio selection based on risk appetite: Evidence from ChatGPT
Schneider, C. J., & Yilmaz, Y. (2025) · Finance research letters · doi:10.1016/j.frl.2025.107517 · Cited by 26
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
Metadata
- Authors
- Sarkar, S., & Vafa, K.
- Affiliations
- Harvard University (US)
- Corresponding author
- Sarkar, Suproteem
- ORCID
- Vafa, Keyon
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2024
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4394730685
- Semantic Scholar
- e5097ccd4477c454e0df52945cef4ba74d255b91
- Cited by
- 23
- References
- 35
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
- Keywords
- pre-trained language models, temporal bias, survivorship bias, pre-training data, information sets, risk factor prediction, candidate biographies, prompting, social science applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.