Lookahead Bias in Pretrained Language Models
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Empirical analysis that uses outputs from pretrained language models can be subject to a form of temporal lookahead bias . This bias arises when a language model’s pre-training data contains information about the future, which then leaks into analysis that should only use information from the past. In this paper we develop direct tests for lookahead bias, based on the assumption that some events are unpredictable given a prespecified information set. Using these tests, we find evidence of lookahead bias in two applications of language models to social science: Predicting risk factors from corporate earnings calls and predicting election winners from candidate biographies. We additionally discuss the limitations of prompting-based approaches to counteract this bias. The issues we raise can be addressed by using models whose pretraining data is free of survivorship bias and contains only language produced prior to the analysis period of interest
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This sample period is selected to avoid look-ahead bias and ensure an out-of-sample evaluation of LLMs like ChatGPT, as the version we evaluate was trained on data up until September 2021.”
From Can ChatGPT forecast stock price movements? Return predictability and large language models (2026) · cites this paper as Sarkar and Vafa (2024)
In the citing paper
“This sample period ensures that our evaluation is out-of-sample as the training data of the ChatGPT models stops in September 2021 (e.g., Sarkar and Vafa (2024) and Lopez-Lira, Tang, and Zhu (2025)).”
From Can ChatGPT forecast stock price movements? Return predictability and large language models (2026) · cites this paper as Sarkar and Vafa (2024)
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Gentzkow, M., Kelly, B., & Taddy, M. (2019) · Journal of Economic Literature · doi:10.1257/jel.20181020 · Cited by 1,236
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629
AI recognition of patient race in medical imaging: a modelling study
Gichoya, J. W., Banerjee, I., Bhimireddy, A. R., Burns, J. L., Celi, L. A., Chen, L.-C., Correa, R., Dullerud, N., Ghassemi, M., Huang, S.-C., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00063-2 · Cited by 611 · cited 1×
“This evidence is consistent with evidence in other domains that shows that machine learning models can infer information from “degraded” data, even if humans cannot (e.g. Gichoya et al., 2022).”
ResultSurvivor Bias and Mutual Fund Performance
Elton, E. J., Gruber, M. J., & Blake, C. R. (1996) · Review of Financial Studies · doi:10.1093/rfs/9.4.1097 · Cited by 595
Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015) · American Economic Review · doi:10.1257/aer.p20151023 · Cited by 584 · cited 1×
“Many questions in social science involve a form of prediction (e.g. Kleinberg et al., 2015): A researcher may be interested in a firm’s risk factors, a politician’s electoral prospects, or a country’s growth opportunities.”
BackgroundCan ChatGPT forecast stock price movements? Return predictability and large language models
Lopez-Lira, A., & Tang, Y. (2026) · Journal of Financial Economics · doi:10.1016/j.jfineco.2026.104335 · Cited by 425
Eggers, A. C., Folke, O., Fowler, A., Hainmueller, J., Hall, A. B., & Snyder, J. M. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2265625 · Cited by 424
Eggers, A. C., Fowler, A., Hainmueller, J., Hall, A. B., & Snyder, J. M. (2015) · American Journal of Political Science · doi:10.1111/ajps.12127 · Cited by 423
BloombergGPT: A Large Language Model for Finance
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., & Mann, G. (2023) · arXiv · Cited by 357
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · cited 1×
“One part of this literature studies a form of leakage known as data contamination , which occurs when a model is (pre)trained on data that is also in the test set used to evaluate the model (Dodge et al., 2021; Golchin and Surdeanu, 2023; Sainz et al., 2023).”
BackgroundApproaching Human-Level Forecasting with Language ModelsPreprint
Halawi, D., Zhang, F., Yueh-Han, C., & Steinhardt, J. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2402.18563 · Cited by 163
Leakage and the Reproducibility Crisis in ML-based Science
Kapoor, S., & Narayanan, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.07048 · Cited by 133
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.722 · Cited by 129
Remote Work Across Jobs, Companies and Space
Hansen, S. E., Lambert, P. J., Bloom, N., Davis, S. J., Muvdi, Y., Sadun, R., & Taska, B. (2023) · Federal Reserve Bank of Dallas, Working Papers · doi:10.24149/wp2629 · Cited by 127
Remote Work across Jobs, Companies, and Space
Hansen, S., Lambert, P. J., Bloom, N., Davis, S., Sadun, R., & Taska, B. (2023) · doi:10.3386/w31007 · Cited by 113
Training Data Leakage Analysis in Language ModelsPreprint
Inan, H. A., Ramadan, O., Wutschitz, L., Jones, D., Rühle, V., Withers, J., & Sim, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05405 · Cited by 81
American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers
Dell, M., Carlson, J., Bryan, T., Silcock, E., Arora, A., Shen, Z., D’Amico-Wong, L., Le, Q., Querubín, P., & Heldring, L. (2023) · neural information processing systems · doi:10.52202/075280-3540 · Cited by 51
Does Finance Benefit Society? A Language Embedding Approach
Jha, M., Liu, H., & Manela, A. (2025) · Review of Financial Studies · doi:10.1093/rfs/hhaf012 · Cited by 51
What Purpose Do Corporations Purport? Evidence from Letters to Shareholders
Rajan, R. G., Ramella, P., & Zingales, L. (2023) · National Bureau of Economic Research · doi:10.3386/w31054 · Cited by 48
Time is Encoded in the Weights of Finetuned Language Models
Nylund, K., Gururangan, S., & Smith, N. A. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.141 · Cited by 39
Surveying Generative AI's Economic Expectations
Bybee, L. (2023) · arXiv · doi:10.48550/arxiv.2305.02823 · Cited by 35
Surveying Generative AI's Economic Expectations
Bybee, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4430515 · Cited by 35
hmBERT: Historical Multilingual Language Models for Named Entity RecognitionPreprint
Schweter, S., März, L., Schmid, K., & Çano, E. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.15575 · Cited by 32
Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2023) · arXiv · doi:10.48550/arxiv.2308.08493 · Cited by 29
Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis
Glasserman, P., & Lin, C. (2024) · The Journal of Financial Data Science · doi:10.3905/jfds.2023.1.143 · Cited by 25
Does Finance Benefit Society? A Language Embedding ApproachPreprint
Jha, M., Liu, H., & Manela, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3655263 · Cited by 24
Approaching Human-Level Forecasting with Language Models
Halawi, D., Zhang, F., Yueh-Han, C., & Steinhardt, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1598 · Cited by 24
The Promise and Peril of Generative AI: Evidence from GPT as Sell-Side AnalystsPreprint
Li, E. X., Shen, M., Tu, Z., & Zhou, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4480947 · Cited by 22
What Purpose Do Corporations Purport? Evidence from Letters to ShareholdersPreprint
Rajan, R. G., Ramella, P., & Zingales, L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4035849 · Cited by 21
Remote Work Across Jobs, Companies, and SpacePreprint
Hansen, S. E., Lambert, P., Bloom, N., Davis, S. J., Sadun, R., & Taska, B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4377782 · Cited by 21
In-Context Unlearning: Language Models as Few Shot Unlearners
Pawelczyk, M., Neel, S., & Lakkaraju, H. (2023) · arXiv · doi:10.48550/arxiv.2310.07579 · Cited by 20 · cited 1×
“Recent work has argued that prompt design may lead language models to “unlearn” information (Pawelczyk et al., 2023) from training data.”
BackgroundA Semantic Approach to Financial Fundamentals
Chen, J., & Sarkar, S. K. (2020) · Cited by 14
Self-citationAssessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment AnalysisPreprint
Glasserman, P., & Lin, C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4586726 · Cited by 13
Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis
Glasserman, P., & Lin, C. (2023) · arXiv · Cited by 12
Proving Test Set Contamination in Black Box Language Models
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2310.17623 · Cited by 11 · cited 1×
“For example, Oren et al. (2023) present a procedure for identifying whether a language model memorizes language from the test set.”
BackgroundAmerican Stories: A Large-Scale Structured Text Dataset of Historical U.S. NewspapersPreprint
Dell, M., Carlson, J., Bryan, T., et al. (2023) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2308.12477 · Cited by 7
What Purpose Do Corporations Purport? Evidence from Letters to ShareholdersPreprint
Rajan, R. G., Ramella, P., & Zingales, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4405063 · Cited by 3
Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models
Lopez-Lira, A., & Tang, Y. (2023) · doi:10.2139/ssrn.4412788
Research works printed but not linked: 4
- Enrique Manjavacas, 1920, MacBERTh: Development and evaluation of a historically pre-trained language model for English (1450-1950)," in "Proceedings of the Workshop on Natural Language Processing for Digital Humanities, 23
- Alec Radford, 2018, Improving language understanding by generative pre-training
- Suproteem K Sarkar, 1920, StoriesLM: A Family of Language Models With Sequentially-Expanding Pretraining Windows
- Van Strien, 1920, headline-similarity
Cited by
Can ChatGPT forecast stock price movements? Return predictability and large language models
Lopez-Lira, A., & Tang, Y. (2026) · Journal of Financial Economics · doi:10.1016/j.jfineco.2026.104335 · Cited by 425
Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy
Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., & Tetlock, P. E. (2024) · Science Advances · doi:10.1126/sciadv.adp1528 · Cited by 191
Is Your LLM Outdated? A Deep Look at Temporal Generalization
Zhu, C., Chen, N., Gao, Y., Zhang, Y., Tiwari, P., & Wang, B. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.381 · Cited by 33
Stock portfolio selection based on risk appetite: Evidence from ChatGPT
Schneider, C. J., & Yilmaz, Y. (2025) · Finance research letters · doi:10.1016/j.frl.2025.107517 · Cited by 26
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
Metadata
- Authors
- Sarkar, S., & Vafa, K.
- Affiliations
- Harvard University (US)
- Corresponding author
- Sarkar, Suproteem
- ORCID
- Vafa, Keyon
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2024
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4394730685
- Semantic Scholar
- e5097ccd4477c454e0df52945cef4ba74d255b91
- Cited by
- 23
- References
- 35
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
- Keywords
- pre-trained language models, temporal bias, survivorship bias, pre-training data, information sets, risk factor prediction, candidate biographies, prompting, social science applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.