Surveying Generative AI's Economic Expectations
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
I introduce a survey of economic expectations formed by querying a large language model (LLM)'s expectations of various financial and macroeconomic variables based on a sample of news articles from the Wall Street Journal between 1984 and 2021. I find the resulting expectations closely match existing surveys including the Survey of Professional Forecasters (SPF), the American Association of Individual Investors, and the Duke CFO Survey. Importantly, I document that LLM based expectations match many of the deviations from full-information rational expectations exhibited in these existing survey series. The LLM's macroeconomic expectations exhibit under-reaction commonly found in consensus SPF forecasts. Additionally, its return expectations are extrapolative, disconnected from objective measures of expected returns, and negatively correlated with future realized returns. Finally, using a sample of articles outside of the LLM's training period I find that the correlation with existing survey measures persists -- indicating these results do not reflect memorization but generalization on the part of the LLM. My results provide evidence for the potential of LLMs to help us better understand human beliefs and navigate possible models of nonrational expectations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“4That LLMs reflect the biases and beliefs of their training corpus is central to much of the current work on AI alignment (Bender et al. (2021), Schramowski et al. (2022)) First, I validate how well GPT’s expectations of the stock market match the expectations of the American Association of…”
Econometric policy evaluation: A critique
Lucas, R. E. (1976) · Carnegie-Rochester Conference Series on Public Policy · doi:10.1016/s0167-2231(76)80003-6 · Cited by 6,110
A comprehensive look at the empirical performance of equity premium prediction
Welch, I., & Goyal, A. (2008) · Review of Financial Studies · doi:10.1093/rfs/hhm014 · Cited by 3,987
Chomsky (1959) · doi:10.2307/411334 · Cited by 3,425
Large pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291 · cited 1×
“4That LLMs reflect the biases and beliefs of their training corpus is central to much of the current work on AI alignment (Bender et al. (2021), Schramowski et al. (2022)) First, I validate how well GPT’s expectations of the stock market match the expectations of the American Association of…”
Michalopoulos, S., & Xue, M. M. (2021) · The Quarterly Journal of Economics · doi:10.1093/qje/qjab003 · Cited by 165
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Aher, G., Arriaga, R. I., & Kalai, A. T. (2022) · arXiv · doi:10.48550/arxiv.2208.10264 · Cited by 161
Cited by
Large language models empowered agent-based modeling and simulation: a survey and perspectives
Gao, C., Lan, X., Li, N., Yuan, Y., Ding, J., Zhou, Z., Xu, F., & Li, Y. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03611-3 · Cited by 294
Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis
Fan, C., Chen, J., Jin, Y., & He, H. (2023) · arXiv · doi:10.48550/arxiv.2312.05488 · Cited by 44
ChatGPT Hallucinates Non-existent Citations: Evidence from Economics
Buchanan, J., Hill, S., & Shapoval, O. (2024) · The American Economist · doi:10.1177/05694345231218454 · Cited by 40
Lookahead Bias in Pretrained Language Models
Sarkar, S., & Vafa, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4754678 · Cited by 23
ChatGPT Hallucinates Nonexistent Citations: Evidence from Economics
Buchanan, J., Hill, S., & Shapoval, O. (2023) · doi:10.2139/ssrn.4632842 · Cited by 8
Metadata
- Authors
- Bybee, L.
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2023
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4376455614
- Cited by
- 35
- References
- 7
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Surveying Generative AI's Economic Expectations. https://app.cochat.ai/papers/10.2139/ssrn.4430515
- The paper
- Bybee, L. (2023). Surveying Generative AI's Economic Expectations. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4430515
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.