CoChat
Sign Up
PreprintCited by 277Time Series Analysis and ForecastingForecasting Techniques and ApplicationsStock Market Forecasting Methods

Time-LLM: Time Series Forecasting by Reprogramming Large Language Models

Open access
Jin, Ming, Wang, Shiyu, Ma, Lintao, Chu, Zhixuan, Zhang, James Y., Shi, Xiaoming, Chen, Pin-Yu, Liang, Yuxuan +3 more·2023·arXiv
DOI 10.48550/arxiv.2310.01728arXiv 2310.01728
Open this paper in CoChat — free accountOpen PDF

Abstract

Time series forecasting holds significant importance in many real-world dynamic systems and has been extensively studied. Unlike natural language process (NLP) and computer vision (CV), where a single large model can tackle multiple tasks, models for time series forecasting are often specialized, necessitating distinct designs for different tasks and applications. While pre-trained foundation models have made impressive strides in NLP and CV, their development in time series domains has been constrained by data sparsity. Recent studies have revealed that large language models (LLMs) possess robust pattern recognition and reasoning abilities over complex sequences of tokens. However, the challenge remains in effectively aligning the modalities of time series data and natural language to leverage these capabilities. In this work, we present Time-LLM, a reprogramming framework to repurpose LLMs for general time series forecasting with the backbone language models kept intact. We begin by reprogramming the input time series with text prototypes before feeding it into the frozen LLM to align the two modalities. To augment the LLM's ability to reason with time series data, we propose Prompt-as-Prefix (PaP), which enriches the input context and directs the transformation of reprogrammed input patches. The transformed time series patches from the LLM are finally projected to obtain the forecasts. Our comprehensive evaluations demonstrate that Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models. Moreover, Time-LLM excels in both few-shot and zero-shot learning scenarios.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.01728

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Foundation models learn general-purpose representations from large-scale data that can be adapted to diverse downstream tasks.

    “Foundation Models learn general-purpose representations from large-scale data that can be adapted to diverse downstream tasks [6,8,13,19,25,28,30,35,45,60,62,69].”

    From Towards Customizable Foundation Models for Human Activity Recognition with Wearable Devices (2025) · cites this paper as [19]

    The Introduction (page unavailable) says pre-trained models in computer vision and NLP can be fine-tuned for various downstream tasks, and that foundation language models perform diverse NLP tasks. It also discusses transferable representations for time series, but the passages do not establish the claim that foundation models generally learn those representations from large-scale data. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 57

  1. Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018. · arXiv
  2. George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
  4. Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 2023a.
  5. Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: neural hierarchical interpolation for time series forecast- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6989-6997, 2023b.
  6. Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469, 2023. · arXiv
  7. Pin-Yu Chen. Model reprogramming: Resource-efficient cross-domain machine learning. arXiv preprint arXiv:2202.10629, 2022. · arXiv
  8. Zhixuan Chu, Hongyan Hao, Xin Ouyang, Simeng Wang, Yan Wang, Yue Shen, Jinjie Gu, Qing Cui, Longfei Li, Siqiao Xue, et al. Leveraging large language models for pre-trained recommender systems. arXiv preprint arXiv:2308.10837, 2023. · arXiv
  9. Shohreh Deldari, Hao Xue, Aaqib Saeed, Jiayuan He, Daniel V Smith, and Flora D Salim. Beyond just vision: A review on self-supervised representation learning on multimodal and temporal data. arXiv preprint arXiv:2206.02353, 2022. · arXiv
  10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 2023.
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
  12. Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Transfer learning for time series classification. In IEEE International Conference on Big Data, pp. 1367-1376. IEEE, 2018.
  13. Nate Gruver, Marc Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 2023.
  14. Julien Herzen, Francesco Lassig, Samuele Giuliano Piazzetta, Thomas Neuer, Leo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasieka, Andrzej Skrodzki, Nicolas Huguenin, et al. Darts: User-friendly modern machine learning for time series. The Journal of Machine Learning Re- search, 23(1):5442-5447, 2022.
  15. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
  16. Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I Webb, Irwin King, and Shirui Pan. A survey on graph neural networks for time series: Forecasting, classifica- tion, imputation, and anomaly detection. arXiv preprint arXiv:2307.03759, 2023a. · arXiv
  17. Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196, 2023b. · arXiv
  18. Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
  19. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
  20. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020.
  21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
  22. Na Li, Donald M Arnold, Douglas G Down, Rebecca Barty, John Blake, Fei Chiang, Tom Courtney, Marianne Waito, Rick Trifunov, and Nancy M Heddle. From demand forecasting to inventory ordering decisions for red blood cells through integrating machine learning, statistical modeling, and inventory optimization. Transfusion, 62(1):87-99, 2022.
  23. Hengbo Liu, Ziqing Ma, Linxiao Yang, Tian Zhou, Rui Xia, Yi Wang, Qingsong Wen, and Liang Sun. Sadi: A self-adaptive decomposed interpretable framework for electric load forecasting under extreme events. In IEEE International Conference on Acoustics, Speech and Signal Pro- cessing, 2023a.
  24. Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming- Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525, 2023b. · arXiv
  25. Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881-9893, 2022.
  26. Ziyang Ma, Wen Wu, Zhisheng Zheng, Yiwei Guo, Qian Chen, Shiliang Zhang, and Xie Chen. Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition. arXiv preprint arXiv:2309.10294, 2023. · arXiv
  27. Spyros Makridakis and Michele Hibon. The m3-competition: results, conclusions and implications. International journal of forecasting, 16(4):451-476, 2000.
  28. Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Re- sults, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802-808, 2018.
  29. Igor Melnyk, Vijil Chenthamarakshan, Pin-Yu Chen, Payel Das, Amit Dhurandhar, Inkit Padhi, and Devleena Das. Reprogramming pretrained language models for antibody sequence infilling. In International Conference on Machine Learning, 2023.
  30. Suvir Mirchandani, Fei Xia, Pete Florence, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, Andy Zeng, et al. Large language models as general pattern machines. In Proceedings of the 7th Annual Conference on Robot Learning, 2023.
  31. Diganta Misra, Agam Goyal, Bharat Runwal, and Pin Yu Chen. Reprogramming under constraints: Revisiting efficient and reliable transferability of lottery tickets. arXiv preprint arXiv:2308.14969, 2023. · arXiv
  32. Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
  33. Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural ba- sis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
  34. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
  35. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  36. Stephen H Schneider and Robert E Dickinson. Climate modeling. Reviews of Geophysics, 12(3): 447-493, 1974.
  37. Yihong Tang, Ao Qu, Andy HF Chow, William HK Lam, SC Wong, and Wei Ma. Domain adver- sarial spatial-temporal network: a transferable framework for short-term traffic forecasting across cities. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 1905-1915, 2022.
  38. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
  39. Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
  40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 30, 2017.
  41. Ria Vinod, Pin-Yu Chen, and Payel Das. Reprogramming language models for molecular represen- tation learning. In Annual Conference on Neural Information Processing Systems, 2020.
  42. Yan Wang, Zhixuan Chu, Xin Ouyang, Simeng Wang, Hongyan Hao, Yue Shen, Jinjie Gu, Siqiao Xue, James Y Zhang, Qing Cui, et al. Enhancing recommender systems with large language model reasoning graphs. arXiv preprint arXiv:2308.10835, 2023. · arXiv
  43. Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. In International Joint Conference on Artificial Intelligence, 2023.
  44. Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022. · arXiv
  45. Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.
  46. Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
  47. Hao Xue and Flora D Salim. Prompt-based time series forecasting: A new task and dataset. arXiv preprint arXiv:2210.08964, 2022. · arXiv
  48. Chao-Han Huck Yang, Yun-Yun Tsai, and Pin-Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In International Conference on Machine Learning, pp. 11808-11819. PMLR, 2021.
  49. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. · arXiv
  50. Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
  51. Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, et al. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. arXiv preprint arXiv:2306.10125, 2023. · arXiv
  52. Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186, 2022a. · arXiv
  53. Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised con- trastive pre-training for time series via time-frequency consistency. Advances in Neural Informa- tion Processing Systems, 35:3988-4003, 2022b.
  54. Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106-11115, 2021.
  55. Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pp. 27268-27286. PMLR, 2022.
  56. Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained lm. Advances in Neural Information Processing Systems, 36, 2023a.
  57. Yunyi Zhou, Zhixuan Chu, Yijia Ruan, Ge Jin, Yuchen Huang, and Sheng Li. ptse: A multi- model ensemble method for probabilistic time series forecasting. In The 32nd International Joint Conference on Artificial Intelligence, 2023b. Table 11: Additional comparison with other baselines in long-term forecasting tasks. We set the forecasting horizons H ∈ {24, 36, 48, 60} for ILI and {96, 192, 336, 720} for the others. A lower value indicates better performance. Red: the best, Blue: the second best. Methods TIME-LLM N-BEATS N-HiTS AutoARIMA AutoTheta AutoETS Metric MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE ET T h1 96 0.362 0.392 0.496 0.475 0.392 0.407 0.933 0.635 1.266 0.758 1.264 0.756 192 0.398 0.418 0.544 0.504 0.442 0.438 0.868 0.621 1.188 0.749 1.181 0.745 336 0.430 0.427 0.592 0.533 0.497 0.471 0.964 0.663 1.310 0.799 1.292 0.792 720 0.442 0.457 0.639 0.588 0.559 0.533 1.043 0.705 1.510 0.882 1.405 0.842 Avg 0.408 0.423 0.568 0.525 0.473 0.462 0.952 0.656 1.319 0.797 1.286 0.784 ET T h2 96 0.268 0.328 0.384 0.431 0.321 0.368 0.390 0.417 0.461 0.430 0.444 0.403 192 0.329 0.375 0.496 0.493 0.398 0.421 0.545 0.492 0.754 0.537 0.771 0.461 336 0.368 0.409 0.585 0.542 0.453 0.459 0.697 0.562 1.355 0.683 1.526 0.522 720 0.372 0.420 0.792 0.651 0.775 0.609 0.907 0.658 3.971 1.061 5.183 0.633 Avg 0.334 0.383 0.564 0.529 0.487 0.464 0.635 0.532 1.635 0.678 1.981 0.505 ET T m1 96 0.272 0.334 0.393 0.412 0.327 0.368 1.091 0.661 1.211 0.704 1.519 0.768 192 0.310 0.358 0.425 0.427 0.376 0.400 1.119 0.682 1.237 0.724 1.535 0.784 336 0.352 0.384 0.464 0.454 0.407 0.423 1.125 0.698 1.231 0.735 1.472 0.782 720 0.383 0.411 0.521 0.488 0.471 0.456 1.243 0.745 1.394 0.801 1.591 0.825 Avg 0.329 0.372 0.451 0.445 0.395 0.412 1.145 0.697 1.268 0.741 1.529 0.790 ET T m2 96 0.161 0.253 0.204 0.302 0.188 0.273 0.435 0.375 0.245 0.316 0.359 0.333 192 0.219 0.293 0.282 0.358 0.274 0.338 0.995 0.494 0.413 0.401 0.756 0.396 336 0.271 0.329 0.378 0.425 0.384 0.406 2.324 0.648 0.790 0.528 1.747 0.467 720 0.352 0.379 0.555 0.523 0.501 0.488 9.064 1.020 2.451 0.847 6.856 0.639 Avg 0.251 0.313 0.355 0.402 0.337 0.376 3.205 0.634 0.975 0.523 2.430 0.459 W eather 96 0.147 0.201 0.185 0.244 0.160 0.222 0.255 0.273 0.279 0.266 0.331 0.277 192 0.189 0.234 0.225 0.282 0.202 0.265 0.390 0.353 0.337 0.316 0.498 0.345 336 0.262 0.279 0.274 0.323 0.253 0.303 0.775 0.457 0.472 0.385 0.898 0.423 720 0.304 0.316 0.340 0.373 0.323 0.354 2.898 0.707 0.818 0.526 2.820 0.580 Avg 0.225 0.257 0.256 0.306 0.235 0.286 1.080 0.448 0.477 0.373 1.137 0.406 Electricity 96 0.131 0.224 0.233 0.327 0.184 0.275 0.520 0.466 0.653 0.532 0.650 0.526 192 0.152 0.241 0.246 0.340 0.190 0.282 0.581 0.499 0.713 0.561 0.704 0.549 336 0.160 0.248 0.262 0.355 0.205 0.298 0.602 0.515 0.797 0.603 0.766 0.577 720 0.192 0.298 0.296 0.383 0.239 0.330 0.685 0.558 1.023 0.688 0.901 0.628 Avg 0.158 0.252 0.259 0.351 0.205 0.296 0.597 0.510 0.797 0.596 0.755 0.570 T raf f ic 96 0.362 0.248 0.608 0.447 0.410 0.329 1.068 0.694 3.207 1.219 3.254 1.221 192 0.374 0.247 0.605 0.448 0.414 0.330 1.380 0.775 3.407 1.262 3.569 1.264 336 0.385 0.271 0.618 0.454 0.428 0.337 1.448 0.790 3.473 1.274 3.971 1.275 720 0.430 0.288 0.650 0.467 0.456 0.354 1.481 0.799 3.952 1.382 6.784 1.379 Avg 0.388 0.264 0.620 0.454 0.427 0.338 1.344 0.765 3.510 1.284 4.395 1.285 ILI 24 1.285 0.727 6.809 1.870 2.675 1.080 4.909 1.329 5.991 1.510 4.869 1.315 36 1.404 0.814 6.850 1.890 3.081 1.194 5.079 1.440 5.922 1.539 4.917 1.422 48 1.523 0.807 6.788 1.876 2.973 1.176 4.276 1.339 4.637 1.329 3.966 1.301 60 1.531 0.854 6.908 1.893 3.259 1.232 3.855 1.276 4.378 1.345 3.540 1.229 Avg 1.435 0.801 6.839 1.882 2.997 1.171 4.530 1.346 5.232 1.431 4.323 1.317

Cited by

Sorted by citations

Related

Authors
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., & Wen, Q.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
123-150
DOI
10.48550/arxiv.2310.01728
arXiv
2310.01728
OpenAlex
W4387355843
Cited by
276
References
0
Citations by year
2026: 85 · 2025: 109 · 2024: 81 · 2023: 1
FWCI
107.54
Type
book chapter
Language
en
Open access
Licence
Locations
  • CRC Press eBooks (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
Keywords
time series forecasting, large language models, multimodal foundation models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review