Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its DataCite record; 61 of the 61 references it cites were found in a registry or an index; none of the 24 sources its claims cite is checked for support yet.

Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

Open access

Abstract

Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.

Overview

CoChat Summary

TL;DR

The paper introduces Test of Time (ToT), a benchmark with synthetic semantic and arithmetic temporal reasoning tasks designed to avoid pre-training leakage and systematically vary graph structure, question type, and fact order. Evaluating Claude-3-Sonnet, GPT-4, and Gemini 1.5 Pro, it finds large effects of graph structure, question type, and fact ordering, with models strongest on single-fact and timezone questions and weakest on timeline and duration questions.

Key findings

  • Graph structure strongly affects accuracy: GPT-4 ranged from 40.25% on complete graphs to 92.00% on AWE graphs (Table 3); Gemini 1.5 Pro had the best average rank (1.43), followed by GPT-4 (1.57) and Claude-3-Sonnet (3.00).
  • Single-fact questions outperformed multi-fact questions: EventAtWhatTime was easiest (Claude 90.29%, GPT-4 89.43%, Gemini 93.14%), while Timeline was hardest (24.00%, 28.29%, 36.29%) (Table 5). On timeline questions, Gemini had the best precision/recall (0.81/0.83 across all graphs; 0.82/0.65 on complete graphs), while GPT-4 had the lowest (0.60/0.56 across all; 0.36/0.23 on complete) (Table 6).
  • Fact order mattered: shuffled order yielded the lowest accuracy and TargetAndStartTime the highest; e.g., Gemini 63.04% shuffled vs 75.00% TargetAndStartTime, GPT-4 60.71% vs 62.60%, Claude 45.71% vs 73.57% (Table 7).
  • On ToT-Arithmetic, models excelled on Timezone (Claude 74.00%, GPT-4 88.00%, Gemini 90.00%) and struggled most on Duration (15.00%, 16.00%, 13.50%) (Table 8); GPT-4 had the best average rank (1.29). When GPT-4 or Gemini erred on Duration questions, approximately 21% and 25% of their responses were within one day of the ground truth, respectively.

Methods

  • ToT-Semantic: synthetic graphs generated from Erdős-Rényi, scale-free, Barabási-Albert, stochastic block model, star, complete, and Anonymized Wikidata Extract (AWE) structures; relations and time intervals are assigned, then eight question types are generated and facts sorted five ways, yielding 2,800 questions (7 graph generators × 8 question types × 5 sortings × 10 samples). An expanded dataset of approximately 50,000 examples is also reported.
  • ToT-Arithmetic: seed questions were expanded by 15 annotators, filtered, categorized into seven categories (AddSubtract, Compare, Duration, Schedule, Timezone, Trick, MultiOp), functionalized, and sampled to 1,850 questions (350 AddSubtract, 350 Compare, 200 Duration, 250 Schedule, 100 Timezone, 250 Trick, 350 MultiOp).
  • Evaluation: Claude-3-Sonnet, GPT-4, and Gemini 1.5 Pro were tested; GPT4-Turbo was used for ToT-Semantic because of its higher context limit, and GPT4 for ToT-Arithmetic because of its higher quality. Multiple quality-check rounds verified labels and output-format instructions.

Limitations

  • Stated: the benchmark focuses on single-sentence time anchoring and only explicit temporal facts, excluding static facts; future work could extend to multi-sentence and static-fact settings.
  • Stated: dataset creation is automated; manual sampling found no errors in the examined subset, but rare errors may still exist.
  • Evident: dataset-size reporting is inconsistent across sections (e.g., ToT-Semantic 2,800 in Section 3.1 vs 46,480 in the datasheet; ToT-Arithmetic 1,850 in Section 3.2 vs 2,800 in the datasheet), expanded ToT-Semantic results are reported only for Gemini 1.5 Pro due to computational demands, and cross-task comparisons use GPT4-Turbo versus GPT4.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.09170

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Comparison of the same temporal query using real (left) and anonymized (right) entity names. Gemini Advanced correctly answered the query with real names but failed with anonymized names, suggesting that LLMs might rely on their parametric knowledge to solve temporal tasks.
Figure 1 Comparison of the same temporal query using real (left) and anonymized (right) entity names. Gemini Advanced correctly answered the query with real names but failed with anonymized names, suggesting that LLMs might rely on their parametric knowledge to solve temporal tasks.CC-BY · From the PDF
Figure 2 Steps for creating the ToT-Semantic dataset.
Figure 2 Steps for creating the ToT-Semantic dataset.CC-BY · From the PDF
Figure 3 Steps for creating the ToT-Arithmetic dataset. The green and blue colors represent the operations done by the authors and the annotators respectively.
Figure 3 Steps for creating the ToT-Arithmetic dataset. The green and blue colors represent the operations done by the authors and the annotators respectively.CC-BY · From the PDF
Figure 4 Accuracy of models for different number of edges and nodes.
Figure 4 Accuracy of models for different number of edges and nodes.CC-BY · From the PDF
Figure 5 A visualization of a representative graph from each graph generator: Erd˝ os-Rényi (ER), Scale-Free Networks (SFN), Barabási-Albert (BA), Stochastic Block Model (SBM), star-graph, and complete-graph.
Figure 5 A visualization of a representative graph from each graph generator: Erd˝ os-Rényi (ER), Scale-Free Networks (SFN), Barabási-Albert (BA), Stochastic Block Model (SBM), star-graph, and complete-graph.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 7 Not valid · 13 Not checked · 122 Own results and interpretations

33 more claims

Create a free account to see all 142 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 49
  • Probably not a substantial claim. 72
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 7 · the rest aren’t loaded

2 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 60 references

Research works printed but not linked: 8

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
  2. Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and Kevin Murphy. Machine learning on graphs: A model and comprehensive taxonomy. Journal of Machine Learning Research, 23(89):1-64, 2022.
  3. Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. In ICLR, 2024.
  4. Jonathan Halcrow, Alexandru Mosoi, Sam Ruth, and Bryan Perozzi. Grale: Designing networks for graph learning. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2523-2532, 2020.
  5. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
  6. Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robot Learning, pages 1084-1110. PMLR, 2023.
  7. John McCarthy. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes, pages 75-91, London, 1959. Her Majesty's Stationary Office. URL http://www-formal.stanford.edu/jmc/mcc59.html.
  8. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.

Cited by

See all 101

Metadata

Authors
Fatemi, B., Kazemi, M., Tsitsulin, A., Malkan, K., Yim, J., Palowitch, J., Seo, S., Halcrow, J., & Perozzi, B.
Affiliations
Google (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4399695687
Cited by
101
References
60
Citations by year
2026: 45 · 2025: 47 · 2024: 8
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
temporal reasoning, large language models, synthetic dataset, benchmarking, temporal logic, factual consistency, evaluation framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning. https://app.cochat.ai/papers/10.48550/arxiv.2406.09170
The paper
Fatemi, B., Kazemi, M., Tsitsulin, A., Malkan, K., Yim, J., Palowitch, J., Seo, S., Halcrow, J., & Perozzi, B. (2024). Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning. arXiv. https://doi.org/10.48550/arxiv.2406.09170

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.