The DOI matches its DataCite record; 61 of the 61 references it cites were found in a registry or an index; none of the 24 sources its claims cite is checked for support yet.
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Open accessAbstract
Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
Overview
CoChat Summary
TL;DR
The paper introduces Test of Time (ToT), a benchmark with synthetic semantic and arithmetic temporal reasoning tasks designed to avoid pre-training leakage and systematically vary graph structure, question type, and fact order. Evaluating Claude-3-Sonnet, GPT-4, and Gemini 1.5 Pro, it finds large effects of graph structure, question type, and fact ordering, with models strongest on single-fact and timezone questions and weakest on timeline and duration questions.
Key findings
- Graph structure strongly affects accuracy: GPT-4 ranged from 40.25% on complete graphs to 92.00% on AWE graphs (Table 3); Gemini 1.5 Pro had the best average rank (1.43), followed by GPT-4 (1.57) and Claude-3-Sonnet (3.00).
- Single-fact questions outperformed multi-fact questions: EventAtWhatTime was easiest (Claude 90.29%, GPT-4 89.43%, Gemini 93.14%), while Timeline was hardest (24.00%, 28.29%, 36.29%) (Table 5). On timeline questions, Gemini had the best precision/recall (0.81/0.83 across all graphs; 0.82/0.65 on complete graphs), while GPT-4 had the lowest (0.60/0.56 across all; 0.36/0.23 on complete) (Table 6).
- Fact order mattered: shuffled order yielded the lowest accuracy and TargetAndStartTime the highest; e.g., Gemini 63.04% shuffled vs 75.00% TargetAndStartTime, GPT-4 60.71% vs 62.60%, Claude 45.71% vs 73.57% (Table 7).
- On ToT-Arithmetic, models excelled on Timezone (Claude 74.00%, GPT-4 88.00%, Gemini 90.00%) and struggled most on Duration (15.00%, 16.00%, 13.50%) (Table 8); GPT-4 had the best average rank (1.29). When GPT-4 or Gemini erred on Duration questions, approximately 21% and 25% of their responses were within one day of the ground truth, respectively.
Methods
- ToT-Semantic: synthetic graphs generated from Erdős-Rényi, scale-free, Barabási-Albert, stochastic block model, star, complete, and Anonymized Wikidata Extract (AWE) structures; relations and time intervals are assigned, then eight question types are generated and facts sorted five ways, yielding 2,800 questions (7 graph generators × 8 question types × 5 sortings × 10 samples). An expanded dataset of approximately 50,000 examples is also reported.
- ToT-Arithmetic: seed questions were expanded by 15 annotators, filtered, categorized into seven categories (AddSubtract, Compare, Duration, Schedule, Timezone, Trick, MultiOp), functionalized, and sampled to 1,850 questions (350 AddSubtract, 350 Compare, 200 Duration, 250 Schedule, 100 Timezone, 250 Trick, 350 MultiOp).
- Evaluation: Claude-3-Sonnet, GPT-4, and Gemini 1.5 Pro were tested; GPT4-Turbo was used for ToT-Semantic because of its higher context limit, and GPT4 for ToT-Arithmetic because of its higher quality. Multiple quality-check rounds verified labels and output-format instructions.
Limitations
- Stated: the benchmark focuses on single-sentence time anchoring and only explicit temporal facts, excluding static facts; future work could extend to multi-sentence and static-fact settings.
- Stated: dataset creation is automated; manual sampling found no errors in the examined subset, but rare errors may still exist.
- Evident: dataset-size reporting is inconsistent across sections (e.g., ToT-Semantic 2,800 in Section 3.1 vs 46,480 in the datasheet; ToT-Arithmetic 1,850 in Section 3.2 vs 2,800 in the datasheet), expanded ToT-Semantic results are reported only for Gemini 1.5 Pro due to computational demands, and cross-task comparisons use GPT4-Turbo versus GPT4.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
0 Valid · 7 Not valid · 13 Not checked · 122 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, GPT-4 accuracy more than doubled between complete graphs (40.25%) and AWE graphs (92.00%).”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1 Investigating the impact of temporal structure on LLM temporal reasoning
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While the average number of nodes does not appear to consistently influence LLM performance across structures, the number of edges does show some correlation.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1.1 Influence of graph size on LLM performance
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 accuracy varied drastically across graph types, from 20.33% for complete graphs to 90.83% for AWE graphs.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1 Investigating the impact of temporal structure on LLM temporal reasoning
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated in Figure 4, increasing either the number of edges or nodes in the ToT-Semantic dataset leads to a decrease in LLM performance.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1 Investigating the impact of temporal structure on LLM temporal reasoning
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An analysis of timeline questions in Table 5 reveals that they pose the greatest challenge across all tasks.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > Performance variations within question types.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
33 more claims
Create a free account to see all 142 claims and every source verdict.
- It may state two separate claims as one. 49
- Probably not a substantial claim. 72
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 3.2.1 Task Creation
“This resulted in a dataset with 1850 questions in total.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.2 Composition
“Our ToT benchmark contains 46, 480 data points in the ToT-Semantic dataset and 2, 800 data points in the ToT-Arithmetic dataset.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1.1 Influence of graph size on LLM performance
“SFN graphs, despite having far fewer edges on average than Star graphs”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.5 Distribution
“| SFN | 17.52 53.46 | | Star | 16.16 34.12 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning > 4. Generate a question
“Which entity was the R17 of E23 at the time when E32 started being the R21 of E23?”
§ Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning > Step 2: Question Generation.
“E23 was the R21 of E32 from 2007 to 2013.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1.1 Influence of graph size on LLM performance
“exhibit slightly better average performance in Table 3.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.5 Distribution
“| SFN | 58.75 | 75.25 | 75.75 | | Star | 59.50 | 80.25 | 74.25 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.3 Impact of temporal fact order on LLM performance
“performance is consistently lowest when facts are presented in a shuffled order”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.5 Distribution
“| Shuffle | 45.71 | 60.71 | 63.04 | | RelationAndStartTime | 54.29 | 65.36 | 68.57 | | StartTimeAndRelation | 47.68 | 60.54 | 64.64 |”
2 more findings
Create a free account to see every finding of this check.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Emergence of Scaling in Random Networks
Barabási, A.-L., & Albert, R. (1999) · Science · doi:10.1126/science.286.5439.509 · Cited by 36,135
Statistical mechanics of complex networks
Albert, R., & Barabási, A.-L. (2002) · Reviews of Modern Physics · doi:10.1103/revmodphys.74.47 · Cited by 20,471
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
DeepWalk: online learning of social representations
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623732 · Cited by 10,123
Self-citationChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Wikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342
Stochastic blockmodels: First steps
Holland, P. W., Laskey, K. B., & Leinhardt, S. (1983) · Social Networks · doi:10.1016/0378-8733(83)90021-7 · Cited by 2,755
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Research works printed but not linked: 8
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
- Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and Kevin Murphy. Machine learning on graphs: A model and comprehensive taxonomy. Journal of Machine Learning Research, 23(89):1-64, 2022.
- Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. In ICLR, 2024.
- Jonathan Halcrow, Alexandru Mosoi, Sam Ruth, and Bryan Perozzi. Grale: Designing networks for graph learning. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2523-2532, 2020.
- Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
- Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robot Learning, pages 1084-1110. PMLR, 2023.
- John McCarthy. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes, pages 75-91, London, 1959. Her Majesty's Stationary Office. URL http://www-formal.stanford.edu/jmc/mcc59.html.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.
Cited by
Gemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.19786 · Cited by 1,156
Self-citationBIG-Bench Extra HardPreprint
Kazemi, M., Fatemi, B., Bansal, H., Palowitch, J., Anastasiou, C., Mehta, S. V., Jain, L. K., Aglietti, V., Jindal, D., Chen, P., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.19187 · Cited by 109
Self-citationTIMER: temporal instruction modeling and evaluation for longitudinal clinical records
Cui, H., Unell, A., Chen, B., Fries, J., Alsentzer, E., Koyejo, S., & Shah, N. H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01965-9 · Cited by 79
Kozlowski, A. C., & Evans, J. A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251337316 · Cited by 70
Zhu, Y., Bai, X., Chen, K., Xiang, Y., Yu, J., & Zhang, M. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1482 · Cited by 43
Metadata
- Authors
- Fatemi, B., Kazemi, M., Tsitsulin, A., Malkan, K., Yim, J., Palowitch, J., Seo, S., Halcrow, J., & Perozzi, B.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.09170
- OpenAlex
- W4399695687
- Cited by
- 101
- References
- 60
- Citations by year
- 2026: 45 · 2025: 47 · 2024: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- temporal reasoning, large language models, synthetic dataset, benchmarking, temporal logic, factual consistency, evaluation framework
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning. https://app.cochat.ai/papers/10.48550/arxiv.2406.09170
- The paper
- Fatemi, B., Kazemi, M., Tsitsulin, A., Malkan, K., Yim, J., Palowitch, J., Seo, S., Halcrow, J., & Perozzi, B. (2024). Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning. arXiv. https://doi.org/10.48550/arxiv.2406.09170
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.