The DOI matches its DataCite record; 72 of the 72 references it cites were found in a registry or an index; none of the 24 sources its claims cite is checked for support yet.
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Open accessAbstract
Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
Overview
CoChat Summary
TL;DR
The paper introduces Test of Time (ToT), a benchmark with synthetic semantic and arithmetic temporal reasoning tasks designed to avoid pre-training leakage and systematically vary graph structure, question type, and fact order. Evaluating Claude-3-Sonnet, GPT-4, and Gemini 1.5 Pro, it finds large effects of graph structure, question type, and fact ordering, with models strongest on single-fact and timezone questions and weakest on timeline and duration questions.
Key findings
- Graph structure strongly affects accuracy: GPT-4 ranged from 40.25% on complete graphs to 92.00% on AWE graphs (Table 3); Gemini 1.5 Pro had the best average rank (1.43), followed by GPT-4 (1.57) and Claude-3-Sonnet (3.00).
- Single-fact questions outperformed multi-fact questions: EventAtWhatTime was easiest (Claude 90.29%, GPT-4 89.43%, Gemini 93.14%), while Timeline was hardest (24.00%, 28.29%, 36.29%) (Table 5). On timeline questions, Gemini had the best precision/recall (0.81/0.83 across all graphs; 0.82/0.65 on complete graphs), while GPT-4 had the lowest (0.60/0.56 across all; 0.36/0.23 on complete) (Table 6).
- Fact order mattered: shuffled order yielded the lowest accuracy and TargetAndStartTime the highest; e.g., Gemini 63.04% shuffled vs 75.00% TargetAndStartTime, GPT-4 60.71% vs 62.60%, Claude 45.71% vs 73.57% (Table 7).
- On ToT-Arithmetic, models excelled on Timezone (Claude 74.00%, GPT-4 88.00%, Gemini 90.00%) and struggled most on Duration (15.00%, 16.00%, 13.50%) (Table 8); GPT-4 had the best average rank (1.29). When GPT-4 or Gemini erred on Duration questions, approximately 21% and 25% of their responses were within one day of the ground truth, respectively.
Methods
- ToT-Semantic: synthetic graphs generated from Erdős-Rényi, scale-free, Barabási-Albert, stochastic block model, star, complete, and Anonymized Wikidata Extract (AWE) structures; relations and time intervals are assigned, then eight question types are generated and facts sorted five ways, yielding 2,800 questions (7 graph generators × 8 question types × 5 sortings × 10 samples). An expanded dataset of approximately 50,000 examples is also reported.
- ToT-Arithmetic: seed questions were expanded by 15 annotators, filtered, categorized into seven categories (AddSubtract, Compare, Duration, Schedule, Timezone, Trick, MultiOp), functionalized, and sampled to 1,850 questions (350 AddSubtract, 350 Compare, 200 Duration, 250 Schedule, 100 Timezone, 250 Trick, 350 MultiOp).
- Evaluation: Claude-3-Sonnet, GPT-4, and Gemini 1.5 Pro were tested; GPT4-Turbo was used for ToT-Semantic because of its higher context limit, and GPT4 for ToT-Arithmetic because of its higher quality. Multiple quality-check rounds verified labels and output-format instructions.
Limitations
- Stated: the benchmark focuses on single-sentence time anchoring and only explicit temporal facts, excluding static facts; future work could extend to multi-sentence and static-fact settings.
- Stated: dataset creation is automated; manual sampling found no errors in the examined subset, but rare errors may still exist.
- Evident: dataset-size reporting is inconsistent across sections (e.g., ToT-Semantic 2,800 in Section 3.1 vs 46,480 in the datasheet; ToT-Arithmetic 1,850 in Section 3.2 vs 2,800 in the datasheet), expanded ToT-Semantic results are reported only for Gemini 1.5 Pro due to computational demands, and cross-task comparisons use GPT4-Turbo versus GPT4.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
0 Valid · 7 Not valid · 13 Not checked · 122 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, GPT-4 accuracy more than doubled between complete graphs (40.25%) and AWE graphs (92.00%).”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1 Investigating the impact of temporal structure on LLM temporal reasoning
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While the average number of nodes does not appear to consistently influence LLM performance across structures, the number of edges does show some correlation.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1.1 Influence of graph size on LLM performance
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 accuracy varied drastically across graph types, from 20.33% for complete graphs to 90.83% for AWE graphs.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1 Investigating the impact of temporal structure on LLM temporal reasoning
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated in Figure 4, increasing either the number of edges or nodes in the ToT-Semantic dataset leads to a decrease in LLM performance.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1 Investigating the impact of temporal structure on LLM temporal reasoning
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An analysis of timeline questions in Table 5 reveals that they pose the greatest challenge across all tasks.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > Performance variations within question types.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
33 more claims
Create a free account to see all 142 claims and every source verdict.
- It may state two separate claims as one. 49
- Probably not a substantial claim. 72
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 3.2.1 Task Creation
“This resulted in a dataset with 1850 questions in total.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.2 Composition
“Our ToT benchmark contains 46, 480 data points in the ToT-Semantic dataset and 2, 800 data points in the ToT-Arithmetic dataset.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1.1 Influence of graph size on LLM performance
“SFN graphs, despite having far fewer edges on average than Star graphs”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.5 Distribution
“| SFN | 17.52 53.46 | | Star | 16.16 34.12 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning > 4. Generate a question
“Which entity was the R17 of E23 at the time when E32 started being the R21 of E23?”
§ Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning > Step 2: Question Generation.
“E23 was the R21 of E32 from 2007 to 2013.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.1.1 Influence of graph size on LLM performance
“exhibit slightly better average performance in Table 3.”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.5 Distribution
“| SFN | 58.75 | 75.25 | 75.75 | | Star | 59.50 | 80.25 | 74.25 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > 4.3 Impact of temporal fact order on LLM performance
“performance is consistently lowest when facts are presented in a shuffled order”
§ EXAMPLE: Add days function def add_days(start_time, end_time): date = random_date() n = random.randint(10,100)question = f"If today is {date}, what is the day {n} days from now?" answer = current_day + datetime.timedelta(days= n) return question, answer > H.5 Distribution
“| Shuffle | 45.71 | 60.71 | 63.04 | | RelationAndStartTime | 54.29 | 65.36 | 68.57 | | StartTimeAndRelation | 47.68 | 60.54 | 64.64 |”
2 more findings
Create a free account to see every finding of this check.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Emergence of Scaling in Random Networks
Barabási, A.-L., & Albert, R. (1999) · Science · doi:10.1126/science.286.5439.509 · Cited by 36,135
Statistical mechanics of complex networks
Albert, R., & Barabási, A.-L. (2002) · Reviews of Modern Physics · doi:10.1103/revmodphys.74.47 · Cited by 20,471
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
DeepWalk: online learning of social representations
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623732 · Cited by 10,123
Self-citationChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Wikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342
Stochastic blockmodels: First steps
Holland, P. W., Laskey, K. B., & Leinhardt, S. (1983) · Social Networks · doi:10.1016/0378-8733(83)90021-7 · Cited by 2,755
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Renyi, E. (1959) · Cited by 2,334
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
HellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 2,074
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Self-citationHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
Large Language Models for Mathematical Reasoning: Progresses and Challenges
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., & Yin, W. (2024) · Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (EACL SRW) · doi:10.18653/v1/2024.eacl-srw.17 · Cited by 701
Selection-Inference: Exploiting Large Language Models for Interpretable Logical ReasoningPreprint
Creswell, A., Shanahan, M., & Higgins, I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09712 · Cited by 463
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
HyTE: Hyperplane-based Temporally aware Knowledge Graph Embedding
Dasgupta, S. S., Ray, S. N., & Talukdar, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1225 · Cited by 406
Large Language Models for Mathematical Reasoning: Progresses and ChallengesPreprint
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., & Yin, W. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.00157 · Cited by 382
(2022) · doi:10.52202/068431-0278 · Cited by 358
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Wang, Y., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y., Zhai, B., Yuan, J., You, Q., & Yang, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.06805 · Cited by 218
SLAPS: Self-Supervision Improves Structure Learning for Graph Neural NetworksPreprint
Fatemi, B., Asri, L. E., & Kazemi, S. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05034 · Cited by 216
Self-citationTowards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models
Tan, Q., Ng, H. T., & Bing, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.828 · Cited by 214
Time-aware language models as temporal knowledge bases
Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00459 · Cited by 212
PLANNER: a language for proving theorems in robots
Hewitt, C. E. (1969) · International Joint Conference on Artificial Intelligence · Cited by 210
Let Your Graph Do the Talking: Encoding Structured Data for LLMsPreprint
Perozzi, B., Fatemi, B., Zelle, D., Tsitsulin, A., Kazemi, M., Al‐Rfou, R., & Halcrow, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.05862 · Cited by 204
Self-citationJoint Reasoning for Temporal and Causal Relations
Ning, Q., Feng, Z., Wu, H., & Roth, D. (2018) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p18-1212 · Cited by 178
TempQuestions: A Benchmark for Temporal Question Answering
Jia, Z., Abujabal, A., Roy, R. S., Strötgen, J., & Weikum, G. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3184558.3191536 · Cited by 159
Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
Saparov, A., Pang, R. Y., Padmakumar, V., Joshi, N., Kazemi, M., Kim, N., & He, H. (2023) · neural information processing systems · doi:10.52202/075280-0136 · Cited by 154
Self-citationLAMBADA: Backward Chaining for Automated Reasoning in Natural Language
Kazemi, M., Kim, N., Bhatia, D. S., Xu, X., & Ramachandran, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.361 · Cited by 142
Self-citationProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language
Tafjord, O., Mishra, B. D., & Clark, P. (2020) · arXiv · doi:10.48550/arxiv.2012.13048 · Cited by 122 · cited 1×
“Previous work has developed synthetic datasets for probing and improving various kinds of reasoning including logical reasoning [53, 35, 51]and mathematical reasoning [52, 33].”
ProofWriter: Generating implications, proofs, and abductive statements over natural language
Tafjord, O., Dalvi, B., & Clark, P. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.317 · Cited by 122
GeomVerse: A Systematic Evaluation of Large Models for Geometric ReasoningPreprint
Kazemi, M., Alvari, H., Anand, A., Wu, J., Chen, X., & Soricut, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.12241 · Cited by 116
Self-citationUnderstanding Transformer Reasoning Capabilities via Graph Algorithms
Sanford, C., Fatemi, B., Hall, E., Tsitsulin, A., Kazemi, M., Halcrow, J., Perozzi, B., & Mirrokni, V. (2024) · neural information processing systems · doi:10.52202/079017-2490 · Cited by 105
Self-citationTemporal knowledge graph question answering via subgraph reasoning
Chen, Z., Zhao, X. H., Liao, J., Li, X., & Kanoulas, E. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109134 · Cited by 88
BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information
Kazemi, M., Yuan, Q., Bhatia, D. S., Kim, N., Xu, X., Imbrasaitė, V., & Ramachandran, D. (2023) · neural information processing systems · doi:10.52202/075280-1697 · Cited by 86
Self-citationUnderstanding Transformer Reasoning Capabilities via Graph Algorithms
Sanford, C., Fatemi, B., Hall, E., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.18512 · Cited by 84
Self-citationGraphWorld: Fake Graphs Bring Real Insights for GNNs
Palowitch, J., Tsitsulin, A., Mayer, B., & Perozzi, B. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539203 · Cited by 81
Self-citationFunctional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap
Srivastava, S., B, A. M., P, A., Menon, S., Sukumar, A., T, A. S., Philipose, A., Prince, S., & Thomas, S. (2024) · arXiv · doi:10.48550/arxiv.2402.19450 · Cited by 79
Do Large Language Models Know about Facts?Preprint
Hu, X., Chen, J., Li, X., Guo, Y., Wen, L., Yu, P. S., & Guo, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05177 · Cited by 76
Complex Temporal Question Answering on Knowledge Graphs
Jia, Z., Pramanik, S., Saha Roy, R., & Weikum, G. (2021) · Proceedings of the 30th ACM International Conference on Information & Knowledge Management · doi:10.1145/3459637.3482416 · Cited by 74
Temporal Reasoning in Natural Language Inference
Vashishtha, S., Poliak, A., Lal, Y. K., Durme, B. V., & White, A. S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.363 · Cited by 65
Temporal Reasoning on Implicit Events from Distant Supervision
Zhou, B., Richardson, K., Ning, Q., Khot, T., Sabharwal, A., & Roth, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.107 · Cited by 61
TempTabQA: Temporal Question Answering for Semi-Structured Tables
Gupta, V., Kandoi, P., Vora, M. B., Zhang, S., He, Y., Reinanda, R., & Srikumar, V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.149 · Cited by 53
Large Language Models Can Learn Temporal Reasoning
Xiong, S., Payani, A., Kompella, R., & Fekri, F. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.563 · Cited by 47
Large Language Models Can Learn Temporal Reasoning
Xiong, S., Payani, A., Kompella, R., & Fekri, F. (2024) · arXiv · doi:10.48550/arxiv.2401.06853 · Cited by 47
Language Models Represent Space and Time
Gurnee, W., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02207 · Cited by 46 · cited 1×
“Temporal reasoning has recently gained substantial attention [e.g., 63, 7, 32, 42, 28, 24, 38, 58].”
Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
Saparov, A., & He, H. (2022) · arXiv · doi:10.48550/arxiv.2210.01240 · Cited by 46
TempTabQA: Temporal Question Answering for Semi-Structured Tables
Gupta, V., Kandoi, P., Vora, M., Zhang, S., He, Y., Reinanda, R., & Srikumar, V. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2311.08002 · Cited by 45
Halcrow, J., Moșoi, A., Ruth, S., & Perozzi, B. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403302 · Cited by 42
Self-citationA Dataset for Answering Time-Sensitive Questions
Chen, W., Wang, X., & Wang, W. Y. (2021) · arXiv · doi:10.48550/arxiv.2108.06314 · Cited by 40 · cited 1×
“Our benchmark, Test of Time, ToT , is centered around the observation that temporal reasoning often involves two primary skills: 1) understanding the semantics and logic of time, and 2) the ability to carry out temporal arithmetic.”
Time is Encoded in the Weights of Finetuned Language Models
Nylund, K., Gururangan, S., & Smith, N. A. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.141 · Cited by 39
Synthetic Graph Generation to Benchmark Graph LearningPreprint
Tsitsulin, A., Rózemberczki, B., Palowitch, J., & Perozzi, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.01376 · Cited by 34
Self-citationPathfinder Discovery Networks for Neural Message Passing
Rózemberczki, B., Englert, P. A. J., Kapoor, A., Blais, M., & Perozzi, B. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449882 · Cited by 29
Self-citationYang, S., Li, X., Bing, L., & Lam, W. P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.728 · Cited by 26
TIQ: A Benchmark for Temporal Question Answering with Implicit Time Constraints
Jia, Z., Christmann, P., & Weikum, G. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651895 · Cited by 22
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
Liška, A., Kočiský, T., Gribovskaya, E., Terzi, T., Sezener, E., Agrawal, D., d'Autume, C. D. M., Scholtes, T., Zaheer, M., Young, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11388 · Cited by 19
UGSL: A Unified Framework for Benchmarking Graph Structure LearningPreprint
Fatemi, B., Abu-El-Haija, S., Tsitsulin, A., Kazemi, M., Zelle, D., Bulut, N., Halcrow, J., & Perozzi, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.10737 · Cited by 14
Self-citationExamining the Effects of Degree Distribution and Homophily in Graph Learning ModelsPreprint
Yasir, M. H., Palowitch, J., Tsitsulin, A., Tran-Thanh, L., & Perozzi, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08881 · Cited by 7
Self-citationTemporal Reasoning on Implicit Events from Distant SupervisionPreprint
Zhou, B., Richardson, K., Ning, Q., Khot, T., Sabharwal, A., & Roth, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12753 · Cited by 4
Premise Order Matters in Reasoning with Large Language Models
Chen, X., Chi, R. A., Wang, X., & Zhou, D. (2024) · arXiv · doi:10.48550/arxiv.2402.08939 · Cited by 3
Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language ModelsPreprint
Beniwal, H., Patel, D., Kowsik Nandagopan, D., Ladia, H., Yadav, A., & Singh, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.11997
SYGMA: System for Generalizable Modular Question Answering OverKnowledge Bases
Neelam, S., Sharma, U., Karanam, H., Ikbal, S., Kapanipathi, P., Abdelaziz, I., Mihindukulasooriya, N., Lee, Y.-S., Srivastava, S., Pendus, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.13430
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Joint Reasoning for Temporal and Causal Relations
Ning, Q., Feng, Z., Wu, H., & Roth, D. (2019) · arXiv · doi:10.48550/arxiv.1906.04941
Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models
Tan, Q., Ng, H. T., & Bing, L. (2023) · arXiv · doi:10.48550/arxiv.2306.08952
Time is Encoded in the Weights of Finetuned Language Models
Nylund, K., Gururangan, S., & Smith, N. A. (2023) · arXiv · doi:10.48550/arxiv.2312.13401
Research works printed but not linked: 6
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
- Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and Kevin Murphy. Machine learning on graphs: A model and comprehensive taxonomy. Journal of Machine Learning Research, 23(89):1-64, 2022.
- Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. In ICLR, 2024.
- Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robot Learning, pages 1084-1110. PMLR, 2023.
- John McCarthy. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes, pages 75-91, London, 1959. Her Majesty's Stationary Office. URL http://www-formal.stanford.edu/jmc/mcc59.html.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.
Cited by
Gemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.19786 · Cited by 1,156
“BBEH refers to BIG-Bench Extra Hard (Kazemi et al., 2025), a challenging LLM reasoning benchmark that aggregates several reasoning tasks (Fatemi et al., Hessel et al., 2022; Kazemi et al., 2023, 2024b; Kıcıman et al., 2023; Nie et al., 2024; Sánchez et al., 2024; Shah et al., 2024; Tyen et al.,…”
Self-citationBIG-Bench Extra HardPreprint
Kazemi, M., Fatemi, B., Bansal, H., Palowitch, J., Anastasiou, C., Mehta, S. V., Jain, L. K., Aglietti, V., Jindal, D., Chen, P., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.19187 · Cited by 109
Self-citationTIMER: temporal instruction modeling and evaluation for longitudinal clinical records
Cui, H., Unell, A., Chen, B., Fries, J., Alsentzer, E., Koyejo, S., & Shah, N. H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01965-9 · Cited by 79
Kozlowski, A. C., & Evans, J. A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251337316 · Cited by 71
Zhu, Y., Bai, X., Chen, K., Xiang, Y., Yu, J., & Zhang, M. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1482 · Cited by 43
Metadata
- Authors
- Fatemi, B., Kazemi, M., Tsitsulin, A., Malkan, K., Yim, J., Palowitch, J., Seo, S., Halcrow, J., & Perozzi, B.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.09170
- OpenAlex
- W4399695687
- Cited by
- 101
- References
- 60
- Citations by year
- 2026: 45 · 2025: 47 · 2024: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- temporal reasoning, large language models, synthetic dataset, benchmarking, temporal logic, factual consistency, evaluation framework
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.