TRAM: Benchmarking Temporal Reasoning for Large Language Models
Open accessAbstract
Reasoning about time is essential for understanding the nuances of events described in natural language.Previous research on this topic has been limited in scope, characterized by a lack of standardized benchmarks that would allow for consistent evaluations across different studies.In this paper, we introduce TRAM, a temporal reasoning benchmark composed of ten datasets, encompassing various temporal aspects of events such as order, arithmetic, frequency, and duration, designed to facilitate a comprehensive evaluation of the TeR capabilities of large language models (LLMs).We evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios, and establish baselines with BERT-based and domainspecific models.Our findings indicate that the best-performing model lags significantly behind human performance.It is our aspiration that TRAM will spur further progress in enhancing the TeR capabilities of LLMs.Our data and code are available at https:// github.com/EternityYW/TRAM-Benchmark.Q: It is also a love story , between Ace and Tobio, a trans woman.How often do they break up? A. Once B. Always C. Once per weekFrequency (Commonsense) Q: A historic event is documented to have happened 'before you know it'.When did it take place? A. The next day B. Without hesitation C. Before long Ambiguity Resolution (Interpretation) Q: She noticed that all the wall clocks in the store were set to ten past ten.What's the more plausible CAUSE? A. It is a common display setting for clocks and watches.B. B. It was ten minutes past ten at that moment.Temporal Causality (Cause) Q: I woke up so late this morning.I was panicked when I saw what time it was.I had to be at work on time.I threw myself together quickly.Which of the two endings is the most plausible correct ending to the story? A. I was able to get a job at a local restaurant.B.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most existing temporal reasoning benchmarks cannot be used directly for studying temporal reasoning in multi-session dialogues because they are usually built on shorter texts that contain clear temporal information.
“However, most existing temporal reasoning benchmarks cannot be used directly for this study, because they are usually built on shorter texts, such as stories and Wikipedia articles, that contain clear temporal information (Chen et al., 2021;Wang and Zhao, 2024;Xiong et al., 2024).”
From TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025) · cites this paper as Wang and Zhao, 2024
Section 3.3 describes a TRAM storytelling task built from four-sentence stories selected for evident temporal aspects; Section 3.1 also describes questions drawn from Wikipedia. These examples support the description of short texts with clear temporal information, but the paper does not establish that most existing benchmarks have those features or that they cannot be used directly for multi-session dialogue research. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredIncorporating CoT generally improves performance.
“Furthermore, we find that incorporating CoT generally improves performance, aligning with previous findings (Wang and Zhao, 2024;Xiong et al., 2024).”
From TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025) · cites this paper as Wang and Zhao, 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 6
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. In International Conference on Learning Representations.
- Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2.
- Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys.
- Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In 2011 AAAI Spring Symposium Series.
- Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computa- tional Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 1-9.
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Evaluating Very Long-Term Conversational Memory of LLM Agents
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.747 · Cited by 96 · see citing works
Zero shot health trajectory prediction using transformer
Renc, P., Jia, Y., Samir, A. E., Wąs, J., Li, Q., Bates, D. W., & Sitek, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01235-0 · Cited by 67 · see citing works
Chain-of-History Reasoning for Temporal Knowledge Graph Forecasting
Xia, Y., Wang, D., Liu, Q., Wang, L., Wu, S., & Zhang, X.-Y. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.955 · Cited by 41 · see citing works
Natural Language Processing with Commonsense Knowledge: A SurveyPreprint
Xie, Y., Liu, Z., Ma, Z., Meng, F., Xiao, Y., Miao, F., & Pu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04674 · Cited by 23 · see citing works
TIMER: temporal instruction modeling and evaluation for longitudinal clinical records
Cui, H., Unell, A., Chen, B., Fries, J., Alsentzer, E., Koyejo, S., & Shah, N. H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01965-9 · Cited by 22 · see citing works
CausalEval: Towards Better Causal Reasoning in Language Models
Yu, L., Chen, D., Xiong, S., Wu, Q., Li, D., Chen, Z., Liu, X., & Pan, L. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.622 · Cited by 12 · see citing works
Are Large Language Model Temporally Grounded?
Qiu, Y., Zhao, Z., Ziser, Y., Korhonen, A., Ponti, E., & Cohen, S. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.391 · Cited by 9 · see citing works
Ojemann, W., Xie, K., Liu, K., Chang, E., Roth, D., Litt, B., & Ellis, C. A. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00198-5 · Cited by 9 · see citing works
Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?
Su, Z., Li, J., Zhang, J., Zhu, T., Qu, X., Zhou, P., Bowen, Y., Cheng, Y., & Zhang, M. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.703 · Cited by 8 · see citing works
TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation
Zhang, Y., Yuan, S., Hu, C., Richardson, K., Xiao, Y., & Chen, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.215 · Cited by 7 · see citing works
NarraBench: A Comprehensive Framework for Narrative Benchmarking
Hamilton, S., Wilkens, M. J., & Piper, A. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.176 · Cited by 5 · see citing works
KGAgent4CTI: unlocking the power of LLM in threat intelligence
Yang, L., Zhu, H., Mu, J., & Li, Q. (2026) · Cybersecurity · doi:10.1186/s42400-025-00458-2 · Cited by 5 · see citing works
TellTime: An AI-Augmented Calendar with a Voice Interface for Collecting Time-Use Data
Hoefer, M. J. D., Rychecky, R., Gong, M., & Voida, S. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712116 · Cited by 5 · see citing works
Weng, M., Cao, D., Yang, W., Sharma, Y., & Liu, Y. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817485 · Cited by 4 · see citing works
MM-Forecast: A Multimodal Approach to Temporal Event Forecasting with Large Language Models
Li, H., Yang, Z., Ma, Y., Bin, Y., Yang, Y., & Chua, T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681593 · Cited by 4 · see citing works
MRAG: a modular retrieval framework for time-sensitive question answering
Zhang, S., Xue, Y., Zhang, Y., Wu, X., Luu, A. T., & Zhao, C. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.167 · Cited by 3
Position Paper: Large Language Models need Episodic Memory
Hao, G., Zhang, Y., Ma, G., Chen, Y., Alexandre, F., & Yu, S. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229266 · Cited by 2 · see citing works
Song, X., Liang, B., Sun, Y., Zhang, C., Wang, B., & Xu, R. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730173 · Cited by 1 · see citing works
Ma, Z., Li, J., Wen, Y., & Yu, H. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116512 · Cited by 1 · see citing works
LLM-Symbolic Integration for Robust Temporal Tabular Reasoning
Kulkarni, A., Dixit, K., Srikumar, V., Roth, D., & Gupta, V. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1022 · Cited by 1 · see citing works
Discrete Minds in a Continuous World: Do Language Models Know Time Passes?
Wang, M., Bai, Y., Vu, T.-T., Shareghi, E., & Haffari, G. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1016 · Cited by 1 · see citing works
Ge, Y., Romeo, S., Cai, J., Shu, R., Sunkara, M., Benajiba, Y., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.01630 · Cited by 3 · see citing works
Enhancing Temporal Reasoning in Large Language Models Through a Boosting ApproachPreprint
Zhang, M., Wan, W., & Fang, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5378499
Robocskbench: Benchmarking Embodied Commonsense Capabilities of Large Language Models
Töberg, J.-P., Kenneweg, S., & Cimiano, P. (2025) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/ur65550.2025.11078036
Sohn, T. S., Dillitzer, M., Bach, J., Corso, J. J., Brühl, T., Schwager, R., Eberhardt, T. D., & Sax, E. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00371
Rethinking Learning: The Role of Unlearning in Generative AI-Based Conceptual Modeling
Sakib, S. K., Liddle, S. W., Lynch, C. J., Agrawal, A., & Giabbanelli, P. J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-08623-5_2
EventPuzzle: A Benchmark for Multi-Perspective Event Prediction Based on Event Arguments
Ding, G., Zhou, J., Li, Y., Guo, X., Wang, X., & Zha, D. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761325
Do Large Language Models (LLMs) Understand Chronology? (Student Abstract)
Wongchamcharoen, P. K., & Glasserman, P. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i48.42295
TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Models
Holtermann, C., Krebs, N., & Lauscher, A. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.187
Rani, N. M., & Jayaraj, M. M. (2026) · Management and Labour Studies · doi:10.1177/0258042x261432794
Temporal Knowledge Graph-Driven Multi-Agent Collaborative Retrieval Augmented Generation Framework
Niu, Y., Ai, Y., Wan, X., Gao, S., Zhao, Y., Kuang, J., & Liu, J. (2025) · China Automation Congress (CAC) · doi:10.1109/cac67268.2025.11487625
Self-citationCheng, Y., Moakhar, A. S., Fan, C., Hosseini, P., Faghih, K., Sodagar, Z., Wang, W., & Feizi, S. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1848
On the Cultural Anachronism and Temporal Reasoning in Vision Language Models
Ranjan, M., Jha, P., Kumari, K., & Shen, Z. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1514
Yin, B., & Cheng, X. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3560-5_12
A Comprehensive Taxonomy of Temporal Dimensions in Natural Language Queries
Lauriola, I. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820426
Sun, F., Tong, Z., Wei, H., Racharak, T., & Nguyen, L.-M. (2026) · IEEE Access · doi:10.1109/access.2026.3731685
Kanithi, P., Pradhan, A., Cui, H., Johnstone, K., Luk, A., Hammond, R., Brown, C. S., Walker, A. S., & Eyre, D. W. (2026) · medRxiv · doi:10.64898/2026.09.29.26363894
Jeong, E., Jeong, W., Oh, K. H., & Jun, J. K. (2026) · Journal of Medical Internet Research · doi:10.2196/91101
- Authors
- Wang, Y., & Zhao, Y.
- Affiliations
- Stanford University (US)
- Meta (United States) (US)
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 6389-6415
- OpenAlex
- W4402683894
- Cited by
- 24
- References
- 0
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 2 · 2023: 1 · 2021: 1
- FWCI
- 6.50
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- temporal reasoning, large language models, benchmark, order of events, frequency, duration, ambiguity resolution, temporal causality, commonsense reasoning, GPT-4, Llama 2, zero-shot evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
