CoChat
Sign Up
Cited by 1Topic ModelingNatural Language Processing TechniquesSyntax, Semantics, Linguistic Variation

TRAVELER: A Benchmark for Evaluating Temporal Reasoning Across Vague, Implicit and Explicit References

Open access
Kenneweg, Svenja, Deigmöller, Jörg, Cimiano, Philipp, Eggert, Julian·2026·SN Computer Science
DOI 10.1007/s42979-026-04973-y
Open this paper in CoChat — free account

Abstract

Temporal references to future, past or ongoing events are ubiquitous in natural language. Yet, there is no comprehensive benchmark dataset allowing to assess the ability of Large Language Models (LLMs) to correctly resolve these references. In this work we close this gap and provide such a benchmark that supports the evaluation of the interpretation of Explicit (e.g., “on 30.12.2024”), Implicit (e.g., “yesterday”) and Vague (e.g., “recently”) temporal references. We introduce TRAVELER, a synthetic question-answering benchmark that evaluates a system’s ability to perform temporal reasoning with respect to a set of past events. The benchmark comprises 3,300 English questions, categorized by their temporal reference and automatically generated via four templates from events sets that contain 5-100 events from a household domain. For the Vague category, ground-truth answers were established via human surveys on Prolific, following a procedure inspired by Kenneweg et al. As reported by Kenneweg et al. (in: Proceedings of the Workshop on Cognitive Aspects of the Lexicon @, Torino, 2024). LREC-COLING 2024). To demonstrate the benchmark’s applicability, we evaluate four state-of-the-art LLMs on it. All benchmarked LLMs can answer questions over events sets with a handful of events and Explicit temporal references successfully, but performance clearly deteriorates with larger event set length and when temporal references get less explicit. Notably, the Vague question category exhibits the lowest performance across all models. TRAVELER exposes limitations in current LLMs’ event temporal reasoning capabilities: Their performance clearly deteriorates with longer event sets and when including Vague temporal references. The benchmark, which is publicly available ( https://gitlab.ub.uni-bielefeld.de/s.kenneweg/TRAVELER ) also offers the possibility to test the event-temporal reasoning capabilities of other models beyond those tested in this work.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s42979-026-04973-y

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • TimeX was compared against a CoT step-by-step baseline adapted from recent temporal question-answering work.

    “We compared TimeX against a CoT step-by-step baseline, adapted from recent temporal question-answering work that evaluates temporal reasoning using the “when”-type template55.”

    From TimeX: Phenotype Onset Extraction from Clinical Narratives (2026) · cites this paper as [55]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. [1][1] Kenneweg, S., Jackson, B.B., Deigmoeller, J., Eggert, J., Cimiano, P.: An empirical study on vague deictic temporal adverbials. In: Proceedings of the Workshop on Cognitive Aspects of the Lexicon @ LREC-COLING 2024, pp. 26-31. ELRA and ICCL, Torino, Italia (2024) -
  2. [2][2] Lambalgen, M., Hamm, F.: The proper treatment of events. Bulletin of Symbolic Logic 12 (1), 139-141 (2006) -
  3. [3][3] Wang, Y., Zhao, Y.: TRAM: Benchmarking Temporal Reasoning for Large Language Models (2024). https://arxiv.org/abs/2310.00835 - · arXiv
  4. [9][9] Kenneweg, S., Deigm¨ oller, J., Cimiano, P., Eggert, J.: Benchmarking the Ability of Large Language Models to Reason About Event Sets. In: Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KEOD, pp. 74-82. SciTePress, Set´ ubal, Portugal (2024). https://doi.org/10.5220/0013046100003838 . INSTICC - · DOI
  5. [10][10] Davidson, D.: The Logical Form of Action Sentences. In: Essays on Actions and Events. Oxford University Press, ??? (2001). https://doi.org/10.1093/ 0199246270.003.0006 . https://doi.org/10.1093/0199246270.003.0006 - · DOI
  6. [11][11] Vendler, Z.: Verbs and times. The Philosophical Review 66 (2), 143-160 (1957). Accessed 2023-10-17 -
  7. [12][12] Moens, M., Steedman, M.: Temporal ontology and temporal reference. In: International Conference on Computational Logic (1988). https://api.semanticscholar.org/CorpusID:5225810 -
  8. [13][13] Sun, Q., Li, S., Huynh, D., Reynolds, M., Liu, W.: TimelineKGQA: A Comprehensive Question-Answer Pair Generator for Temporal Knowledge Graphs (2025). https://arxiv.org/abs/2501.04343 - · arXiv
  9. [14][14] Jia, Z., Abujabal, A., Saha Roy, R., Str¨ otgen, J., Weikum, G.: Tempquestions: A benchmark for temporal question answering. In: Companion Proceedings of the The Web Conference 2018. WWW '18, pp. 1057-1062. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE (2018). https://doi.org/10.1145/3184558.3191536 . https://doi.org/10.1145/3184558.3191536 - · DOI
  10. [15][15] Saxena, A., Chakrabarti, S., Talukdar, P.: Question Answering Over Temporal Knowledge Graphs (2021). https://arxiv.org/abs/2106.01515 - · arXiv
  11. [16][16] Jia, Z., Christmann, P., Weikum, G.: Faithful Temporal Question Answering over Heterogeneous Sources (2024). https://arxiv.org/abs/2402.15400 - · arXiv
  12. [17][17] Huang, R.: Domain-sensitive temporal tagging by jannik str¨ otgen, Michael gertz. Computational Linguistics 44 (2), 375-377 (2018) https://doi.org/10.1162/COLI r 00319 - · DOI
  13. [18][18] Alonso, O., Gertz, M., Baeza-Yates, R.: On the value of temporal information in information retrieval. SIGIR Forum 41 , 35-41 (2007) https://doi.org/10.1145/ 1328964.1328968 -
  14. [20][20] Kamp, H., Sassoon, G.W.: Vagueness. In: Aloni, M., Dekker, P. (eds.) The Cambridge Handbook of Formal Semantics. Cambridge University Press, ??? (2016) -
  15. [21][21] Kennedy, C.: Vagueness and grammar: The semantics of relative and absolute gradable adjectives. Linguistics and Philosophy 30 (2007) https://doi.org/10. 1007/s10988-006-9008-0 -
  16. [22][22] Lassiter, D., Goodman, N.D.: Context, scale structure, and statistics in the interpretation of positive-form adjectives. Proceedings of SALT 23 , 587-610 (2014) -
  17. [23][23] Solt, S., Gotzner, N.: Experimenting with degree, vol. 22 (2012). https://doi.org/ 10.3765/salt.v0i0.2636 - · DOI
  18. [24][24] Jia, Z., Abujabal, A., Saha Roy, R., Str¨ otgen, J., Weikum, G.: Tequila: Temporal question answering over knowledge bases. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. CIKM '18, pp. 1807-1810. ACM, ??? (2018). https://doi.org/10.1145/3269206.3269247 . http://dx.doi.org/10.1145/3269206.3269247 - · DOI
  19. [25][25] Chen, W., Wang, X., Wang, W.Y.: A Dataset for Answering Time-Sensitive Questions (2021). https://arxiv.org/abs/2108.06314 - · arXiv
  20. [26][26] Jia, Z., Pramanik, S., Roy, R.S., Weikum, G.: Complex temporal question answering on knowledge graphs. CoRR abs/2109.08935 (2021) 2109.08935 - · arXiv
  21. [28][28] Srivastava, A., al.: Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (2023) -
  22. [30][30] Zhou, B., Richardson, K., Ning, Q., Khot, T., Sabharwal, A., Roth, D.: Temporal reasoning on implicit events from distant supervision. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y. (eds.) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1361-1371. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.107 . https://aclanthology.org/2021.naacl-main.107/ - · DOI
  23. [31][31] Su, Z., Li, J., Zhang, J., Zhu, T., Qu, X., Zhou, P., Bowen, Y., Cheng, Y., zhang, M.: Living in the Moment: Can Large Language Models Grasp Co-Temporal - [Reasoning? (2024). https://arxiv.org/abs/2406.09072](https://arxiv.org/abs/2406.09072) - · arXiv
  24. [32][32] Wei, Y., Su, Y., Ma, H., Yu, X., Lei, F., Zhang, Y., Zhao, J., Liu, K.: MenatQA: A new dataset for testing the temporal comprehension and reasoning abilities of large language models. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1434-1447. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/ v1/2023.findings-emnlp.100 . https://aclanthology.org/2023.findings-emnlp.100/ -
  25. [33][33] Chen, Z., Liao, J., Zhao, X.: Multi-granularity temporal question answering over knowledge graphs. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11378-11392. Association for Computational Linguistics, Toronto, Canada (2023). https://doi.org/10.18653/v1/2023.acl-long.637 . https://aclanthology.org/2023.acl-long.637/ - · DOI
  26. [34][34] Jia, Z., Christmann, P., Weikum, G.: Tiq: A benchmark for temporal question answering with implicit time constraints, pp. 1394-1399 (2024). https://doi.org/ 10.1145/3589335.3651895 - · DOI
  27. [35][35] Neelam, S., Sharma, U., Karanam, H., Ikbal, S., Kapanipathi, P., Abdelaziz, I., Mihindukulasooriya, N., Lee, Y.-S., Srivastava, S., Pendus, C., Dana, S., Garg, D., Fokoue, A., Bhargav, G.P.S., Khandelwal, D., Ravishankar, S., Gurajada, S., Chang, M., Uceda-Sosa, R., Roukos, S., Gray, A., Lima, G., Riegel, R., Luus, F., Subramaniam, L.V.: A Benchmark for Generalizable and Interpretable Temporal Question Answering over Knowledge Bases (2022). https://arxiv.org/abs/2201. 05793 -
  28. [36][36] Meem, J.A., Rashid, M.S., Dong, Y., Hristidis, V.: PAT-Questions: A SelfUpdating Benchmark for Present-Anchored Temporal Question-Answering (2024). https://arxiv.org/abs/2402.11034 - · arXiv
  29. [38][38] Plaat, A., Wong, A., Verberne, S., Broekens, J., Stein, N., Back, T.: Reasoning with Large Language Models, a Survey (2024). https://arxiv.org/abs/2407.11511 - · arXiv
  30. [39][39] Ling, W., Yogatama, D., Dyer, C., Blunsom, P.: Program induction by rationale generation: Learning to solve and explain algebraic word problems. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158-167. Association for Computational Linguistics, Vancouver, Canada (2017). https://doi.org/10.18653/v1/P17-1015 . https://aclanthology.org/P17-1015 - · DOI · Paper page
  31. [40][40] Patel, A., Bhattamishra, S., Goyal, N.: Are NLP models really able to solve simple math word problems? In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.168 . https://aclanthology.org/2021.naacl-main.168 - · DOI · Paper page
  32. [46][46] Chu, Z., Chen, J., Chen, Q., Yu, W., Wang, H., Liu, M., Qin, B.: TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models (2024). https://arxiv.org/abs/2311.17667 - · arXiv
  33. [49][49] Tan, Q., Ng, H.T., Bing, L.: Towards benchmarking and improving the temporal reasoning capability of large language models. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14820-14835. Association for Computational Linguistics, Toronto, Canada (2023). https://doi.org/10. 18653/v1/2023.acl-long.828 . https://aclanthology.org/2023.acl-long.828/ -
  34. [50][50] Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T.T., Moazam, H., Miller, H., Zaharia, M., Potts, C.: DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023). https://arxiv.org/abs/2310.03714 · arXiv

Cited by

Sorted by citations

Authors
Kenneweg, S., Deigmöller, J., Cimiano, P., & Eggert, J.
Affiliations
Bielefeld University (DE)
Corresponding author
Kenneweg, Svenja
ORCID
Venue
SN Computer Science
Publisher
Springer Science and Business Media LLC
Published
2026-04-22
Volume
7
Issue
5
ISSN
2661-8907 (ISSN-L)
DOI
10.1007/s42979-026-04973-y
OpenAlex
W4414771173
Cited by
1
References
0
Citations by year
2026: 1
FWCI
3.46
Type
journal article
Language
en
Open access
Licence
Locations
  • SN Computer Science (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Universität Bielefeld
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
Keywords
temporal reasoning, temporal reference, benchmark dataset, synthetic question generation, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review