CoChat
Sign Up
Cited by 9Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Are Large Language Model Temporally Grounded?

Open access
Qiu, Yifu, Zhao, Zheng, Ziser, Yftah, Korhonen, Anna, Ponti, Edoardo, Cohen, Shay·2024·Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
DOI 10.18653/v1/2024.naacl-long.391
Open this paper in CoChat — free accountOpen PDF

Abstract

Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Ponti, Shay Cohen. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.naacl-long.391

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Previous studies have shown that LLMs' performance in temporal reasoning remains suboptimal.

    “previous studies have shown that their performance in temporal reasoning remains suboptimal (Chu et al., 2024;Qiu et al., 2024).”

    From TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025) · cites this paper as Qiu et al., 2024

  • These approaches face limitations in cross-session reasoning, as they focus more on storage organization than on modeling information evolution across conversations.

    “However, these approaches face limitations in cross-session reasoning, as they focus more on storage organization than on modeling information evolution across conversations (Qiu et al., 2024; Chu et al., 2024).”

    From Pre-Storage Reasoning for Episodic Memory: Shifting Inference Burden to Memory for Personalized Dialogue (2025) · cites this paper as Qiu et al., 2024

    The Introduction and Section 2 examine temporal reasoning about events in textual narratives, while Section 4.6 discusses consistency on a temporal-reasoning task. Our copy does not address cross-session reasoning or approaches focused on storage organization rather than information evolution across conversations (searched: cross-session reasoning, storage organization, memory, conversations, sessions). (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021. Can language models encode perceptual struc- ture without grounding? A case study in color. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 109-132, Online. Association for Computational Linguistics. · DOI · Paper page
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  3. Bibo Cai, Xiao Ding, Zhouhao Sun, Bing Qin, Ting Liu, Baojun wang, and Lifeng Shang. 2023. Self- supervised logic induction for explainable fuzzy temporal commonsense reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):12580-12588. · DOI
  4. Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black. 2021. Grounding 'grounding' in NLP. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 4283-4305, On- line. Association for Computational Linguistics. · DOI
  5. Ernest C Davenport Jr and Nader A El-Sanhurry. 1991. Phi/phimax: review and synthesis. Educational and psychological measurement, 51(4):821-828.
  6. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chan- dra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaïd Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on compo- sitionality. ArXiv, abs/2305.18654. · arXiv
  7. Dylan Ebert, Chen Sun, and Ellie Pavlick. 2022. Do trajectories encode verb meaning? In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2860-2871, Seattle, United States. Association for Computational Linguistics. · DOI
  8. David Ha and Jürgen Schmidhuber. 2018. Recurrent world models facilitate policy evolution. In Proceed- ings of the 32nd International Conference on Neu- ral Information Processing Systems, NIPS'18, page 2455-2467, Red Hook, NY, USA.
  9. Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S Yu, and Zhijiang Guo. 2023. Do large language models know about facts? arXiv preprint arXiv:2310.05177. · arXiv
  10. Brandon Kynoch and Hugo Latapie. 2023. Re- callm: An architecture for temporal context under- standing and question answering. arXiv preprint arXiv:2307.02738. · arXiv
  11. Shih-Ting Lin, Nathanael Chambers, and Greg Durrett. 2021. Conditional generation of temporally-ordered event sequences. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7142-7157, Online. Association for Computational Linguistics. · DOI
  12. Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. 2023b. Grounding complex natural language commands for temporal tasks in unseen environ- ments. In Conference on Robot Learning, pages 1084-1110. PMLR.
  13. Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and An- drew M. Dai. 2023c. Mind's eye: Grounded language model reasoning through simulation. In The Eleventh International Conference on Learning Representa- tions.
  14. Hector Llorens, Nathanael Chambers, Naushad UzZa- man, Nasrin Mostafazadeh, James Allen, and James Pustejovsky. 2015. SemEval-2015 task 5: QA Tem- pEval -evaluating temporal information understand- ing with question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 792-800, Denver, Colorado. Association for Computational Linguistics. · DOI
  15. Max M Louwerse and Rolf A Zwaan. 2009. Language encodes geographical information. Cognitive Sci- ence, 33(1):51-73.
  16. Congda Ma, Kotaro Funakoshi, Kiyoaki Shirai, and Manabu Okumura. 2023. Coherent story generation with structured knowledge. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 681-690, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
  17. Brian W Matthews. 1975. Comparison of the pre- dicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)- Protein Structure, 405(2):442-451.
  18. Jack Merullo, Dylan Ebert, Carsten Eickhoff, and El- lie Pavlick. 2022. Pretraining on interactions for learning grounded affordance representations. In Proceedings of the 11th Joint Conference on Lexi- cal and Computational Semantics, pages 258-277, Seattle, Washington. Association for Computational Linguistics. · DOI
  19. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  20. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470-3487, Dublin, Ireland. Association for Computational Linguistics. · DOI · Paper page
  21. Nasrin Mostafazadeh, Alyson Grealish, Nathanael Chambers, James Allen, and Lucy Vanderwende. 2016. CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures. In Pro- ceedings of the Fourth Workshop on Events, pages 51-61, San Diego, California. Association for Com- putational Linguistics. · DOI
  22. Kai Nylund, Suchin Gururangan, and Noah A Smith. 2023. Time is encoded in the weights of finetuned language models. arXiv preprint arXiv:2312.13401. · arXiv
  23. James Pustejovsky, José M Castano, Robert Ingria, Robert J Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir R Radev. 2003. TimeML: Robust spec- ification of event and temporal expressions in text. In Proceedings of the 5th International Workshop on Computational Semantics (IWCS-5), pages 337-353.
  24. Afshin Rahimi, Trevor Cohn, and Timothy Baldwin. 2017. A neural model for user geolocation and lexi- cal dialectology. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 209-216, Vancouver, Canada. Association for Computational Linguistics. · DOI
  25. Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023a. Towards benchmarking and improving the temporal reasoning capability of large language models. arXiv preprint arXiv:2306.08952. · arXiv
  26. Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023b. Towards robust temporal reasoning of large lan- guage models via a multi-hop qa dataset and pseudo- instruction tuning. arXiv preprint arXiv:2311.09821. · arXiv
  27. Zeno Vendler. 1967. Linguistics in philosophy. Cornell University Press.
  28. Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generaliza- tion via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics. · DOI · Paper page
  29. Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
  30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
  32. BigScience Workshop et al. 2023. Bloom: A 176b- parameter open-access multilingual language model.
  33. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.
  34. Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2023. Back to the future: Towards ex- plainable temporal reasoning with large language models. arXiv preprint arXiv:2310.01074. · arXiv
  35. Tian Yun, Chen Sun, and Ellie Pavlick. 2021. Does vision-and-language pretraining improve lexical grounding? In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 4357- 4366, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  36. Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "Going on a vacation" takes longer than "Go- ing for a walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China. Association for Computational Linguistics. · DOI · Paper page
  37. Bo Zhou, Yubo Chen, Kang Liu, Jun Zhao, Jiexin Xu, Xiaojian Jiang, and Qiuxia Li. 2022. Generating temporally-ordered event sequences via event op- timal transport. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 1875-1884, Gyeongju, Republic of Korea. In- ternational Committee on Computational Linguistics.

Cited by

Sorted by citations

Authors
Qiu, Y., Zhao, Z., Ziser, Y., Korhonen, A., Ponti, E., & Cohen, S.
Affiliations
  • University of Edinburgh (GB)
  • University of Cambridge (GB)
ORCID
Venue
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2024
Pages
7064-7083
DOI
10.18653/v1/2024.naacl-long.391
OpenAlex
W4401042537
Cited by
9
References
3
Citations by year
2026: 2 · 2025: 6 · 2024: 1
FWCI
2.60
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
UK Research and Innovation (EP/S022481/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, temporal grounding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review