RealTime QA: What's the Answer Right Now?
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications of question answering and beyond.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, the static nature of parametric memory can lead to information lag, posing challenges in maintaining the relevance of the stored data as it may not reflect the most current information [27, 166].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [166]
In the citing paper
“Abstention is an active area of LLM-related research (Kasai et al. 2023;Röttger et al. 2024;Ahdritz et al. 2024;Kim and Thorne 2024;Madhusudhan et al. 2024) and is central to Islamic tradition.”
From Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions (2025) · cites this paper as Kasai et al. 2023
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 101–137 of 137
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Haas, L., Yona, G., D'Antonio, G., Goldshtein, S., & Das, D. (2025) · arXiv · doi:10.48550/arxiv.2509.07968 · Cited by 1
FastLearn: A Rapid Learning Agent for Chat Models to Acquire Latest Knowledge
Fu, C., Wang, G., Lu, R., & Tang, S. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00035 · Cited by 2
Detecting and Correcting Hallucinations in LLMs via Substantive Uncertainty and Iterative Validation
Chen, Z., Cheng, Y., & Gao, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3352-7_14 · Cited by 1
Dissecting GraphRAG: A Modular Analysis of Knowledge Structuring for Factoid Question Answering
Nishida, N., Munne, R. F., Liu, S., Tokunaga, N., Yamagata, Y., Cheng, F., Kozaki, K., & Matsumoto, Y. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.615 · Cited by 2
Dammu, P. P. S., Palkhiwala, A., Roosta, T. G., & Shah, C. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808627 · Cited by 1
UKP-SQuARE v3: A Platform for Multi-Agent QA Research
Puerto, H., Baumgärtner, T., Sachdeva, R. S., Fang, H., Zhang, H., Tariverdian, S., Wang, K., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.55 · Cited by 1
Takahashi, R., Kamoda, G., Heinzerling, B., Sakaguchi, K., & Inui, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.06032 · Cited by 1
Self-citationRobust, efficient, and knowledge-augmented text generation with pre-trained language models
Li, J. (2025) · doi:10.71781/77 · Cited by 1
Large Language Model based Knowledge Creation Verified by Knowledge Graphs and News Articles
Wilberg, F. (2025) · reposiTUm (TU Wien) · doi:10.34726/hss.2025.126769 · Cited by 1
Zhou, Y., Zhang, F., Wang, S., Xu, Y., & Zhang, H. (2026) · arXiv (Cornell University) · Cited by 3
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
Gao, Y., Zhao, R., Deng, Y., & Zhang, W. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1249
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y.-M., & Vu, T. (2025) · arXiv · doi:10.48550/arxiv.2506.01062
DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
Gupta, N., Chatterjee, R., Haas, L., Tao, C., Wang, A., Liu, C., Oiwa, H., Gribovskaya, E., Ackermann, J., Blitzer, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.20975
MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
Chen, S., Moreira, P., Xiao, Y., Schmidgall, S., Warner, J., Aerts, H., Hartvigsen, T., Gallifant, J., & Bitterman, D. S. (2025) · arXiv · doi:10.48550/arxiv.2505.14963
Yoon, S.-Y., Kim, J., & Hwang, S.-W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.06716 · Cited by 2
Spotlight on Recommender Systems
Gebremeskel, G. G. (2025) · Radboud University Press eBooks · doi:10.54195/9789465151137
Stora språkmodeller som resurser för lärare: möjligheter och utmaningar inom högre utbildning
Håkansson, H., & Tennander, M. (2024) · Lund University Publications Student Papers (Lund University)
Towards Generalized Open Domain Question Answering Systems
Liu, L. (2024) · UCL Discovery (University College London)
Robertson, A., Liang, H., Gani, M., Kumar, R., & Rajamohan, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.19643 · Cited by 3
Tian, J., Zeng, W., Wu, J., Liu, L., & Zhao, X. H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6426-2_48
GRAFF: GRaph-Augmented Fine-grained Fusion for Large Language Models
Chaudhary, H., Wang, R., Ramesh, G., & Hu, J. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.293 · Cited by 3
Robertson, A., Liang, H., Gani, M., Kumar, R., & Rajamohan, S. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.206 · Cited by 2
Wang, Z., & Chen, W. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-06264-2_2
Contributions to evaluating and improving the faithfulness for text generation
Zhang, K. (2025) · theses.fr (ABES) · doi:10.70675/28453a20z78f2z4fa2za6edz700065e69e97
Integrating Language and Vision Models for Real-Time Assistance for the Visually Impaired
Angadi, A. V., Koppal, A. N., Atrey, A. S., Surya, C. R. S., & Mamatha, H. R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14757-8_27
DHEval: A Dynamic Hallucination Evaluation Protocol Robust to Data Contamination
Yin, J., Li, D., Wang, W., Yang, P., Wang, H., Ma, Y., Huang, Y., & Qi, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462032
High-Precision Retrieval-Augmented Generation for Enhanced Question Answering Systems
Dai, Z. (2026) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp69961.2026.11540851
Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams
Lee, Y., Lim, Y., Jung, W., Choi, W., & Yoon, S. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817994
Chen, S., Zhu, J., & Huang, C. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-0288-1_15
Interactivity in Grounded Voice Assistants: Streaming Speech-to-Speech RAG and Ambiguity Resolution
Ahmer, S. (2026) · Aaltodoc (Aalto University)
Kayadibi, S. Y. (2026) · Research Square · doi:10.21203/rs.3.rs-10876467/v1
Sekharan, C. N., & Detroja, M. R. (2026) · IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi69914.2026.11685844
Wu, X., Li, S., Liu, S., & Guo, Z. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.105205
Completing the knowledge lifecycle for language models
Zhang, Z. (2024) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Structured but Silent: Probing Capability Requirements in LLM Hidden StatesPreprint
Choo, K., Song, M., Kang, Y., & Park, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.08018
Lam, P. B. N., & Anh, N. T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.03171
Metadata
- Authors
- Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N., Choi, Y., & Inui, K.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 49025-49043
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133215177
- Cited by
- 52
- References
- 0
- Citations by year
- 2026: 17 · 2025: 11 · 2024: 6 · 2023: 18
- FWCI
- 13.38
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Allen Institute for Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Keywords
- real-time question answering, current events
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.