RealTime QA: What's the Answer Right Now?
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications of question answering and beyond.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, the static nature of parametric memory can lead to information lag, posing challenges in maintaining the relevance of the stored data as it may not reflect the most current information [27, 166].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [166]
In the citing paper
“Abstention is an active area of LLM-related research (Kasai et al. 2023;Röttger et al. 2024;Ahdritz et al. 2024;Kim and Thorne 2024;Madhusudhan et al. 2024) and is central to Islamic tradition.”
From Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions (2025) · cites this paper as Kasai et al. 2023
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.389 · Cited by 299
Evaluation of Retrieval-Augmented Generation: A Survey
Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z. (2025) · Communications in Computer and Information Science · doi:10.1007/978-981-96-1024-2_8 · Cited by 173
Metadata
- Authors
- Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N., Choi, Y., & Inui, K.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 49025-49043
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133215177
- Cited by
- 52
- References
- 0
- Citations by year
- 2026: 17 · 2025: 11 · 2024: 6 · 2023: 18
- FWCI
- 13.38
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Allen Institute for Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Keywords
- real-time question answering, current events
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: RealTime QA: What's the Answer Right Now? https://app.cochat.ai/papers/10.52202/075280-2130
- The paper
- Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N., Choi, Y., & Inui, K. (2023). RealTime QA: What's the Answer Right Now? Advances in Neural Information Processing Systems 36. https://doi.org/10.52202/075280-2130
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.