The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
QuAC: Question Answering in Context
Open accessAbstract
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total).The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text.QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation.We also report results for a number of reference models, including a recently state-ofthe-art reading comprehension architecture extended to model dialog context.Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data.Dataset, baseline, and leaderboard available at http://quac.ai.How was perversion handled?How long was he there?How popular did she become?How did Mark Felt contact Woodword?How did the meeting go?How did it do on the charts?When was she born?When was it founded?When was the breakup?
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most datasets that are currently available for training question generation systems are crowd-sourced [42, 47, 84, 92, 114, 115, 119, 120, 122, 123], which largely impacts the quality of generated questions.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [114]
In the citing paper
“For question answering (QA), on SQuADv2 [97], QuAC [24], and many other datasets, fine-tuned models have superior performance, while on CoQA [98], LLMs perform as well as fine-tuned models [25].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“BiDAF++ We use a re-implementation of a top-performing SQuAD model (Peters et al., 2018) that augments bidirectional attention flow (Seo et al., 2016, BiDAF) with self-attention (Clark and Gardner, 2018) and contextualized embeddings.”
MethodSelf-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“Reading Comprehension Our work builds on span based reading comprehension (Rajpurkar et al., 2016; Joshi et al., 2017; Trischler et al., 2016), while also incorporating innovations such as curating questions independently of supporting text to reduce trivial lexical overlap (Joshi et al., 2017;…”
BackgroundSelf-citationYago: a core of semantic knowledge
Suchanek, F. M., Kasneci, G., & Weikum, G. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242667 · Cited by 2,512
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Self-citationSupervised Learning of Universal Sentence Representations from Natural Language Inference Data
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., & Bordes, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1070 · Cited by 2,049 · cited 1×
“Pretrained InferSent To test the importance of lexical matching in our dataset, we output the sentence in s whose pretrained InferSent representation (Conneau et al., 2017) has the highest cosine similarity to that of the question.”
MethodBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 1×
“BiDAF++ We use a re-implementation of a top-performing SQuAD model (Peters et al., 2018) that augments bidirectional attention flow (Seo et al., 2016, BiDAF) with self-attention (Clark and Gardner, 2018) and contextualized embeddings.”
MethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 8×
“adopted in other reading comprehension datasets such as SQuAD, TriviaQA (Joshi et al., 2017), and NewsQA (Trischler et al.”
BackgroundMethodInfluentialSelf-citationDeep Reinforcement Learning for Dialogue Generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1127 · Cited by 1,393 · cited 1×
“19 Dialog fits into an increasing interest in open domain dialog, mostly studied in the context of social chit-chat (Li et al., 2016; Ritter et al., 2011; Fang et al., 2017; Ghazvininejad et al., 2018).”
BackgroundAllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,376 · cited 1×
“The AllenNLP (Gardner et al., 2017) implementation we use reaches 82.”
MethodSelf-citation
Research works printed but not linked: 3
- Hao Fang, Hao Cheng, Elizabeth Clark, Ariel Holtz- man, Maarten Sap, Mari Ostendorf, Yejin Choi, and Noah A Smith. 2017. Sounding board-university of washingtons alexa prize submission. Alexa Prize Proceedings.
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. Conference of the North American Chapter of the Association for Computational Lin- guistics.
- Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Signh, Tim Rocktschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpreta- tion of natural language rules in conversational ma- chine reading. In Proceedings of Empirical Methods in Natural Language Processing.
Cited by
Sorted by citations
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
“Academic researchers have adopted our proposal and released datasets with accompanying datasheets [e.g., 7, 10, 23, 26].”
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
“Academic researchers have adopted our proposal and released datasets with accompanying datasheets [e.g., 7, 10, 23, 26].”
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Pre-Training With Whole Word Masking for Chinese BERT
Cui, Y., Che, W., Liu, T., Qin, B., & Yang, Z. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3124365 · Cited by 1,783
“As we traverse several popular machine reading comprehension benchmarks, such as SQuAD [3], CoQA [4], QuAC [5], NaturalQuestions [6], RACE [7], we can see that most of the top-performing models are based on BERT and its variants [8], [9], [10], demonstrating that the pre-trained language models…”
MethodA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“StoryCloze [308], LAMBADA [309], LCSTS [310], AdGen [311], E2E [312], CHID [313], CHID-FC [286] Physical Knowledge and World Understanding PIQA [314], TriviaQA [315], ARC [316], ARC-Easy [316], ARC-Challenge [316], PROST [317], Open-BookQA [318], WebNLG [319], DogWhistle Insider & Outsider [320] Contextual Language Un…”
InfluentialFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
“Conversational question-answering: QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) 2.”
BackgroundCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
An introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools
Lauriola, I., Lavelli, A., & Aiolli, F. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.103 · Cited by 755
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
“For question answering (QA) , on SQuADv2 [97], QuAC [24], and many other datasets, fine-tuned models have superior performance, while on CoQA [98], LLMs perform as well as fine-tuned models [25].”
BackgroundWizard of Wikipedia: Knowledge-Powered Conversational agentsPreprint
Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., & Weston, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01241 · Cited by 755
“d performing retrieval over the entirety of Wikipedia (Chen et al., 2017). Recently, the QuAC dataset investigates similar themes, but as a sequence of questions and answers in dialogue form instead (Choi et al., 2018). In this work we do not address question answering, but focus on natural human dialogues which conta…”
BackgroundMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment AnalysisPreprint
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02232 · Cited by 553
“Many datasets have been created for MRC from formally written and objective texts, e.g., Wikipedia (WikiReading (Hewlett et al., 2016), SQuAD (Rajpurkar et al., 2016, 2018), Wiki-Hop (Welbl et al., 2018), DRCD (Shao et al., 2018), QuAC (Choi et al., 2018), HotpotQA (Yang et al., 2018)) news and other articles (CNN/Dai…”
BackgroundDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393
“All of these datasets throw in additional complexities to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2019; Choi et al., 2018), requiring passage retrieval (Joshi et al.”
BackgroundDon’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381
“For example, both CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018) take steps to prevent annotators from us-ing words that occur in the context passage, VQA 2.0 (Goyal et al., 2018) selects examples to limit the effectiveness of question-only models, and others have filtered examples solvable…”
BackgroundInfluentialSelf-citationBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 655
“Yes/No questions do appear as a subset of some existing datasets (Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018).”
BackgroundInfluentialClark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00317 · Cited by 477
“Recognizing the need to align research with the impact it will have on real users, the community has responded with datasets of information-seeking questions such as WikiQA (Yang et al., 2015), MS MARCO (Nguyen et al., 2016), QuAC Pronounced tie dye Q. A. —like the colorful t-shirt.”
BackgroundInfluentialSelf-citationELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319
“Extractive QA Extractive question answering datasets such as TREC (Voorhees, 2003), SQuAD (Rajpurkar et al., 2016, 2018), NewsQA (Trischler et al., 2017), SearchQA (Dunn et al., 2017), and QuAC (Choi et al., 2018) constrain the answer to a word or short phrase from the input and evaluate using exact match or F1 with t…”
BackgroundMethodAsking Clarifying Questions in Open-Domain Information-Seeking Conversations
Aliannejadi, M., Zamani, H., Crestani, F., & Croft, W. B. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331265 · Cited by 356
“o the task of question answering. This makes it challenging and more expensive to pair two workers as participants of the same conversation. There are some examples of this approach in the literature [11, 36]; however, they address the task of reading comprehension, a task that is considerably simpler than identifying…”
BackgroundBERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1242 · Cited by 324
“Many datasets have been created for MRC from formally written and objective texts, e.g., Wikipedia (WikiReading (Hewlett et al., 2016), SQuAD (Rajpurkar et al., 2016, 2018), Wiki-Hop (Welbl et al., 2018), DRCD (Shao et al., 2018), QuAC (Choi et al., 2018), HotpotQA (Yang et al., 2018)) news and other articles (CNN/Dai…”
BackgroundRecent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 313
“Question Answering Rajpurkar et al. (2018) LIF A dataset constructed based onQuAC.”
BackgroundMethodInfluentialDREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
Sun, K., Yu, D., Chen, J., Yu, D., Choi, Y., & Cardie, C. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00264 · Cited by 252
“data set construction (Hermann et al., 2015; Hill et al., 2016; Onishi et al., 2016; Rajpurkar et al., 2016; Bajgar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Choi et al., 2018).”
BackgroundSelf-citationBERT with History Answer Embedding for Conversational Question Answering
Qu, C., Yang, L., Qiu, M., Croft, W. B., Zhang, Y., & Iyyer, M. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331341 · Cited by 229
Self-citationMRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 254
“Interest in MRQA also stems from their use in industry applications, such as search engines (Kwiatkowski et al., 2019) and dialogue systems (Reddy et al., 2019; Choi et al., 2018).”
BackgroundSelf-citationFoundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000074 · Cited by 424
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No QuestionsPreprint
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10044 · Cited by 1,070
“Yes/No questions do appear as a subset of some existing datasets (Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018).”
BackgroundInfluentialReCoRD: Bridging the Gap between Human and Machine Commonsense Reading ComprehensionPreprint
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 339
“There are other large-scale datasets (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017; Dunn et al., 2017; Kocisky et al., 2018; Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018) targeting different aspects of reading comprehension.”
BackgroundRetrospective Reader for Machine Reading Comprehension
Zhang, Z., Yang, J., & Zhao, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17705 · Cited by 215
SParC: Cross-Domain Semantic Parsing in Context
Yu, T., Zhang, R., Yasunaga, M., Tan, Y. C., Lin, X., Li, S., Er, H., Li, I., Pang, B., Chen, T., et al. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1443 · Cited by 229
“Recently, some conversational question answering datasets have been introduced, such as QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2018).”
BackgroundRecent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesPreprint
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 289
“While the conversational element of CoQA is not new, e.g., QuAC by Choi, He, Iyyer, Yatskar, Yih, Choi, Liang, and Zettlemoyer (2018), CoQA becomes unique by including questions pertaining directly to commonsense reasoning, following Ostermann et al. (2018) in addressing a growing need for reading comprehension datase…”
BackgroundMethodInfluentialTransformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. A. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 194
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 213
“CANARD [89] is a dataset of rewritten QuAC questions (50% of the original QuAC), and SaAC [9] is similarly based on a collection of TREC resources.”
BackgroundInfluentialCan You Unpack That? Learning to Rewrite Questions-in-Context
Elgohary, A., Peskov, D., & Boyd-Graber, J. L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1605 · Cited by 181
“C ontext A bstraction: N ecessary A dditional R ewritten D iscourse—a new dataset that rewrites Q U AC (Choi et al., 2018) questions.”
BackgroundMethodInfluentialNeural Machine Reading Comprehension: Methods and Trends
Liu, S., Zhang, X., Zhang, S., Wang, H., & Zhang, W. (2019) · Applied Sciences · doi:10.3390/app9183698 · Cited by 160
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Talmor, A., & Berant, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1485 · Cited by 172
“Thus, we do not evaluate abstractive (Nguyen et al., 2016) or conversational datasets (Choi et al., 2018; Reddy et al., 2018).”
BackgroundOpen-Retrieval Conversational Question Answering
Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W. B., & Iyyer, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401110 · Cited by 163
Self-citationGoing Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer
Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., & Pałka, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_47 · Cited by 174
Retrieving and Reading: A Comprehensive Survey on Open-domain Question AnsweringPreprint
Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., & Chua, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00774 · Cited by 343
“The recently released conversational MRC datasets like CoQA [130] and QuAC [131] are aimed at enabling a Reader to answer the latest question by comprehending not only the given context passage but also the conversation history so far.”
BackgroundInfluentialLearning and Evaluating General Linguistic IntelligencePreprint
Yogatama, D., d’Autume, C. D. M., Connor, J. T., Kočiský, T., Chrzanowski, M., Kong, L., Lazaridou, A., Wang, L., Yu, L., Dyer, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11373 · Cited by 212
“We use two additional reading comprehension datasets: TriviaQA (Joshi et al., 2017) and QuAC 4 (Choi et al., 2018) Semantic Role Labeling.”
MethodInterpretation of Natural Language Rules in Conversational Machine Reading
Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., & Riedel, S. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1233 · Cited by 141
“In parallel to our work, both Choi et al. (2018) and Reddy et al. (2018) have to began to investigate QA dialogs with background text.”
BackgroundA Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138
Improving Machine Reading Comprehension with General Reading Strategies
Sun, K., Yu, D., Yu, D., & Cardie, C. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1270 · Cited by 134
“, 2018) are free-form human generated texts based on given documents (Nguyen et al., 2016; Reddy et al., 2018; Choi et al., 2018).”
BackgroundMethodInfluentialMulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132
Conversational question answering: a survey
Zaib, M., Zhang, W. E., Sheng, Q. Z., Mahmood, A., & Zhang, Y. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01744-y · Cited by 146
“The answer Ai can either be a free-form text with evidence [76] or a text span [13].”
BackgroundMethodInfluentialDatasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 268
Query Resolution for Conversational Search with Limited Supervision
Voskarides, N., Li, D., Ren, P., Kanoulas, E., & de Rijke, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401130 · Cited by 117
“omplex information seeking conversations [34], researchers have proposed information seeking tasks such as conversational question answering (QA) over simple contexts, such as a single-paragraph text [7, 37]. In contrast to conversational QA over simple contexts, in conversational search, a user aims to interactively…”
InfluentialMulti-step Reasoning via Recurrent Dual Attention for Visual Dialog
Gan, Z., Cheng, Y., Kholy, A. E., Li, L., Liu, J., & Gao, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1648 · Cited by 112
“, 2018), and text-only conversational question answering (Reddy et al., 2018; Choi et al., 2018).”
BackgroundNeural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · now publishers, Inc. eBooks · doi:10.1561/9781680835533 · Cited by 125
Attentive History Selection for Conversational Question Answering
Qu, C., Yang, L., Qiu, M., Zhang, Y., Chen, C., Croft, W. B., & Iyyer, M. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357905 · Cited by 115
Self-citationBeat the ai: Investigating adversarial human annotation for reading comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · UCL Discovery (University College London) · Cited by 102
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 126
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Ravichander, A., Black, A. W., Wilson, S., Norton, T. B., & Sadeh, N. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1500 · Cited by 116
“Our evaluation metric for answer-sentence selection is sentence-level F1, implemented similar to (Choi et al., 2018; Rajpurkar et al., 2016).”
A Survey on Machine Reading Comprehension—Tasks, Evaluation Metrics and Benchmark Datasets
Zeng, C., Li, S., Li, Q., Hu, J., & Hu, J. (2020) · Applied Sciences · doi:10.3390/app10217640 · Cited by 117
Pretrained Transformers for Text Ranking: BERT and BeyondPreprint
Lin, J., Nogueira, R. F., & Yates, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06467 · Cited by 101
Anik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103
“Many machine learning researchers have begun adopting this procedure when releasing their datasets [21,93,100] and this approach is starting to gain traction in some organizations (e.”
BackgroundA review on big data based on deep neural network approaches
Rithani, M., Kumar, R. P., & Doss, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10512-5 · Cited by 106
“QuAC Dataset was released in 2018 (Choi et al. 2018), in a conversational style, it comprises of 14 000 questions and the situations that go with them.”
MethodA Survey on Machine Reading Comprehension Systems
Baradaran, R., Ghiasi, R., & Amirkhani, H. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000395 · Cited by 108
COBERT: COVID-19 Question Answering System Using BERT
Alzubi, J. A. A., Jain, R., Singh, A., Parwekar, P., & Gupta, M. (2021) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-021-05810-5 · Cited by 100
“The COBERT is model inspired by the DrQA Open Domain Question Answer model, developed by Chen et al. [25].”
BackgroundMethodA Survey of Cutting-edge Multimodal Sentiment Analysis
Singh, U., Abhishek, K., & Azad, H. K. (2024) · ACM Computing Surveys · doi:10.1145/3652149 · Cited by 103
“…Modalities-Physiologicalsignals Self-reported DBNQADataset[89,300],SQuAD[252],CoQA[257],MSMARCO[225],HOTPOTQA[349],NEWSQ[318],QuAC[37],CliCR[306],RACE[348],Quasar[157],StoryClozeTest[305] Question-Answer Visual…”
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Qin, L., Galley, M., Brockett, C., Liu, X., Gao, X., Dolan, B., Choi, Y., & Gao, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1539 · Cited by 100
“Questions are treated as entirely independent in these “single-turn” MRC models, so recent work (e.g., CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018)) focuses on multi-turn MRC, modeling sequences of questions and answers in a conversation.”
BackgroundSelf-citationA Review on Question Generation from Natural Language Text
Zhang, R., Guo, J., Chen, L., Fan, Y., & Cheng, X. (2021) · ACM Transactions on Information Systems · doi:10.1145/3468889 · Cited by 102
“Qu estion A nswering in C ontext (QuAC) [40] is a large-scale dataset of information-seeking dialogues over sections from Wikipedia articles, which contains 14K crowdsourced QA dialogues (100K total QA pairs).”
BackgroundConversational Information Seeking
Zamani, H., Trippas, J. R., Dalton, J., & Radlinski, F. (2023) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000081 · Cited by 178
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
Baheti, A., Ritter, A., Li, J., & Dolan, B. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1431 · Cited by 87
Li, J., Liu, M., Kan, M., Zheng, Z., Wang, Z., Lei, W., Liu, T., & Qin, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.238 · Cited by 96
doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset
Feng, S., Wan, H., Gunasekara, C., Patel, S., Joshi, S., & Lastras, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.652 · Cited by 103
Documenting Computer Vision Datasets
Miceli, M., Yang, T., Naudts, L., Schuessler, M., Serbanescu, D. A., & Hanna, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445880 · Cited by 94
“preprocessing, uses, distribution [22, 23, 47], and maintenance [11, 22, 27, 47].”
Keyvan, K., & Huang, J. X. (2022) · ACM Computing Surveys · doi:10.1145/3534965 · Cited by 95
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Episodic Memory in Lifelong Language Learning
d'Autume, C. D. M., Ruder, S., Kong, L., & Yogatama, D. (2019) · arXiv · doi:10.48550/arxiv.1906.01076 · Cited by 95
Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Sun, K., Yu, D., Yu, D., & Cardie, C. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00305 · Cited by 94
“…researchers have explored a number of options including adding unanswerable (Trischler et al., 2017; Rajpurkar et al., 2018) or conversational (Choi et al., 2018; Reddy et al., 2019) questions that might require reasoning (Zhang et al., 2018a), and designing abstractive answers (Nguyen et…”
BackgroundTopiOCQA: Open-domain Conversational Question Answering with Topic Switching
Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00471 · Cited by 97
“, 2019) and QuAC (Choi et al., 2018) have facilitated much of the research in this area.”
BackgroundMethodASQA: Factoid Questions Meet Long-Form Answers
Stelmakh, I., Luan, Y., Dhingra, B., & Chang, M.-W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.566 · Cited by 176
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 132
Towards goal-oriented semantic signal processing: Applications and future challenges
Kalfa, M., Gok, M., Atalık, A., Tegin, B., Duman, T. M., & Arıkan, O. (2021) · Digital Signal Processing · doi:10.1016/j.dsp.2021.103134 · Cited by 81
“Examples include question answering [38–40], captioning of images and videos [41–44], and discourse parsing [45–47].”
BackgroundConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering
Chen, Z., Li, S., Smiley, C., Ma, Z., Shah, S., & Wang, W. Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.421 · Cited by 198
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents
Shuster, K., Da, J. U., Roller, S., Dinan, E., Boureau, Y.-L., & Weston, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.222 · Cited by 78
A Short Survey of Pre-trained Language Models for Conversational AI-A New Age in NLP
Zaib, M., Sheng, Q. Z., & Zhang, W. E. (2020) · Australasian Computer Science Week Multiconference (ACSW) · doi:10.1145/3373017.3373028 · Cited by 81
“High-quality conversational datasets such as QuAC [4] and CoQA [16] have provided the researchers a great source to work deeply in the field of CMC.”
BackgroundLet the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions
Abbasiantaeb, Z., Yuan, Y., Kanoulas, E., & Aliannejadi, M. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635856 · Cited by 98
“) dataset [11], which serves as a widely recognized benchmark for evaluating the effectiveness of CQA models.”
BackgroundMethodInfluentialGraph Neural Networks for Natural Language Processing: A SurveyPreprint
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06090 · Cited by 70
SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis
Efimov, P., Chertok, A., Boytsov, L., & Braslavski, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58219-7_1 · Cited by 74
Conversational Information Seeking
Dalton, J., Fischer, S., Owoicho, P., Radlinski, F., Rossetto, F., Trippas, J. R., & Zamani, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532678 · Cited by 76
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
Rogers, A., Kovaleva, O., Downey, M., & Rumshisky, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6398 · Cited by 73
“The most interesting alternative was suggested in QuAC: one Turker tries to learn the information from a text that only the other Turker can see (Choi et al. 2018).”
BackgroundTWEETQA: A Social Media Focused Question Answering Dataset
Xiong, W., Wu, J., Wang, H., Kulkarni, V., Yu, M., Chang, S., Guo, X., & Wang, W. Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1496 · Cited by 79
“More recently, researchers propose even challenging datasets that require QA within dialogue or conversational context (Reddy et al., 2018; Choi et al., 2018).”
BackgroundFQuAD: French Question Answering Dataset
d’Hoffschmidt, M., Belblidia, W., Heinrich, Q., Brendlé, T., & Vidal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.107 · Cited by 102
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2021) · ACM Transactions on Information Systems · doi:10.1145/3446426 · Cited by 79
“Fortunately, open-domain QA research has produced QuAC [13], a diverse, large-scale dataset that contains conversational natural language questions of exploratory information needs, as well as CANARD [25], a derived conversational question-in-context rewriting dataset with human-generated questions…”
BackgroundChen, Y., Wu, L., & Zaki, M. J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/171 · Cited by 73
“, 2018) or add previous answer locations to the passage (Choi et al., 2018; Yatskar, 2018), and treat the task as a single-turn MRC while ignoring the important information from the conversation flow.”
BackgroundMethodInfluentialLin, S.-C., Yang, J.-H., Nogueira, R., Tsai, M.-F., Wang, C.-J., & Lin, J. (2020) · arXiv · doi:10.48550/arxiv.2005.02230 · Cited by 62
“Most QA research [26, 40, 50, 56, 59], including conversational QA studies [10, 41], focuses on a restricted version of the open-domain QA problem posed in [8, 18, 20]: returning answers from a finite set of relevant documents—a relevant article [40, 56] or multi-hop hyperlinked documents [58].”
BackgroundMeasuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00486 · Cited by 71
“2019), QUAC (Choi et al. 2018), and CAST-19 (Dalton et al.”
MethodMacaw: An Extensible Conversational Information Seeking Platform
Zamani, H., & Craswell, N. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401415 · Cited by 72
SituatedQA: Incorporating Extra-Linguistic Contexts into QA
Zhang, M. J. Q., & Choi, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.586 · Cited by 108
Self-citationCareful Selection of Knowledge to Solve Open Book Question Answering
Banerjee, P., Pal, K. K., Mitra, A., & Baral, C. R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1615 · Cited by 56
“…answering (Rajpurkar et al., 2016; Joshi et al., 2017; Khashabi et al., 2018; Richardson et al., 2013; Lai et al., 2017; Reddy et al., 2018; Choi et al., 2018; Tafjord et al., 2018; Mitra et al., 2019) and many attempts have been made to solve these challenges (Devlin et al., 2018; Vaswani…”
A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
Yatskar, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1241 · Cited by 60
“In this work, we analyze three such new proposed datasets, SQuAD 2.0 (Rajpurkar et al., 2018), QuAC (Choi et al., 2018), and CoQA (Reddy et al., 2018).”
MethodInfluentialSelf-citationBeyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access
Kim, S., Eric, M., Gopalakrishnan, K., Hedayatnia, B., Liu, Y., & Hakkani-Tür, D. Z. (2020) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.35 · Cited by 64
The role of large language models in agriculture: harvesting the future with LLM intelligence
Shaikh, T. A., Rasool, T., Veningston, K., & Yaseen, S. M. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00359-4 · Cited by 67
XPersona: Evaluating Multilingual Personalized Chatbot
Lin, Z., Liu, Z., Winata, G. I., Cahyawijaya, S., Madotto, A., Bang, Y., Ishii, E., & Fung, P. (2021) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2021.nlp4convai-1.10 · Cited by 66
Interactive Question Answering Systems: Literature Review
Biancofiore, G. M., Deldjoo, Y., Noia, T. D., Sciascio, E. D., & Narducci, F. (2024) · ACM Computing Surveys · doi:10.1145/3657631 · Cited by 61
“Initiative system-driven [47, 167], user-driven [31, 110], mixed [26].”
Arabic machine reading comprehension on the Holy Qur’an using CL-AraBERT
Malhas, R., & Elsayed, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103068 · Cited by 64
The 10 Research Topics in the Internet of Things
Zhang, W. E., Sheng, Q. Z., Mahmood, A., Tran, D. H., Zaib, M., Hamad, S. A., Aljubairy, A., Alhazmi, A., Sagar, S., & Ma, C. (2020) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic50333.2020.00015 · Cited by 54
“Most of the IoT conversational agents are single-turn agents where they do not keep track of the previous conversation, and thus provide inaccurate answers [80], [81].”
Background
Metadata
- Authors
- Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 2174-2184
- OpenAlex
- W2888302696
- Cited by
- 714
- References
- 28
- Citations by year
- 2026: 20 · 2025: 41 · 2024: 84 · 2023: 105 · 2022: 101 · 2021: 150 · 2020: 115 · 2019: 88 · 2018: 10
- FWCI
- 36.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- information-seeking questions, open-ended questions, unanswerable questions, reading comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
