On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Most work on modeling the conversation history in Conversational Question Answering (CQA) reports a single main result on a common CQA benchmark. While existing models show impressive results on CQA leaderboards, it remains unclear whether they are robust to shifts in setting (sometimes to more realistic ones), training data size (e.g., from large to small sets) and domain. In this work, we design and conduct the first large-scale robustness study of history modeling approaches for CQA. We find that high benchmark scores do not necessarily translate to strong robustness, and that various methods can perform extremely differently under different settings. Equipped with the insights from our study, we design a novel prompt-based history modeling approach and demonstrate its strong robustness across various settings. Our approach is inspired by existing methods that highlight historic answers in the passage. However, instead of highlighting by modifying the passage token embeddings, we add textual prompts directly in the passage text. Our approach is simple, easy to plug into practically any model, and highly effective, thus we recommend it as a starting point for future model developers. We also hope that our study and insights will raise awareness to the importance of robustness-focused evaluation, in addition to obtaining high leaderboard scores, leading to better CQA systems.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous research [184,185] has highlighted the lack of robustness in prompts for LLMs, as even minor alterations can yield substantially different outcomes.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [185]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Modern approaches, which are the focus of our work, leverage Transformer-based (Vaswani et al., 2017) pre-trained language models.”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We use the cross-entropy loss and AdamW optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2019).”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2×
“(2021) and fine-tune T5-base (Raffel et al., 2020) on the CANARD dataset (Elgohary et al.”
BackgroundMethodAdvances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“CQA is an extension of the traditional single-turn QA task (Rajpurkar et al., 2016),”
BackgroundMethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 5×
“However, in contrast to most prompting approaches, which predefine the prompt’s location in the input (Liu et al., 2021), our prompts are inserted in different locations for each example.”
BackgroundMethodInfluentialLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 5×
“However, in contrast to most prompting approaches, which predefine the prompt’s location in the input (Liu et al., 2021), our prompts are inserted in different locations for each example.”
BackgroundMethodInfluentialLongformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · cited 9×
“5 For the base LM (used in all settings except high-resource ) we found that a Longformer that was further pre-trained on SQuADv2 (Rajpurkar et al., 2018), 6 achieved consistently better performance than the base Longformer.”
MethodInfluentialKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 1×
“5 For the base LM (used in all settings except high-resource ) we found that a Longformer that was further pre-trained on SQuADv2 (Rajpurkar et al., 2018), 6 achieved consistently better performance than the base Longformer.”
MethodBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · cited 2×
“In addition to Longformer, we evaluated MarCQAp using RoBERTa (Liu et al., 2019), and BigBird (Zaheer et al., 2020) in the standard setting.”
BackgroundCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
QuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714 · cited 23×
“Since QuAC-NH contains incorrect answers, if a k is incorrect we discard E k to avoid corrupting the evaluation set with incorrectly labeled examples.”
BackgroundMethodInfluentialThe Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing
Dror, R., Baumer, G., Shlomov, S., & Reichart, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1128 · Cited by 384
Self-citationQuestion Rewriting for Conversational Question Answering
Vakulenko, S., Longpre, S., Tu, Z., & Anantha, R. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441748 · Cited by 223
How many data points is a prompt worth?
Le Scao, T., & Rush, A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.208 · Cited by 179
BERT with History Answer Embedding for Conversational Question Answering
Qu, C., Yang, L., Qiu, M., Croft, W. B., Zhang, Y., & Iyyer, M. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331341 · Cited by 155 · cited 14×
“…the conversation turns to the input is lacking (Qu et al., 2019a), leading to various proposals of architecture components that explicitly model the conversation history (Choi et al., 2018; Huang et al., 2019; Yeh and Chen, 2019; Qu et al., 2019a,b; Chen et al., 2020; Kim et al., 2021).”
BackgroundMethodResultInfluentialCan You Unpack That? Learning to Rewrite Questions-in-Context
Elgohary, A., Peskov, D., & Boyd-Graber, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1605 · Cited by 150 · cited 2×
“For the external QR model we follow Lin et al. (2020); Vakulenko et al. (2021); Kim et al. (2021) and fine-tune T5-base (Raffel et al., 2020) on the CANARD dataset (El-gohary et al., 2019).”
BackgroundAttentive History Selection for Conversational Question Answering
Qu, C., Yang, L., Qiu, M., Zhang, Y., Chen, C., Croft, W. B., & Iyyer, M. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357905 · Cited by 131 · cited 14×
“…the conversation turns to the input is lacking (Qu et al., 2019a), leading to various proposals of architecture components that explicitly model the conversation history (Choi et al., 2018; Huang et al., 2019; Yeh and Chen, 2019; Qu et al., 2019a,b; Chen et al., 2020; Kim et al., 2021).”
BackgroundMethodResultInfluentialOpen-Domain Question Answering Goes Conversational via Question Rewriting
Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., & Chappidi, S. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.44 · Cited by 130
SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
Zhu, C., Zeng, M., & Huang, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03593 · Cited by 100 · cited 1×
“Early work used recurrent neural networks (RNNs) and variants of attention mechanisms (Reddy et al., 2019; Choi et al., 2018; Zhu et al., 2018).”
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01909 · Cited by 87
Open-Retrieval Conversational Question Answering
Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W. B., & Iyyer, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401110 · Cited by 84 · cited 5×
“…the conversation turns to the input is lacking (Qu et al., 2019a), leading to various proposals of architecture components that explicitly model the conversation history (Choi et al., 2018; Huang et al., 2019; Yeh and Chen, 2019; Qu et al., 2019a,b; Chen et al., 2020; Kim et al., 2021).”
BackgroundMethodInfluentialChen, Y., Wu, L., & Zaki, M. J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/171 · Cited by 82
Statistical Significance Testing for Natural Language Processing
Dror, R., Peled-Cohen, L., Shlomov, S., & Reichart, R. (2020) · Synthesis lectures on human language technologies · doi:10.2200/s00994ed1v01y202002hlt045 · Cited by 75
Self-citationTopiOCQA: Open-domain Conversational Question Answering with Topic Switching
Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00471 · Cited by 74 · cited 1×
“Applying MarCQAp in a multi document settings (Qu et al., 2020; Anan-tha et al., 2021; Adlakha et al., 2022) may result in partial history representation, since the retrieved document may contain only part of the historic answers, therefore MarCQAp will only highlight the answers which appear in…”
MethodDoQA - Accessing Domain-Specific FAQs via Conversational QA
Campos, J. A., Otegi, A., Soroa, A., Deriu, J., Cieliebak, M., & Agirre, E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.652 · Cited by 59
FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension
Yeh, Y. T., & Chen, Y.-N. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5812 · Cited by 57 · cited 9×
“BiDAF++ w/ 2-Context (Choi et al., 2018) 60.1 HAE (Qu et al., 2019a) 62.4 FlowQA (Huang et al., 2019) 64.1 GraphFlow (Chen et al., 2020) 64.9 HAM (Qu et al., 2019b) 65.4 FlowDelta (Yeh and Chen, 2019) 65.5 GHR (Qian et al., 2022) 73.7 RoR (Zhao et al., 2021) 74.9 MarCQAp (Ours) 74.0 chunks, and 61% the resulted chunks…”
BackgroundMethodInfluentialPADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains
Ben-David, E., Oved, N., & Reichart, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00468 · Cited by 54 · cited 2×
“Thus, we expect MarCQAp to perform well not only in high-resource settings, but also in low-resource and domain adaptation settings, in which prompting methods have shown to be particularly useful (Brown et al., 2020; Le Scao and Rush, 2021; Ben-David et al., 2022).”
BackgroundMethodSelf-citationFlowQA: Grasping Flow in History for Conversational Machine Comprehension
Huang, H.-Y., Choi, E., & Yih, W.-T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06683 · Cited by 54 · cited 4×
“BiDAF++ w/ 2-Context (Choi et al., 2018) 60.1 HAE (Qu et al., 2019a) 62.4 FlowQA (Huang et al., 2019) 64.1 GraphFlow (Chen et al., 2020) 64.9 HAM (Qu et al., 2019b) 65.4 FlowDelta (Yeh and Chen, 2019) 65.5 GHR (Qian et al., 2022) 73.7 RoR (Zhao et al., 2021) 74.9 MarCQAp (Ours) 74.0 chunks, and 61%…”
BackgroundMethodInfluentialKim, G., Kim, H., Park, J., & Kang, J. (2021) · doi:10.18653/v1/2021.acl-long.478 · Cited by 45 · cited 6×
“…the conversation turns to the input is lacking (Qu et al., 2019a), leading to various proposals of architecture components that explicitly model the conversation history (Choi et al., 2018; Huang et al., 2019; Yeh and Chen, 2019; Qu et al., 2019a,b; Chen et al., 2020; Kim et al., 2021).”
BackgroundMethodInfluentialHTLM: Hyper-Text Pre-Training and Prompting of Language Models
Aghajanyan, A., Okhonko, D., Lewis, M., Joshi, M., Xu, H., Ghosh, G., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2107.06955 · Cited by 35 · cited 1×
“For instance, Aghajanyan et al. (2022) showed the usefulness of structured pre-training by adding HTML symbols to the input text.”
BackgroundConversational Machine Comprehension: a Literature Review
Gupta, S., Rawat, B. P. S., & Yu, H. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.247 · Cited by 33
RoR: Read-over-Read for Long Document Machine Reading Comprehension
Zhao, J., Bao, J., Wang, Y., Zhou, Y., Wu, Y., He, X., & Zhou, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.160 · Cited by 22
Ditch the Gold Standard: Re-evaluating Conversational Question Answering
Li, H., Gao, T., Goenka, M., & Chen, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.555 · Cited by 22
Capturing Conversational Interaction for Question Answering via Global History Reasoning
Qian, J., Zou, B., Dong, M., Li, X., Aw, A. T., & Yu, H. Z. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.159 · Cited by 6 · cited 3×
“A contemporaneous work proposed a global history attention component, designed to capture long-distance dependencies be-tween conversation turns (Qian et al., 2022”
BackgroundInfluential
Cited by
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Check
Ye, L., Lei, Z., Yin, J., Chen, Q., Zhou, J., & He, L. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657980 · Cited by 51
Enhancing Efficiency and Flexibility of Rapid Prototyping for Scalable Multimodal Intelligent Agents
Chandrasekaran, M. (2024) · Artificial Intelligence for Business (AIxB) · doi:10.1109/aixb62249.2024.00019 · Cited by 9
“Previous studies have revealed a notable deficiency in the robustness of prompts for LLMs, whereby slight modifications might provide very disparate results [10], [11].”
q2d: Turning Questions into Dialogs to Teach Models How to Search
Bitton, Y., Cohen-Ganor, S., Hakimi, I., Lewenberg, Y., Aharoni, R., & Weinreb, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.843 · Cited by 8
Metadata
- Authors
- Gekhman, Z., Oved, N., Keller, O., Szpektor, I., & Reichart, R.
- Affiliations
- Technion – Israel Institute of Technology (IL)
- Google (Israel) (IL)
- Corresponding author
- Gekhman, Zorik
- Oved, Nadav
- Keller, Orgad
- Szpektor, Idan
- Reichart, Roi
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2023
- Volume
- 11
- Pages
- 351-366
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4367165010
- Semantic Scholar
- 3eba01e41208e69942fc0cdaca391d4ed49a99e7
- Cited by
- 9
- References
- 39
- Citations by year
- 2026: 3 · 2024: 3 · 2023: 3
- FWCI
- 0.80
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Keywords
- conversational question answering, robustness, domain shift, training data size, robustness analysis, benchmark scores
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.