Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 69

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and Challenges

Keyvan, Kimiya, Huang, Jimmy Xiangji · 2023 · ACM Computing Surveys

Abstract

The advent of recent Natural Language Processing technology has led human and machine interactions more toward conversation. In Conversational Search Systems (CSS) like chatbots and Virtual Personal Assistants such as Apple’s Siri, Amazon Alexa, Microsoft’s Cortana, and Google Assistant, both user and device have a limited platform to communicate through chatting or voice. In the information-seeking process, often users do not know how to properly describe their information need in a machine understandable language. Consequently, it is hard for the assistant agent to predict the user’s intent and yield relevant results by only relying on the original query. Studies have shown many unsatisfactory results can be enhanced with the benefit of CSS, which can dig deeper into the user’s query to reveal the real need. This survey intends to provide a comprehensive and comparative overview of ambiguous query clarification task in the context of conversational search technology. We investigate different approaches, their evaluation methods, and future work. We also address the importance of understanding a query for retrieving the most relevant document(s) and satisfying user’s need by predicting their potential request. This work provides an overview of characteristics of ambiguous queries and contributes to better understanding of the existing technologies and challenges in CSS focus on disambiguation of unclear queries from various dimensions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3534965

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 172

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, et al. 2016. TensorFlow: A system for Large-scale machine learning. In Proceedings of the 12th USENIX Symposium on OSDI 16. 265–283.
  2. 10.1145/3459637.3482231 · DOI
  3. Mohammad Aliannejadi Julia Kiseleva Aleksandr Chuklin Jeff Dalton and Mikhail Burtsev. 2020. ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ). Retrieved from https://arXiv:2009.11352.
  4. Mohammad Aliannejadi Julia Kiseleva Aleksandr Chuklin Jeffrey Dalton and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue corpora with clarifying questions. Retrieved from https://arXiv:2109.05794. · DOI
  5. 10.1145/3331184.3331265 · DOI · Paper page
  6. James F. Allen. 1983. Recognizing intentions from natural language utterances. In Comput. Models Discourse, Michael Brady and Robert C. Berwick (Eds.). MIT Press, 107–166.
  7. 10.1075/nlp.1.02all · DOI · Paper page
  8. Avishek Anand, Lawrence Cavedon, Hideo Joho, Mark Sanderson, and Benno Stein. 2020. Conversational search (dagstuhl seminar 19461). In Dagstuhl Reports, Vol. 9. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
  9. 10.1007/978-3-540-76298-0_52 · DOI · Paper page
  10. 10.1287/opre.18.5.947 · DOI
  11. 10.1145/3343413.3377941 · DOI
  12. Phoebe Ayers, Charles Matthews, and Ben Yates. 2008. How Wikipedia Works: And How You Can Be a Part of It. No Starch Press.
  13. 10.1007/11880561_9 · DOI · Paper page
  14. Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. Retrieved from https://arXiv:1409.0473.
  15. Krisztian Balog. 2021. Conversational AI from an information retrieval perspective: Remaining challenges and a case for user simulation. In Proceedings of the 2nd International DESIRES Conference, Vol. 2950. CEUR-WS.org, 80–90.
  16. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 65–72.
  17. 10.1007/s10579-009-9081-4 · DOI · Paper page
  18. Isabel L. Beck et al. 1997. Questioning the Author: An Approach for Enhancing Student Engagement with Text.Newark, DE: International Reading Association.
  19. Keping Bi, Qingyao Ai, and W. Bruce Croft. 2021. Asking clarifying questions based on negative feedback in conversational search. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval. 157–166.
  20. Matthew W. Bilotti, Boris Katz, Jimmy Lin, et al. 2004. What works better for question answering: Stemming or morphological query expansion. In Proceedings of the Information Retrieval for Question Answering Workshop at SIGIR (IR4QA’04). 1–3.
  21. Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? Analyzing clarification questions in CQA. In Proceedings of the Conference on CHIIR. 345–348.
  22. 10.1016/S0169-7552(98)00110-X · DOI · Paper page
  23. 10.1145/1401890.1401995 · DOI · Paper page
  24. Yang Trista Cao, Rao, et al.2019. Controlling the specificity of clarification question generation. In Proceedings of the WNLP@ ACL. 53–56.
  25. 10.1145/2071389.2071390 · DOI · Paper page
  26. 10.1145/1645953.1646033 · DOI · Paper page
  27. 10.1207/s15516709cog1302_7 · DOI · Paper page
  28. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In Proceedings of the International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=r1xMH1BtvB.
  29. Charles L. Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the Trec 2009 Web Track. Technical Report. In Proceedings of the TREC’09.
  30. Anni Coden, Daniel Gruhl, Neal Lewis, and Pablo N. Mendes. 2015. Did you mean A or B? Supporting clarification dialog for entity disambiguation. In Proceedings of the SumPre-HSWI@ ESWC.
  31. 10.1163/9789004368811_005 · doi:10.1163/9789004368811_005
  32. J. Shane Culpepper, Fernando Diaz, and Mark D. Smucker. 2018. Research frontiers in information retrieval: Report from the 3rd strategic workshop on information retrieval in Lorne (SWIRL’18). In Proceedings of the ACM SIGIR Forum, Vol. 52. ACM New York, NY, 34–90.
  33. 10.5555/534996 · doi:10.5555/534996
  34. Jeffrey Dalton Chenyan Xiong and Jamie Callan. 2020. TREC CAsT 2019: The conversational assistance track overview. Retrieved from https://arXiv:2003.13624. · doi:10.6028/nist.sp.1266.cast-overview
  35. 10.1145/3397271.3401206 · DOI
  36. 10.1145/1718487.1718493 · DOI
  37. Marco De Boni and Suresh Manandhar. 2003. An analysis of clarification dialogue for question answering. In Proceedings of the NAACL-HLT. 48–55.
  38. 10.1007/s10462-020-09866-x · DOI · Paper page
  39. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the NAACL-HLT, Vol. 1. 4171–4186.
  40. Kaustubh D. Dhole. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. Retrieved from https://arXiv:2008.07559.
  41. Kaustubh D. Dhole and Christopher D. Manning. 2020. Syn-QG: Syntactic and shallow semantic rules for question generation. Retrieved from https://arXiv:2004.08694.
  42. 10.3102/00346543054003327 · DOI
  43. James Thomas Dillon. 1988. Questioning and Teaching: A Manual of Practice. Croom Helm.
  44. Efthimis N. Efthimiadis. 1996. Query expansion. Annu. Rev. Info. Sci. Technol. 31 (1996), 121–87.
  45. 10.18653/v1/D19-1605 · DOI
  46. Christiane Fellbaum. 1998. WordNet. An Electronic Lexical Database (Language, Speech and Communication). · DOI · Paper page
  47. 10.18653/v1/2021.eacl-main.197 · DOI
  48. 10.1007/BF00309225 · DOI
  49. 10.1145/2911451.2914701 · doi:10.1145/2911451.2914701
  50. 10.1109/COMPSAC.2019.00070 · DOI
  51. 10.1561/1500000074 · DOI · Paper page
  52. Art Graesser, Vasile Rus, and Zhiqiang Cai. 2008. Question classification schemes. In Proceedings of the Workshop on Question Generation. 10–17.
  53. Arthur C. Graesser, Natalie Person, and John Huber. 1992. Mechanisms that generate questions. Quest. Info. Syst. 2 (1992), 167–187.
  54. Xiaoxiao Guo Tim Klinger Clemens Rosenbaum Joseph P. Bigus Murray Campbell Ban Kawas Kartik Talamadupula Gerry Tesauro and Satinder Singh. 2016. Learning to query reason and answer questions on ambiguous texts. In Proceedings of the 5th International Conference on Learning Representations (ICLR) . 12 pages.
  55. 10.1017/S1351324903003176 · DOI
  56. 10.1002/(SICI)1097-4571(199101)42:1<7::AID-ASI2>3.0.CO;2-P · DOI · Paper page
  57. 10.1145/3397271.3401061 · DOI
  58. 10.1145/2505515.2505682 · DOI
  59. Ahmed Hassan Awadallah, Ranjitha Gurunath Kulkarni, Umut Ozertem, and Rosie Jones. 2015. Characterizing and predicting voice query reformulation. In Proceedings of the 24th ACM CIKM. 543–552.
  60. 10.1016/S0020-7373(83)80049-2 · DOI
  61. 10.1016/j.ins.2011.03.007 · doi:10.1016/j.ins.2011.03.007
  62. 10.18653/v1/2020.coling-industry.8 · DOI
  63. 10.1145/1645953.1645966 · DOI · Paper page
  64. 10.1023/A:1026028229881 · DOI
  65. Xiangji Huang, Ming Zhong, and Luo Si. 2005. York university at TREC 2005: Genomics track. In Proceedings of the 14th Text Retrieval Conference (TREC’05). Citeseer.
  66. 10.5555/1613715.1613835 · doi:10.5555/1613715.1613835
  67. Wei Jeng, Daqing He, and Jiepu Jiang. 2013. Users’ perceived difficulties and corresponding reformulation strategies in voice search. In Proceedings of the 7th Annual Symposium on Human-computer Interaction and Information Retrieval.
  68. 10.1145/2736277.2741669 · DOI
  69. Jiepu Jiang, Wei Jeng, and Daqing He. 2013. How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search. In Proceedings of the 36th International ACM SIGIR. 143–152.
  70. 10.1145/3269206.3271808 · DOI
  71. 10.1145/3404835.3462859 · DOI
  72. 10.21437/Interspeech.2010-558 · doi:10.21437/interspeech.2010-558
  73. 10.1016/0004-3702(82)90035-2 · DOI
  74. 10.1007/s10791-012-9216-x · DOI
  75. 10.1145/2468356.2468519 · doi:10.1145/2468356.2468519
  76. 10.1145/https://doi.org · doi:10.1145/https://doi.org
  77. 10.1145/3468868 · DOI
  78. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Retrieved from https://arXiv:1412.6980.
  79. 10.1145/https://doi.org · doi:10.1145/https://doi.org
  80. 10.1016/S0004-3702(99)00101-0 · DOI
  81. 10.1145/146802.146810 · DOI · Paper page
  82. Vaibhav Kumar and Alan W. Black. 2020. ClarQ: A large-scale and diverse dataset for clarification question generation. Retrieved from https://arXiv:2006.05986. · DOI
  83. 10.5555/908336 · doi:10.5555/908336
  84. Margaret Li Jason Weston and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. Retrieved from https://arXiv:1909.03087.
  85. 10.3115/1072228.1072378 · DOI · Paper page
  86. 10.1007/978-3-030-38825-6_13 · DOI
  87. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 74–81.
  88. 10.1145/3451160 · DOI
  89. Chang Liu, Jacek Gwizdka, Jingjing Liu, Tao Xu, and Nicholas J. Belkin. 2010. Analysis and evaluation of query reformulations in different task types. Proceedings of the ASIS&T 47, 1 (2010), 1–9.
  90. 10.1145/2009916.2009974 · DOI
  91. Yinhan Liu Myle Ott Naman Goyal Jingfei Du et al. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Retrieved from https://arxiv.org/abs/1907.11692. · arXiv
  92. 10.1007/978-981-15-5573-2 · DOI
  93. 10.1145/3445029 · DOI
  94. Tom Lotze Stefan Klut Mohammad Aliannejadi and Evangelos Kanoulas. 2021. Ranking clarifying questions based on predicted user engagement. Retrieved from https://arXiv:2103.06192.
  95. Julie Beth Lovins. 1968. Development of a stemming algorithm. Mech. Transl. Comput. Linguistics 11, 1–2 (1968), 22–31.
  96. 10.3115/v1/P14-5010 · DOI · Paper page
  97. 10.1016/j.ipm.2021.102682 · DOI
  98. Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Retrieved from https://arXiv:1301.3781.
  99. Sewon Min Julian Michael Hannaneh Hajishirzi and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. Retrieved from https://arXiv:2004.10645. · DOI · Paper page
  100. 10.17762/turcomat.v12i2.1526 · DOI
  101. Dan Moldovan Sanda Harabagiu Marius Pasca A. Harabagiu Rada Mihalcea Roxana Girju Richard Goodrum and Vasile Rus. 1999. Lasso: A tool for surfing the answer net. In Proceedings of the 8th Text Retrieval Conference (TREC’99) . 9 pages. · doi:10.6028/nist.sp.500-246.smu
  102. 10.1037/0022-0663.88.2.314 · DOI
  103. G. Craig Murray, Jimmy Lin, and Abdur Chowdhury. 2006. Identification of user sessions with hierarchical agglomerative clustering. Proceedings of the ASIS&T 43, 1 (2006), 1–9.
  104. Wenjie Ou and Yue Lin. 2020. A clarifying question selection system from NTES_ALONG in Convai3 challenge. Retrieved from https://arXiv:2010.14202.
  105. 10.1145/3459637.3481932 · DOI
  106. 10.1016/j.ipm.2021.102734 · DOI
  107. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of ACL. 311–318.
  108. 10.1145/3437963.3441786 · DOI
  109. Gustavo Penha and Claudia Hauff. 2020. Challenges in the evaluation of conversational search systems. In Proceedings of the Converse@ KDD.
  110. 10.3115/v1/D14-1162 · DOI · Paper page
  111. Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003. On the means for clarification in dialogue. In Current and New Directions in Discourse and Dialogue. Springer, 235–255. · DOI
  112. 10.1145/3209978.3210124 · DOI
  113. Filip Radlinski Krisztian Balog Bill Byrne and Karthik Krishnamoorthi. 2019. Coached conversational preference elicitation: A case study in understanding movie preferences. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue (SIGdial) . 353–360. · DOI
  114. 10.1145/3020165.3020183 · DOI · Paper page
  115. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. Retrieved from https://arXiv:1805.04655. · DOI
  116. Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification questions. Retrieved from https://arXiv:1904.02281.
  117. 10.18653/v1/N19-1013 · DOI
  118. Gary Ren, Xiaochuan Ni, M. Malik, and Q. Ke. 2018. Conversational query understanding using sequence to sequence modeling. Proceedings of the WWW. 1715–1724.
  119. 10.1145/3432726 · DOI
  120. 10.1016/j.ipm.2005.05.005 · DOI · Paper page
  121. Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1995. Okapi at TREC-3. In Proceedings of the 3rd Text Retrieval Conference (TREC’95).109–126.
  122. Joseph Rocchio. 1971. Relevance feedback in information retrieval. The Smart Retrieval System-experiments in Automatic Document Processing. 313–323.
  123. 10.1145/3366423.3380193 · DOI
  124. 10.1145/2854946.2854951 · DOI
  125. 10.1609/aaai.v32i1.11332 · DOI · Paper page
  126. 10.1007/978-3-030-72113-8_39 · DOI
  127. 10.1561/1500000040 · DOI
  128. R. P. Schank. 1986. Explanation Patterns: Understanding Mechanically and Creatively. Psychology Press.
  129. David Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at NAACL-HLT’04. 136–143.
  130. Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards facet-driven generation of clarifying questions for conversational search. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval. 167–175.
  131. 10.1007/978-3-030-72113-8_41 · DOI
  132. 10.1145/3488560.3498440 · DOI
  133. Chen Shani Alexander Libov Sofia Tolmach Liane Lewin-Eytan Yoelle Maarek and Dafna Shahaf. 2021. “Alexa what do you do for fun?” Characterizing playful requests with virtual assistants. Retrieved from https://arXiv:2105.05571.
  134. 10.1145/2600428.2609497 · DOI
  135. 10.1145/1242572.1242749 · DOI
  136. 10.1145/2806416.2806493 · DOI · Paper page
  137. 10.1002/1097-4571(2000)9999:9999<::AID-ASI1591>3.0.CO;2-R · DOI · Paper page
  138. Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2014. Towards natural clarification questions in dialogue systems. In Proceedings of the AISB Symposium on Questions, Discourse and Dialogue, Vol. 20.
  139. 10.1177/016327877800100102 · doi:10.1177/016327877800100102
  140. 10.1145/1242572.1242667 · DOI · Paper page
  141. 10.1145/3340531.3418513 · DOI
  142. 10.1007/978-3-030-15712-8_18 · DOI
  143. 10.1145/3458553.3458570 · doi:10.1145/3458553.3458570
  144. 10.1145/3176349.3176387 · DOI
  145. 10.1145/2810355.2810356 · DOI
  146. 10.1016/j.ipm.2019.102162 · DOI
  147. Svitlana Vakulenko. 2019. Knowledge-based conversational search. Retrieved from https://arXiv:1912.06859.
  148. 10.1145/3437963.3441748 · DOI
  149. 10.1007/978-3-030-72240-1_43 · DOI
  150. Hans Van der Meij. 1987. Assumptions of information-seeking questions. Quest. Exch. 1, 2 (1987), 111–118.
  151. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, et al. 2017. Attention is all you need. In Proceedings of the NeurIPS. 5998–6008.
  152. 10.1017/S1351324901002789 · DOI · Paper page
  153. 10.1145/3397271.3401130 · DOI
  154. 10.1145/2629489 · DOI · Paper page
  155. 10.1145/3027063.3053175 · DOI
  156. 10.1145/3459637.3482199 · DOI
  157. Xiao Wang Craig Macdonald and Iadh Ounis. 2020. Deep reinforced query reformulation for information retrieval. Retrieved from https://arXiv:2007.07987.
  158. 10.1108/00330330610681295 · DOI · Paper page
  159. 10.1145/3178876.3186068 · DOI
  160. Dongfang Xu Peter Jansen Jaycie Martin Zhengnan Xie Vikas Yadav Harish Tayyar Madabushi Oyvind Tafjord and Peter Clark. 2019. Multi-class hierarchical question classification for multiple choice science exams. Retrieved from https://arXiv:1908.05441.
  161. 10.18653/v1/D19-1172 · DOI
  162. Liu Yang Hamed Zamani Yongfeng Zhang Jiafeng Guo and W. Bruce Croft. 2017. Neural matching models for question retrieval and next question prediction in conversation. Retrieved from https://arXiv:1707.05409.
  163. 10.1145/3397271.3401323 · DOI
  164. Munazza Zaib Wei Emma Zhang Quan Z. Sheng Adnan Mahmood and Yang Zhang. 2021. Conversational question answering: A survey. Retrieved from https://arXiv:2106.00874.
  165. 10.1145/3397271.3401415 · DOI
  166. 10.1145/3366423.3380126 · DOI · Paper page
  167. 10.1145/3340531.3412772 · DOI
  168. 10.1145/3397271.3401160 · DOI
  169. 10.1145/3446328 · DOI
  170. 10.1145/3397271.3401180 · DOI
  171. 10.1145/3357384.3357967 · DOI
  172. 10.1145/3340531.3412094 · DOI

Cited by

See all 69

Metadata

Authors
Keyvan, K., & Huang, J. X.
Affiliations
York University, Toronto, Canada
Venue
ACM Computing Surveys
Published
2022-12-07
Volume
55
Issue
6
Pages
1-40
ISSN
  • 0360-0300
  • 1557-7341
Cited by
69
References
16
Language
en
Funders
Natural Sciences and Engineering Research Council (RGPIN-2020-07157)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.