The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and Challenges
Abstract
The advent of recent Natural Language Processing technology has led human and machine interactions more toward conversation. In Conversational Search Systems (CSS) like chatbots and Virtual Personal Assistants such as Apple’s Siri, Amazon Alexa, Microsoft’s Cortana, and Google Assistant, both user and device have a limited platform to communicate through chatting or voice. In the information-seeking process, often users do not know how to properly describe their information need in a machine understandable language. Consequently, it is hard for the assistant agent to predict the user’s intent and yield relevant results by only relying on the original query. Studies have shown many unsatisfactory results can be enhanced with the benefit of CSS, which can dig deeper into the user’s query to reveal the real need. This survey intends to provide a comprehensive and comparative overview of ambiguous query clarification task in the context of conversational search technology. We investigate different approaches, their evaluation methods, and future work. We also address the importance of understanding a query for retrieving the most relevant document(s) and satisfying user’s need by predicting their potential request. This work provides an overview of characteristics of ambiguous queries and contributes to better understanding of the existing technologies and challenges in CSS focus on disambiguation of unclear queries from various dimensions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Given these multiple interpretations, ambiguous questions are challenging for existing models (Keyvan and Huang, 2022), but a few prompting techniques have been developed to help address this challenge.”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Keyvan and Huang, 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 1×
“In their second approach called Question Likelihood Maximization (QLM) (see Figure 12), a weakly supervised neural text generation model based on maximum likelihood has been adapted and training made of bidirectional long short-term memory network (BiLSTM) [145] encoders and a LSTM decoder.”
MethodLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · cited 3×
“Followed to Lovin’s work, Porter [123] proposed an algorithm on suffix stripping to reduce the size and complexity of the data.”
MethodClark, H. H. (1996) · doi:10.1017/cbo9780511620539 · Cited by 5,123
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
The vocabulary problem in human-system communication
Furnas, G. W., Landauer, T. K., Gomez, L. M., & Dumais, S. T. (1987) · Communications of the ACM · doi:10.1145/32206.32212 · Cited by 1,504
Question Asking During Tutoring
Graesser, A. C., & Person, N. K. (1994) · American Educational Research Journal · doi:10.3102/00028312031001104 · Cited by 900 · cited 1×
“As summarised in Table 4, Graesser et al. [59] has introduced sixteen question categories by combining Lehnert [92] and Graesser and Person [61] work.”
MethodQuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579
Allen, J. E., Guinn, C. I., & Horvtz, E. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.796083 · Cited by 442
SCHATZMANN, J., WEILHAMMER, K., STUTTLE, M., & YOUNG, S. (2006) · The Knowledge Engineering Review · doi:10.1017/s0269888906000944 · Cited by 336 · cited 1×
“[143], user simulation is “a user model capable of producing responses that a real user might have given in a certain dialogue situation.”
BackgroundTowards Conversational Search and Recommendation: System Ask, User Respond
Zhang, Y., Chen, X., Ai, Q., Yang, L., & Croft, W. B. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271776 · Cited by 319 · cited 3×
“, Can you find me a mobile phone on Amazon? [186]), conversational search and information seeking (e.”
BackgroundInfluentialQuery expansion techniques for information retrieval: a survey
Azad, H. K., & Deepak, A. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.05.009 · Cited by 313
Towards Deep Conversational Recommendations
Li, R., Kahou, S., Schulz, H., Michalski, V., Charlin, L., & Pal, C. (2018) · arXiv · doi:10.48550/arxiv.1812.07617 · Cited by 242
Research works printed but not linked: 172
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, et al. 2016. TensorFlow: A system for Large-scale machine learning. In Proceedings of the 12th USENIX Symposium on OSDI 16. 265–283.
- 10.1145/3459637.3482231 · DOI
- Mohammad Aliannejadi Julia Kiseleva Aleksandr Chuklin Jeff Dalton and Mikhail Burtsev. 2020. ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ). Retrieved from https://arXiv:2009.11352.
- Mohammad Aliannejadi Julia Kiseleva Aleksandr Chuklin Jeffrey Dalton and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue corpora with clarifying questions. Retrieved from https://arXiv:2109.05794. · DOI
- 10.1145/3331184.3331265 · DOI · Paper page
- James F. Allen. 1983. Recognizing intentions from natural language utterances. In Comput. Models Discourse, Michael Brady and Robert C. Berwick (Eds.). MIT Press, 107–166.
- 10.1075/nlp.1.02all · DOI · Paper page
- Avishek Anand, Lawrence Cavedon, Hideo Joho, Mark Sanderson, and Benno Stein. 2020. Conversational search (dagstuhl seminar 19461). In Dagstuhl Reports, Vol. 9. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
- 10.1007/978-3-540-76298-0_52 · DOI · Paper page
- 10.1287/opre.18.5.947 · DOI
- 10.1145/3343413.3377941 · DOI
- Phoebe Ayers, Charles Matthews, and Ben Yates. 2008. How Wikipedia Works: And How You Can Be a Part of It. No Starch Press.
- 10.1007/11880561_9 · DOI · Paper page
- Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. Retrieved from https://arXiv:1409.0473.
- Krisztian Balog. 2021. Conversational AI from an information retrieval perspective: Remaining challenges and a case for user simulation. In Proceedings of the 2nd International DESIRES Conference, Vol. 2950. CEUR-WS.org, 80–90.
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 65–72.
- 10.1007/s10579-009-9081-4 · DOI · Paper page
- Isabel L. Beck et al. 1997. Questioning the Author: An Approach for Enhancing Student Engagement with Text.Newark, DE: International Reading Association.
- Keping Bi, Qingyao Ai, and W. Bruce Croft. 2021. Asking clarifying questions based on negative feedback in conversational search. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval. 157–166.
- Matthew W. Bilotti, Boris Katz, Jimmy Lin, et al. 2004. What works better for question answering: Stemming or morphological query expansion. In Proceedings of the Information Retrieval for Question Answering Workshop at SIGIR (IR4QA’04). 1–3.
- Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? Analyzing clarification questions in CQA. In Proceedings of the Conference on CHIIR. 345–348.
- 10.1016/S0169-7552(98)00110-X · DOI · Paper page
- 10.1145/1401890.1401995 · DOI · Paper page
- Yang Trista Cao, Rao, et al.2019. Controlling the specificity of clarification question generation. In Proceedings of the WNLP@ ACL. 53–56.
- 10.1145/2071389.2071390 · DOI · Paper page
- 10.1145/1645953.1646033 · DOI · Paper page
- 10.1207/s15516709cog1302_7 · DOI · Paper page
- Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In Proceedings of the International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=r1xMH1BtvB.
- Charles L. Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the Trec 2009 Web Track. Technical Report. In Proceedings of the TREC’09.
- Anni Coden, Daniel Gruhl, Neal Lewis, and Pablo N. Mendes. 2015. Did you mean A or B? Supporting clarification dialog for entity disambiguation. In Proceedings of the SumPre-HSWI@ ESWC.
- 10.1163/9789004368811_005 · doi:10.1163/9789004368811_005
- J. Shane Culpepper, Fernando Diaz, and Mark D. Smucker. 2018. Research frontiers in information retrieval: Report from the 3rd strategic workshop on information retrieval in Lorne (SWIRL’18). In Proceedings of the ACM SIGIR Forum, Vol. 52. ACM New York, NY, 34–90.
- 10.5555/534996 · doi:10.5555/534996
- Jeffrey Dalton Chenyan Xiong and Jamie Callan. 2020. TREC CAsT 2019: The conversational assistance track overview. Retrieved from https://arXiv:2003.13624. · doi:10.6028/nist.sp.1266.cast-overview
- 10.1145/3397271.3401206 · DOI
- 10.1145/1718487.1718493 · DOI
- Marco De Boni and Suresh Manandhar. 2003. An analysis of clarification dialogue for question answering. In Proceedings of the NAACL-HLT. 48–55.
- 10.1007/s10462-020-09866-x · DOI · Paper page
- J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the NAACL-HLT, Vol. 1. 4171–4186.
- Kaustubh D. Dhole. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. Retrieved from https://arXiv:2008.07559.
- Kaustubh D. Dhole and Christopher D. Manning. 2020. Syn-QG: Syntactic and shallow semantic rules for question generation. Retrieved from https://arXiv:2004.08694.
- 10.3102/00346543054003327 · DOI
- James Thomas Dillon. 1988. Questioning and Teaching: A Manual of Practice. Croom Helm.
- Efthimis N. Efthimiadis. 1996. Query expansion. Annu. Rev. Info. Sci. Technol. 31 (1996), 121–87.
- 10.18653/v1/D19-1605 · DOI
- Christiane Fellbaum. 1998. WordNet. An Electronic Lexical Database (Language, Speech and Communication). · DOI · Paper page
- 10.18653/v1/2021.eacl-main.197 · DOI
- 10.1007/BF00309225 · DOI
- 10.1145/2911451.2914701 · doi:10.1145/2911451.2914701
- 10.1109/COMPSAC.2019.00070 · DOI
- 10.1561/1500000074 · DOI · Paper page
- Art Graesser, Vasile Rus, and Zhiqiang Cai. 2008. Question classification schemes. In Proceedings of the Workshop on Question Generation. 10–17.
- Arthur C. Graesser, Natalie Person, and John Huber. 1992. Mechanisms that generate questions. Quest. Info. Syst. 2 (1992), 167–187.
- Xiaoxiao Guo Tim Klinger Clemens Rosenbaum Joseph P. Bigus Murray Campbell Ban Kawas Kartik Talamadupula Gerry Tesauro and Satinder Singh. 2016. Learning to query reason and answer questions on ambiguous texts. In Proceedings of the 5th International Conference on Learning Representations (ICLR) . 12 pages.
- 10.1017/S1351324903003176 · DOI
- 10.1002/(SICI)1097-4571(199101)42:1<7::AID-ASI2>3.0.CO;2-P · DOI · Paper page
- 10.1145/3397271.3401061 · DOI
- 10.1145/2505515.2505682 · DOI
- Ahmed Hassan Awadallah, Ranjitha Gurunath Kulkarni, Umut Ozertem, and Rosie Jones. 2015. Characterizing and predicting voice query reformulation. In Proceedings of the 24th ACM CIKM. 543–552.
- 10.1016/S0020-7373(83)80049-2 · DOI
- 10.1016/j.ins.2011.03.007 · doi:10.1016/j.ins.2011.03.007
- 10.18653/v1/2020.coling-industry.8 · DOI
- 10.1145/1645953.1645966 · DOI · Paper page
- 10.1023/A:1026028229881 · DOI
- Xiangji Huang, Ming Zhong, and Luo Si. 2005. York university at TREC 2005: Genomics track. In Proceedings of the 14th Text Retrieval Conference (TREC’05). Citeseer.
- 10.5555/1613715.1613835 · doi:10.5555/1613715.1613835
- Wei Jeng, Daqing He, and Jiepu Jiang. 2013. Users’ perceived difficulties and corresponding reformulation strategies in voice search. In Proceedings of the 7th Annual Symposium on Human-computer Interaction and Information Retrieval.
- 10.1145/2736277.2741669 · DOI
- Jiepu Jiang, Wei Jeng, and Daqing He. 2013. How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search. In Proceedings of the 36th International ACM SIGIR. 143–152.
- 10.1145/3269206.3271808 · DOI
- 10.1145/3404835.3462859 · DOI
- 10.21437/Interspeech.2010-558 · doi:10.21437/interspeech.2010-558
- 10.1016/0004-3702(82)90035-2 · DOI
- 10.1007/s10791-012-9216-x · DOI
- 10.1145/2468356.2468519 · doi:10.1145/2468356.2468519
- 10.1145/https://doi.org · doi:10.1145/https://doi.org
- 10.1145/3468868 · DOI
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Retrieved from https://arXiv:1412.6980.
- 10.1145/https://doi.org · doi:10.1145/https://doi.org
- 10.1016/S0004-3702(99)00101-0 · DOI
- 10.1145/146802.146810 · DOI · Paper page
- Vaibhav Kumar and Alan W. Black. 2020. ClarQ: A large-scale and diverse dataset for clarification question generation. Retrieved from https://arXiv:2006.05986. · DOI
- 10.5555/908336 · doi:10.5555/908336
- Margaret Li Jason Weston and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. Retrieved from https://arXiv:1909.03087.
- 10.3115/1072228.1072378 · DOI · Paper page
- 10.1007/978-3-030-38825-6_13 · DOI
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 74–81.
- 10.1145/3451160 · DOI
- Chang Liu, Jacek Gwizdka, Jingjing Liu, Tao Xu, and Nicholas J. Belkin. 2010. Analysis and evaluation of query reformulations in different task types. Proceedings of the ASIS&T 47, 1 (2010), 1–9.
- 10.1145/2009916.2009974 · DOI
- Yinhan Liu Myle Ott Naman Goyal Jingfei Du et al. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Retrieved from https://arxiv.org/abs/1907.11692. · arXiv
- 10.1007/978-981-15-5573-2 · DOI
- 10.1145/3445029 · DOI
- Tom Lotze Stefan Klut Mohammad Aliannejadi and Evangelos Kanoulas. 2021. Ranking clarifying questions based on predicted user engagement. Retrieved from https://arXiv:2103.06192.
- Julie Beth Lovins. 1968. Development of a stemming algorithm. Mech. Transl. Comput. Linguistics 11, 1–2 (1968), 22–31.
- 10.3115/v1/P14-5010 · DOI · Paper page
- 10.1016/j.ipm.2021.102682 · DOI
- Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Retrieved from https://arXiv:1301.3781.
- Sewon Min Julian Michael Hannaneh Hajishirzi and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. Retrieved from https://arXiv:2004.10645. · DOI · Paper page
- 10.17762/turcomat.v12i2.1526 · DOI
- Dan Moldovan Sanda Harabagiu Marius Pasca A. Harabagiu Rada Mihalcea Roxana Girju Richard Goodrum and Vasile Rus. 1999. Lasso: A tool for surfing the answer net. In Proceedings of the 8th Text Retrieval Conference (TREC’99) . 9 pages. · doi:10.6028/nist.sp.500-246.smu
- 10.1037/0022-0663.88.2.314 · DOI
- G. Craig Murray, Jimmy Lin, and Abdur Chowdhury. 2006. Identification of user sessions with hierarchical agglomerative clustering. Proceedings of the ASIS&T 43, 1 (2006), 1–9.
- Wenjie Ou and Yue Lin. 2020. A clarifying question selection system from NTES_ALONG in Convai3 challenge. Retrieved from https://arXiv:2010.14202.
- 10.1145/3459637.3481932 · DOI
- 10.1016/j.ipm.2021.102734 · DOI
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of ACL. 311–318.
- 10.1145/3437963.3441786 · DOI
- Gustavo Penha and Claudia Hauff. 2020. Challenges in the evaluation of conversational search systems. In Proceedings of the Converse@ KDD.
- 10.3115/v1/D14-1162 · DOI · Paper page
- Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003. On the means for clarification in dialogue. In Current and New Directions in Discourse and Dialogue. Springer, 235–255. · DOI
- 10.1145/3209978.3210124 · DOI
- Filip Radlinski Krisztian Balog Bill Byrne and Karthik Krishnamoorthi. 2019. Coached conversational preference elicitation: A case study in understanding movie preferences. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue (SIGdial) . 353–360. · DOI
- 10.1145/3020165.3020183 · DOI · Paper page
- Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. Retrieved from https://arXiv:1805.04655. · DOI
- Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification questions. Retrieved from https://arXiv:1904.02281.
- 10.18653/v1/N19-1013 · DOI
- Gary Ren, Xiaochuan Ni, M. Malik, and Q. Ke. 2018. Conversational query understanding using sequence to sequence modeling. Proceedings of the WWW. 1715–1724.
- 10.1145/3432726 · DOI
- 10.1016/j.ipm.2005.05.005 · DOI · Paper page
- Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1995. Okapi at TREC-3. In Proceedings of the 3rd Text Retrieval Conference (TREC’95).109–126.
- Joseph Rocchio. 1971. Relevance feedback in information retrieval. The Smart Retrieval System-experiments in Automatic Document Processing. 313–323.
- 10.1145/3366423.3380193 · DOI
- 10.1145/2854946.2854951 · DOI
- 10.1609/aaai.v32i1.11332 · DOI · Paper page
- 10.1007/978-3-030-72113-8_39 · DOI
- 10.1561/1500000040 · DOI
- R. P. Schank. 1986. Explanation Patterns: Understanding Mechanically and Creatively. Psychology Press.
- David Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at NAACL-HLT’04. 136–143.
- Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards facet-driven generation of clarifying questions for conversational search. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval. 167–175.
- 10.1007/978-3-030-72113-8_41 · DOI
- 10.1145/3488560.3498440 · DOI
- Chen Shani Alexander Libov Sofia Tolmach Liane Lewin-Eytan Yoelle Maarek and Dafna Shahaf. 2021. “Alexa what do you do for fun?” Characterizing playful requests with virtual assistants. Retrieved from https://arXiv:2105.05571.
- 10.1145/2600428.2609497 · DOI
- 10.1145/1242572.1242749 · DOI
- 10.1145/2806416.2806493 · DOI · Paper page
- 10.1002/1097-4571(2000)9999:9999<::AID-ASI1591>3.0.CO;2-R · DOI · Paper page
- Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2014. Towards natural clarification questions in dialogue systems. In Proceedings of the AISB Symposium on Questions, Discourse and Dialogue, Vol. 20.
- 10.1177/016327877800100102 · doi:10.1177/016327877800100102
- 10.1145/1242572.1242667 · DOI · Paper page
- 10.1145/3340531.3418513 · DOI
- 10.1007/978-3-030-15712-8_18 · DOI
- 10.1145/3458553.3458570 · doi:10.1145/3458553.3458570
- 10.1145/3176349.3176387 · DOI
- 10.1145/2810355.2810356 · DOI
- 10.1016/j.ipm.2019.102162 · DOI
- Svitlana Vakulenko. 2019. Knowledge-based conversational search. Retrieved from https://arXiv:1912.06859.
- 10.1145/3437963.3441748 · DOI
- 10.1007/978-3-030-72240-1_43 · DOI
- Hans Van der Meij. 1987. Assumptions of information-seeking questions. Quest. Exch. 1, 2 (1987), 111–118.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, et al. 2017. Attention is all you need. In Proceedings of the NeurIPS. 5998–6008.
- 10.1017/S1351324901002789 · DOI · Paper page
- 10.1145/3397271.3401130 · DOI
- 10.1145/2629489 · DOI · Paper page
- 10.1145/3027063.3053175 · DOI
- 10.1145/3459637.3482199 · DOI
- Xiao Wang Craig Macdonald and Iadh Ounis. 2020. Deep reinforced query reformulation for information retrieval. Retrieved from https://arXiv:2007.07987.
- 10.1108/00330330610681295 · DOI · Paper page
- 10.1145/3178876.3186068 · DOI
- Dongfang Xu Peter Jansen Jaycie Martin Zhengnan Xie Vikas Yadav Harish Tayyar Madabushi Oyvind Tafjord and Peter Clark. 2019. Multi-class hierarchical question classification for multiple choice science exams. Retrieved from https://arXiv:1908.05441.
- 10.18653/v1/D19-1172 · DOI
- Liu Yang Hamed Zamani Yongfeng Zhang Jiafeng Guo and W. Bruce Croft. 2017. Neural matching models for question retrieval and next question prediction in conversation. Retrieved from https://arXiv:1707.05409.
- 10.1145/3397271.3401323 · DOI
- Munazza Zaib Wei Emma Zhang Quan Z. Sheng Adnan Mahmood and Yang Zhang. 2021. Conversational question answering: A survey. Retrieved from https://arXiv:2106.00874.
- 10.1145/3397271.3401415 · DOI
- 10.1145/3366423.3380126 · DOI · Paper page
- 10.1145/3340531.3412772 · DOI
- 10.1145/3397271.3401160 · DOI
- 10.1145/3446328 · DOI
- 10.1145/3397271.3401180 · DOI
- 10.1145/3357384.3357967 · DOI
- 10.1145/3340531.3412094 · DOI
Cited by
Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking
Sharma, N., Liao, Q. V., & Xiao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642459 · Cited by 132
The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
Schulhoff, S., Ilie, M., Balepur, N., Kahadze, K., Liu, A., Si, C., Li, Y., Gupta, A., Han, H., Schulhoff, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.06608 · Cited by 118
ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74
Cai, Y., & Tian, S. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13523-7 · Cited by 17
Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries
Malaviya, C., Chang, J. C., Roth, D., Iyyer, M., Yatskar, M., & Lo, K. (2024) · arXiv · doi:10.48550/arxiv.2411.07237 · Cited by 4
Metadata
- Authors
- Keyvan, K., & Huang, J. X.
- Affiliations
- York University, Toronto, Canada
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-40
- ISSN
- 0360-0300
- 1557-7341
- Cited by
- 69
- References
- 16
- Language
- en
- Funders
- Natural Sciences and Engineering Research Council (RGPIN-2020-07157)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.