Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction
Open accessAbstract
With large language models (LLMs), conversational search engines shift how users retrieve information from the web by enabling natural conversations to express their search intents over multiple turns. Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results to understand user experience with the system. However, it is underexplored how and why users ask follow-up queries to continue conversations with conversational search engines and how the follow-up queries signal users' satisfaction. From qualitative analysis of 250 conversational turns from an in-lab user evaluation of Naver Cue:, a commercial conversational search engine, we propose a taxonomy of 18 users' follow-up query patterns from conversational search, comprising two major axes: (1) users' motivations behind continuing conversations (N = 7) and (2) actions of follow-up queries (N = 11). Compared to the existing literature on query reformulations, we uncovered a new set of motivations and actions behind follow-up queries, including asking for subjective opinions or providing natural language feedback on the engine's responses. To analyze conversational search logs with our taxonomy in a scalable and efficient manner, we built an LLM-powered classifier (73% accuracy). With our classifier, we analyzed 2,061 conversational tuples collected from real-world usage logs of Cue: and examined how the conversation patterns from our taxonomy correlates with satisfaction. Our initial findings suggest some signals of dissatisfactions, such as Clarifying Queries, Excluding Condition, and Substituting Condition with follow-up queries. We envision our approach could contribute to automated evaluation of conversation search experience by providing satisfaction signals and grounds for realistic user simulations.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper builds a taxonomy of conversational search follow-up queries from in-lab and real-world Naver Cue: logs, then uses an LLM classifier to test how those patterns correlate with user satisfaction. It finds that certain follow-up patterns—especially Clarifying Queries, Reacting to Response, Excluding Condition, Substituting Condition, and Criticizing Response—are associated with lower satisfaction.
Key findings
- The taxonomy contains 18 themes across two axes: 7 user motivations for continuing the conversation and 11 follow-up query actions.
- The LLM-powered classifier achieved 73% accuracy in the abstract; the paper later reports 74.39% accuracy for Axis 1 (motivations) and 71.11% for Axis 2 (actions).
- Applied to 2,061 real-world conversational tuples, the classifier classified more than 91% of tuples for Axis 1 and more than 80% for Axis 2. Most frequent Axis 1 themes were Exploring Domain, Narrowing Down, Understanding Response, and Clarifying Queries; most frequent Axis 2 themes were Requesting Additional Information, Substituting Condition, and Adding/Specifying Conditions.
- Satisfaction scores differed significantly by theme (Axis 1: H = 56.65, p < 1e-9; Axis 2: H = 79.88, p < 1e-12). Clarifying Queries and Reacting to Response had lower satisfaction scores than other Axis 1 themes; Excluding Condition, Substituting Condition, and Criticizing Response had lower satisfaction scores for Axis 2. Clarifying Queries responses tended to be rated very low in informativeness, while Reacting to Response responses were rated low in credibility/trustworthiness.
Methods
- In-lab data: 72 participants (36 male, 36 female; ages 18–64) completed 108 search sessions with Naver Cue: across 9 predefined tasks; after system errors and exclusions, 100 sessions were used. Conversations were segmented into 607 tuples, and 250 tuples were qualitatively analyzed to build the taxonomy.
- Real-world data: 2,300 conversation tuples were randomly sampled from Naver Cue: live logs between January 1 and February 17, 2024. Eight external evaluators rated satisfaction on a 5-point Likert scale using relevance, informativeness, trustworthiness, expressiveness, and safety; 239 tuples were excluded, leaving 2,061 tuples.
- Classification and analysis: an LLM classifier using GPT-4 (gpt-4-1106-preview, temperature = 1) was iteratively designed for coverage, accuracy, and efficiency, then evaluated on 247 lab tuples not used in taxonomy/classifier construction. Correlations with satisfaction were tested with Kruskal-Wallis and Dunn post-hoc tests.
Limitations
- Stated: the analysis focused only on two consecutive queries, potentially overlooking complex multi-turn conversation structures; satisfaction ratings came from external evaluators rather than users themselves; the search tasks may not capture all conversational patterns; and the classifier was developed with in-lab logs, showing limitations when applied to real-world logs.
- Stated: LLM-powered evaluation may introduce bias, stochastic inconsistency, black-box diagnosis problems, and may fail to capture evolving search-engine usage because the taxonomy is fixed.
- Evident from the text: classifier accuracy is moderate (71.11%–74.39%), so some classification noise is likely; the satisfaction findings are described as initial and correlational, not causal.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 20 cited sources checked · 0 supported · 0 flagged
- Findingown data
For Axis 1, Clarifying Queries and Reacting to Response had significantly different satisfaction scores compared to other themes.
“For Axis 1, the post-hoc analysis with the Dunn test between the themes showed that Clarifying Queries and Reacting to Response had significantly different satisfaction scores compared to other themes (Figure 4).”
Quote in the paperSubstantial claim - Findingown data
For Axis 1 (Purpose), the most frequent themes were Exploring Domain, followed by Narrowing Down, Understanding Response, and Clarifying Queries.
“For Axis 1 (Purpose), the most frequent themes were Exploring Domain, followed by Narrowing Down, Understanding Response, and Clarifying Queries.”
Quote in the paperNot a substantial claim - Findingown data
English prompts consistently yielded higher accuracy than Korean prompts.
“Given that our datasets were in Korean, we also compared the performance of English versus Korean prompts.The English prompts consistently yielded higher accuracy, highlighting a significant increase in performance between the two languages.”
Quote in the paperNot a substantial claim - Findingown data
The paper proposes a taxonomy of 18 users' follow-up query patterns from conversational search, comprising two major axes: users' motivations behind continuing conversations (7 themes) and actions of follow-up queries (11 themes).
“we propose a taxonomy of 18 users' follow-up query patterns from conversational search, comprising two major axes: (1) users' motivations behind continuing conversations (N = 7) and (2) actions of follow-up queries (N = 11).”
- Also stated in: § 1. Introduction · plain
“In total, we organized 18 themes across two axes: (1) users' motivations behind continuing the conversations (N = 7) and (2) patterns of follow-up queries (N = 11).”
- Also stated in: § 3. Taxonomy of Follow-up Queries in Conversational Search · plain
“We describe our taxonomy across two major axes: (1) users' motivations behind continuing conversations (7 themes, Table 2) and (2) the patterns of follow-up queries (11 themes, Table 3).”
Quote in the paperSubstantial claim - Also stated in: § 1. Introduction · plain
- Findingown data
The LLM-powered classifier achieved 73% accuracy.
“we built an LLM-powered classifier (73% accuracy).”
Quote in the paperSubstantial claim
- Findingown data
Clarifying Queries and Reacting to Response had lower satisfaction scores than other themes.
“The heatmap of themes and satisfaction scores (Figure 3) showed that Clarifying Queries and Reacting to Response had lower satisfaction scores than other themes.”
Quote in the paperSubstantial claim - Findingown data
In the case of Clarifying Queries, the responses tend to be rated with very low informativeness.
“In the case of Clarifying Queries, the responses tend to be rated with very low informativeness (Figure 6), while the responses classified as Reacting to Response were rated low regarding credibility (Figure 7).”
Quote in the paperNot a substantial claim - Findingown data
Seeking Different Representations became explicit in the queries of conversational setting, involving queries asking for search results in different or complex forms like photos and tables.
“Seeking Different Representations was one case that became explicit in the queries of conversational setting, which involved queries asking for the search results in different or complex forms like photos and tables (e.g.,”
Quote in the paperNot a substantial claim - Findingown data
Compared to existing literature on query reformulations, the paper uncovered a new set of motivations and actions behind follow-up queries, including asking for subjective opinions or providing natural language feedback on the engine's responses.
“Compared to the existing literature on query reformulations, we uncovered a new set of motivations and actions behind follow-up queries, including asking for subjective opinions or providing natural language feedback on the engine's responses.”
Quote in the paperSubstantial claim - Findingown data
Users are Evaluating Search Results by seeking confirmations and providing feedback on the results to improve search quality, distinguishing it from other themes where information exploration dominated.
“We also observed that users are Evaluating Search Results by seeking confirmations and providing feedback on the results to improve the search quality, distinguishing it from other themes where information exploration dominated.”
Quote in the paperNot a substantial claim - Findingown data
Responses classified as Reacting to Response were rated low regarding credibility.
“In the case of Clarifying Queries, the responses tend to be rated with very low informativeness (Figure 6), while the responses classified as Reacting to Response were rated low regarding credibility (Figure 7).”
Quote in the paperNot a substantial claim - Findingown data
For Axis 2 (Action), the most frequent themes were Requesting Additional Information, Substituting Condition, and Adding/Specifying Conditions.
“For Axis 2 (Action), the most frequent themes were Requesting Additional Information, Substituting Condition, and Adding/Specifying Conditions.”
Quote in the paperNot a substantial claim - Findingown data
For Axis 2, Excluding Condition, Substituting Condition, and Criticizing Response had significantly different satisfaction scores from other themes.
“For Axis 2, the post-hoc analysis with the Dunn test between the themes showed that Excluding Condition, Substituting Condition, and Criticizing Response had significantly different satisfaction scores from other themes.”
Quote in the paperSubstantial claim - Findingown data
Unsatisfactory conversations involve clarifying queries and reacting to the search engine's responses more often.
“we observed that unsatisfactory conversations involve clarifying queries and reacting to search engine's responses more often.”
Quote in the paperSubstantial claim- Rests on an assumption: Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results.
- Interpretationown data
Actively correcting prior queries and reacting to the system response can be potentially negative feedback given by the users.
“This initial observation shows that actively correcting prior queries and reacting to the system response can be potentially negative feedback given by the users.”
Quote in the paperSubstantial claim- Rests on an assumption: Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results.
- Study design and conductown data
The real-world dataset contained 2,061 conversational tuples after excluding 239 tuples where external evaluators could not infer the search intent or rated that the engine completely failed to satisfy the search intent.
“Based on the rating results, we excluded 239 tuples where the external evaluators could not infer the search intent from the query or they rated that the engine completely failed to satisfy the search intent, resulting in a total of 2,061 tuples.”
Quote in the paperSubstantial claim - Interpretationreasoning
The new themes suggest that people had a new mental model for search engines.
“Such new themes of search intentions suggest that people had a new mental model for search engines.”
Quote in the paperSubstantial claim- Rests on an assumption: With conventional search engines, users considered that the search engine's capability would be limited to retrieving documents relevant to the query.
- Interpretationreasoning
With the extensive capabilities of LLM-based search, users would have a new mental model that the search engines would be capable of handling more complex and implicit queries.
“However, with the extensive capabilities of LLM-based search, users would have a new mental model that the search engines would be capable of handling more complex and implicit queries.”
Quote in the paperSubstantial claim - Interpretationreasoning
Users' dissatisfaction with the engine's response may be related to the trustworthiness of the response for Criticizing Response.
“Criticizing Response.It may suggest that the users' dissatisfaction with the engine's response was related to the trustworthiness of the response (Figure 7).”
Quote in the paperNot a substantial claim - Interpretationown data
The co-occurrence of themes suggested that Excluding Condition and Substituting Condition were frequently used to clarify the users' search intents.
“The co-occurrence of the themes suggested that the first two themes were frequently used to clarify the users' search intents (Figure 2).”
Quote in the paperNot a substantial claim - Interpretationreasoning
The analysis method could not capture how users refine their search intents across multiple turns.
“For example, our analysis method could not capture how users refine their search intents across multiple turns.”
Quote in the paperNot a substantial claim
- May combine two claims. 15
- Classified as probably not a substantial claim. 31
May combine two claims.
Findingown dataExcluding Condition, Substituting Condition, and Criticizing Response were correlated with low satisfaction.
“The heatmap of themes and satisfaction scores (Figure 3) showed that Excluding Condition, Substituting Condition, and Criticizing Response were correlated with low satisfaction.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe satisfaction scores were significantly different by each theme (Axis 1: H = 56.65, p < 1e-9; Axis 2: H = 79.88, p < 1e-12).
“We first ran a Kruskal-Wallis test and verified that the satisfaction scores were significantly different by each theme (For Axis 1, H = 56.65,p < 1e-9.For Axis 2, H = 79.88,p < 1e-12).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe LLM-powered classifier achieved an accuracy of 74.39% for user motivations (Axis 1) and 71.11% for the patterns of follow-up queries (Axis 2).
“In the end, the classifier achieved an accuracy of 74.39% for user motivations (Axis 1) and 71.11% for the patterns of follow-up queries (Axis 2).”
- Also stated in: § 6.2. Potential Considerations of Automatic Evaluation · plain
“We observed that our classifier yielded a moderate level of classification accuracy with enough coverage (Axis 1: 74.39%, Axis 2: 71.11%).”
Quote in the paperSubstantial claim- Also stated in: § 6.2. Potential Considerations of Automatic Evaluation · plain
May combine two claims.
Findingown dataThe classifier classified the purposes (Axis 1) of more than 91% of the 2,061 real-world tuples and the actions (Axis 2) of more than 80% of the tuples into taxonomy.
“Our classifier classified the purposes (Axis 1) of more than 91% of the tuples and the actions (Axis 2) of more than 80% of the tuples into taxonomy, while the remaining tuples were left 'unclassified.'”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe classifier showed some limitations with real-world conversation logs.
“we observed some limitations of the classifier with real-world conversation logs.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The few-shot examples used in the classifier prompt are always correct.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe colored squares in Figure 4 signify statistically significant differences in satisfaction scores between the themes.
“The colored squares signify statistically significant differences in satisfaction scores between the themes.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataInitial findings suggest some signals of dissatisfactions, such as Clarifying Queries, Excluding Condition, and Substituting Condition with follow-up queries.
“Our initial findings suggest some signals of dissatisfactions, such as Clarifying Queries, Excluding Condition, and Substituting Condition with follow-up queries.”
Quote in the paperSubstantial claim- Rests on an assumption: Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe usage of LLM classification may not be suited for nuanced and context-specific tasks.
“They pointed out that the usage of LLM classification may not be suited for nuanced and context-specific tasks.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ashwin et al. [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataThe taxonomy was built through standard thematic analysis with independent open coding by two authors on 100 randomly sampled tuples, then three authors coded 150 additional tuples and iterated, and final codes were decided for 250 tuples.
“We followed a standard thematic analysis process to build our taxonomy.We segmented each conversation from the in-lab conversation logs to produce conversational tuples, ending up with 607 tuples in total.Two authors independently conducted initial open coding on a total of 100 randomly sampled tuples and discussed together to produce an initial taxonomy.”
- Also stated in: § -Reacting to · plain
“To revise the taxonomy, three authors additionally coded 150 tuples and iterated on the initial taxonomy.After the final revision of the taxonomy, we decided on the final codes for 250 tuples.”
Quote in the paperNot a substantial claim- Also stated in: § -Reacting to · plain
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningInsights from the follow-up query patterns of individual users could support designing personalized search paths for each user.
“Furthermore, insights from the follow-up query patterns of individual users could support designing personalized search paths for each user.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductown dataIn-lab participants were recruited with prior experience using generative AI applications but not generative search.
“We recruited participants with prior experience using generative AI applications (e.g., Chat-GPT) but not generative search (e.g., Bing Chat, Bard), preventing their prior experience from”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningUnderstanding Response and Requesting Additional Information suggest that users were satisfied with the response in general but needed more information to completely satisfy their search intent.
“On the other hand, Understanding Response and Requesting Additional Information suggest that the users were satisfied with the response in general but needed more information to completely satisfy their search intent, which could inform directions for improving the engine performance.”
Quote in the paperSubstantial claim- Rests on an assumption: Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationown dataThe taxonomy themes show how query reformulation and search results exploration stages are conducted with LLM-powered conversational search.
“Our themes show how query reformulation and search results exploration stages are conducted with LLM-powered conversational search.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPrevious IR studies documented users' interaction patterns signaling satisfaction, such as clicks on search results, query reformulations, and query abandonments.
“Previous IR studies have documented users' interaction patterns signaling users' satisfaction, such as clicks on search results [2], query reformulations [3,4,5], and query abandonments [6].”
Quote in the paperNot a substantial claim0 of 5 sources checked- cites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [4]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationreasoningClarifying Queries and Substituting Condition could directly relate to users' challenges in conveying their search intent to the engine.
“For example, Clarifying Queries and Substituting Condition could directly relate to users' challenges in conveying their search intent to the engine.”
Quote in the paperNot a substantial claim- Rests on an assumption: Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningWith conventional search engines, such intentions would have been expressed as switching to image search or browsing and sensemaking multiple web pages rather than separate queries.
“With conventional search engines, such intentions would have been expressed as switching to image search or browsing and sensemaking multiple web pages rather than separate queries.”
Quote in the paperNot a substantial claim- Rests on an assumption: With conventional search engines, users considered that the search engine's capability would be limited to retrieving documents relevant to the query.
Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningThe taxonomy could contribute to a better understanding of users' intent behind follow-up queries, uncovering new types of user needs in conversational search settings.
“Expanding and comparing with previous taxonomies, our taxonomy could contribute to a better understanding of users' intent behind follow-up queries, thereby uncovering new types of user needs in conversational search settings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataDue to a system error, the log of six participants who completed only one conversation task was lost; two single-turn sessions were excluded; finally 100 sessions were used.
“Due to the system error, we lost the log of six participants who completed only one conversation task.We also excluded two single-turn sessions, and finally, we used 100 sessions in total.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationLLM-powered evaluation could lead to biased evaluation.
“Still, we acknowledge that LLM-powered evaluation could lead to several potential problems.First, LLMs could lead to biased evaluation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationAshwin et al. showed that using LLMs for annotating interview transcripts could lead to biased annotations.
“For example, Ashwin et al. [40] showed that using LLMs for annotating interview transcripts could lead to biased annotations.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ashwin et al. [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenEach participant was randomly assigned to one or two of the nine pre-defined tasks.
“Each participant was randomly assigned to one or two of the nine pre-defined tasks.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThis behavior aligns with interrogating behaviors from conversational searches proposed by Azzopardi et al., but not previously observed in conventional search.
“This behavior aligns with interrogating behaviors from conversational searches proposed by Azzopardi et al. [16,1], but not previously observed in conventional search.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Azzopardi et al. [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Azzopardi et al. [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: With conventional search engines, users considered that the search engine's capability would be limited to retrieving documents relevant to the query.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationExisting work explored interaction patterns of satisfaction/dissatisfaction and predicted user satisfaction in conversational agents.
“In terms of conversational interaction, existing work has explored diverse interaction scenarios with conversational agents to understand interaction patterns of satisfaction/dissatisfaction [8,9,10] and predict user satisfaction [11,12,13].”
Quote in the paperNot a substantial claim0 of 6 sources checked- cites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [13]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductown dataEight external evaluators trained for search evaluation rated satisfaction on a 5-point Likert scale following the company's internal offline evaluation protocol, considering relevance, informativeness, trustworthiness, expressiveness, and safety.
“Following the company's internal offline evaluation protocol, we recruited eight external evaluators trained for the search evaluation.The evaluators rated the”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA thread of research investigated and conceptualized interaction patterns for information-seeking conversations.
“At the intersection of information search and conversational interactions, a thread of research investigated and conceptualized interaction patterns for information-seeking conversations [14,15,16,17].”
Quote in the paperNot a substantial claim0 of 4 sources checked- cites [14]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningThe diverse search tasks could still be insufficient to capture all conversational patterns with search engines.
“it could still be insufficient to capture all conversational patterns with search engines.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe study used diverse types of search tasks to capture realistic use cases of search engines within the taxonomy.
“Third, while we used diverse types of search tasks to capture realistic use cases of search engines within our taxonomy”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe black-box nature of LLMs could cause problems with diagnosing the causes of misclassification.
“Furthermore, the black-box nature of LLMs could cause problems with diagnosing the causes of the misclassification.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
InterpretationreasoningRating by external evaluators might deviate from the thoughts and search intents of the actual users.
“it might deviate from the thoughts and search intents of the actual users.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe stochastic nature of LLMs may make it challenging to produce consistent results.
“Second, the stochastic nature of LLMs may make it challenging to produce consistent results.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe LLM classifier was developed with logs from in-lab evaluation.
“Lastly, as we developed the LLM classifier with logs from in-lab evaluation”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe study only focused on the relationship between two consecutive queries.
“First, we only focused on the relationship between two consecutive queries”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationUsers can react to the responses of the search engine with natural language utterances to express and refine their search intents, explore the search results, and give feedback to the search results.
“Furthermore, users can react to the responses of the search engine with natural language utterances to express and refine their search intents, explore the search results, and give feedback to the search results [1].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningUsing a fixed set of taxonomy could cause the classifier to fail to capture users' evolving usage of search engines.
“Lastly, as we used a fixed set of taxonomy, the classifier could fail to capture users' evolving usage of search engines.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductown dataThe study collected satisfaction ratings from external evaluators rather than the users themselves.
“Second, we collected the satisfaction ratings from the external evaluators rather than the users themselves.”
Quote in the paperSubstantial claim- Rests on an assumption: Rating by external evaluators yields more objective evaluation results.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenMajority voting may require a significant amount of time and resources.
“but it may require a significant amount of time and resources.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningFocusing only on two consecutive queries potentially overlooked the complex structures of whole conversations.
“potentially overlooking the complex structures of whole conversations.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningRunning the classifier multiple times and using majority voting made results more consistent.
“In the classifier iteration process, we ran the classifier multiple times and used the majority voting to make results more consistent”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results.
“Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results to understand user experience with the system.”
The paper relies on this without support.
With conventional search engines, users considered that the search engine's capability would be limited to retrieving documents relevant to the query.
“With conventional search engines, users considered that the search engine's capability would be limited to retrieving documents relevant to the query.”
The paper relies on this without support.
Users' natural language conversation provides richer signals about users' search intents and evaluation of search results than fixed behavioral signals in conventional keyword-based search.
“Compared to the fixed behavioral signals (e.g., clicks, reformulations, dwell time) with conventional keyword-based search, users' natural language conversation provides richer signals about users' search intents and evaluation of search results.”
The paper relies on this without support.
The complexity of natural language utterances makes it challenging for search designers and developers to discern users' search intent and evaluate responses from the conversation.
“However, the complexity of natural language utterances makes it challenging for search designers and developers to discern the users' search intent and evaluate responses from the conversation.”
The paper relies on this without support.
The user can evolve their search intent from interacting with the search engine, posing additional challenges to understanding the dynamic user intent within the conversation session.
“Furthermore, the user can evolve their search intent from interacting with the search engine, posing additional challenges to understanding the dynamic user intent within the conversation session.”
The paper relies on this without support.
Rating by external evaluators yields more objective evaluation results.
“While rating by external evaluators yielded more objective evaluation results”
The authors state this as an assumption.
The few-shot examples used in the classifier prompt are always correct.
“Assume the examples are ˓→ always correct.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
In-lab session total changes from 96 to 100
MinorThe introduction reports 96 in-lab search sessions, but the data-collection section says 100 sessions were used. The latter also matches the stated 108 sessions minus six lost logs and two excluded sessions.
“we collected a total of 96 sessions of conversational search from 72 lay users”
“Due to the system error, we lost the log of six participants who completed only one conversation task.We also excluded two single-turn sessions, and finally, we used 100 sessions in total.”
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 1
- L. Azzopardi, M. Dubiel, M. Halvey, J. Dalton, Conceptualizing agent-human interactions during the conversational search process, 2018. The Second International Workshop on Conversational Approaches to Information Retrieval, CAIR ; Conference date: 12-07-2018 Through 12-07-2018.
- Authors
- Kim, H., Choi, Y., Yang, T., Lee, H., Park, C., Lee, Y., Kim, J. Y., & Kim, J.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Seoul National University (KR)
- Naver (South Korea) (KR)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2407.13166
- OpenAlex
- W4402346220
- Semantic Scholar
- cb6a9ac38b5c6c73405ec43109a68fc1a3fc3055
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- user satisfaction, query rewriting, search result evaluation, user simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
