CoChat
Sign Up
PreprintCited by 3Speech and dialogue systemsInformation Retrieval and Search BehaviorTopic Modeling

User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal

Open access
Liu, Yuhan, Zhang, Michael J. Q., Choi, Eunsol·2025·arXiv
DOI 10.48550/arxiv.2507.23158arXiv 2507.23158
Open this paper in CoChat — free accountOpen PDF

Abstract

Once language models (LMs) are deployed, they can interact with users long-term, ideally evolving based on their feedback. Asking for direct user feedback can be disruptive; thus, we study harvesting implicit user feedback from user-LM interaction logs. We study two user-LM interaction datasets (WildChat and LMSYS). First, we analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs. Second, we study harvesting learning signals from such implicit user feedback. Specifically, we study whether incorporating the contents of user feedback (e.g., user wanted clarification), in addition to the polarity of the feedback, can improve the model performance. We observe mixed results, showing this helps in short human-designed questions (MTBench) but not on longer and more complex questions (WildBench). Together, we provide an in-depth study of implicit user feedback, showing its potential and limitations.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper studies implicit user feedback in human-LLM dialogue logs, showing it is frequent in later turns and useful for understanding users but noisy as a training signal. It finds that incorporating feedback semantics can help on short MTBench questions but not on longer, more complex WildBench questions.

Key findings

  • Dense annotation of 109 conversations (75 LMSYS, 34 WildChat) shows feedback is common in later turns: in WildChat, 85 of 86 annotated turns at turn ≥5 contained feedback; in LMSYS, 24 of 25 did.
  • A new GPT-4o-mini prompt with in-context examples improves automatic feedback identification over prior work: binary accuracy on Sparse rises from 41.4% to 81.1%, and on Dense from 31.5% to 41.6%; fine-grained Dense accuracy rises from 22.3% to 49.0%.
  • Prompts eliciting positive feedback are slightly more toxic than random or negative-feedback prompts in both datasets; in LMSYS, prompts eliciting positive or negative feedback are slightly lower quality than randomly sampled prompts. Model refusals are rare: around 2.5% of responses in LMSYS and fewer than 1% in WildChat.
  • Training on regenerated responses from a stronger model yields gains on MTBench, but incorporating negative feedback semantics does not consistently beat regenerating from scratch; on WildBench, SFT on random data outperformed SFT on negative-feedback data in all evaluations.

Methods

  • Analyzed two real-world user-LLM interaction datasets, LMSYS-chat-1M and WildChat, with new dense annotations over 109 conversations and comparison to prior sparse annotations.
  • Used GPT-4o-mini prompting for automatic feedback classification, Perspective API for toxicity, BenchBuilder-style prompt quality scoring, and embedding clustering to study refusals.
  • Ran response regeneration experiments with GPT-4o-mini as a stronger model, pairwise reward-model evaluation, and SFT/KTO training on 20K conversations, evaluating on MTBench and WildBench with GPT-4 as judge over 5 runs.

Limitations

  • The paper states that users may have different preferences, that it assumes feedback at all conversation positions is equally important, and that it treats feedback as directed at the most recent model response.
  • Evident from the text: automatic feedback detection remains weak in dense and fine-grained settings (e.g., 49.0% fine-grained accuracy on Dense), limiting large-scale use without manual annotation.
  • Evident from the text: feedback signals are noisy because positive feedback can correlate with lower-quality or toxic prompts, and the training gains are mixed across benchmarks, with no consistent benefit on WildBench.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.23158

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

22 of 30 cited sources checked · 10 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § 3.2 Manually Annotated Feedback Dataset · p. 3

    Existing annotations are not comprehensive (not every turn labeled).

    “These existing annotations are not comprehensive (i.e., not every turn in the conversation is labeled).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Don-Yehiya et al., 2024 source
  • Premisereasoning
    § Evaluating Multi-turn Human-LLM Collabora- · p. 8

    Prior feedback harvesting approaches are limited in their task application domain.

    “Most of these approaches are limited in their task application domain.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Hancock et al., 2019 source

      The Introduction describes limitations of earlier feedback methods, particularly their need for scalar rewards or structured feedback. Related Work names several application areas, but neither section says that most approaches are limited in their task application domain. A search for domain restrictions did not establish that claim. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites Pang et al., 2023 source

      The Introduction and §3.1 describe harvesting feedback from human–bot conversations, and Related Work discusses other dialogue methods. The passages found when searching for task-specific approaches and domain limitations do not say that most prior feedback-harvesting approaches are limited in their task application domain. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites Chen et al., 2024 source
  • Interpretationcitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 8

    Concepts from learning sciences can be limited to fully explain user feedback from the real-world LLM-human setting, as only half of the participants (humans) can be characterized.

    “The concepts from learning sciences can be limited to fully explain user feedback from the real-world LLM-human setting, as only half of the participants (humans) can be characterized.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Borges et al., 2023 source

      The Limitations section says the framework’s pedagogical grounding has not been fully validated for LLMs and that failures may reveal differences from human learning. This supports the possibility that learning-sciences concepts do not fully explain LLM feedback, but not the claim that only the human half of an LLM–human interaction can be characterized: the Task section explicitly characterizes the LLM as the learner. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Findingown data
    § Trends of Feedback across Conversation Turns · p. 5

    User prompts from LMSYS that elicited positive responses show the lowest quality.

    “However, such prompts from LMSYS show the lowest quality.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Trends of Feedback across Conversation Turns · p. 4

    Utterances that elicit positive feedback tend to be slightly more toxic than the other two sets (negative feedback and random) in both LMSYS and WildChat.

    “In both datasets, we find that utterances that elicit positive feedback tend to be slightly more toxic than the other two sets.”
    • Also stated in: § 1 Introduction · p. 1 · hedged
      “We find that prompts that elicit positive feedback are slightly lower quality and more toxic than randomly sampled prompts, suggesting potential issue with simply promoting responses that elicited positive feedback.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 17
  • May combine two claims. 18
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 8

    The outcome of a multi-turn interaction can be upper bounded by both human and AI participants.

    “This is due to the outcome of a multi-turn interaction can be upper bounded by both human and AI participants (Chang et al., 2025).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chang et al., 2025 source

      Section 4.2 reports that user-AI accuracy on the math datasets was significantly better than user-alone accuracy, so the human's unaided performance was not an upper bound. Section 4.3 also reports questions where user-AI accuracy exceeded AI-alone accuracy. The paper describes interaction outcomes as varying with the combination, not as bounded by both participants' standalone performance; page numbers are unavailable in our copy. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    User queries are often ambiguous and underspecified, making it challenging for LLMs to generate a satisfactory response in a single attempt.

    “User queries are often ambiguous and underspecified (Liu et al., 2023), making it challenging for LLMs to generate a satisfactory response in a single attempt.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al., 2023 source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 5 Using User Feedback to Improve Model Responses · p. 5

    Prior work has studied training models by guiding them towards responses that elicited positive feedback and away from responses that elicited negative feedback.

    “Prior work has studied training models by guiding them towards responses that elicited positive feedback and away from responses that elicited negative feedback (Ethayarajh et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Ethayarajh et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.3 Automatic Feedback Identification · p. 3

    LLMs have shown promising performances in various classification tasks.

    “LLMs have shown promising performances in various classification tasks (Brown et al., 2020), and prior work (Don-Yehiya et al., 2024; Shaikh et al., 2025) has also explored prompting LLMs (specifically GPT-4o-mini) to classify user feedback in multi-turn user-LLM interactions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Brown et al., 2020 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 9

    Don-Yehiya et al. (2024) also studied naturally occurring, implicit feedback in large-scale human-LLM interactions datasets.

    “Most closely relevant to our work, Don-Yehiya et al. (2024) also studied naturally occurring, implicit feedback in large-scale human-LLM interactions datasets.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Don-Yehiya et al., 2024 source

      Section 4.1 says the paper studies feedback in LMSYS-Chat-1M, a large dataset of real-world human–LLM conversations; Appendix B also reports an analysis of WildChat. But the Introduction distinguishes implicit feedback from the explicit cues the authors say they focus on, so describing their study as one of implicit feedback overstates its scope. Page numbers are unavailable in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2 Background · p. 2

    Shaikh et al. (2025) introduces seven categories of user responses, and five of these categories could be mapped back to the five feedback types from Don-Yehiya et al. (2024).

    “Shaikh et al. (2025) introduces seven categories of user responses, and five of these categories could be mapped back to the five feedback types from Don-Yehiya et al. (2024).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Shaikh et al., 2025 source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites Don-Yehiya et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 8

    LLM performance on multi-turn tasks is worse than on single-turn tasks.

    “(Chang et al., 2025; Laban et al., 2025) shows that LLM performance on multi-turn tasks is worse than on single-turn tasks.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Chang et al., 2025 source

      Section 4.2 reports that GPT-4o's accuracy on the math datasets was significantly lower in user–AI interactions than when the AI answered alone. But the comparison is between user–AI conversations and AI-alone benchmark prompts, not a general comparison of multi-turn and single-turn tasks. The same section says the gap varies by model and dataset and finds no further drop for Llama-3.1-8b on the math datasets. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Laban et al., 2025

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 7.1 Compared Settings · p. 7

    KTO is a method that directly optimizes over non-paired preference data, naturally suitable for learning from raw human feedback from interactions.

    “KTO (Ethayarajh et al., 2024) is a method that directly optimizes over non-paired preference data, which is naturally suitable for learning from raw human feedback from interactions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ethayarajh et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 8

    Prior works have demonstrated the 'interactiveness' of how general users collaborate with language assistants, where ambiguous user queries are usually given at first followed by a series of clarifying actions.

    “Rather than single-pass instruction following, prior works (Lee et al., 2022; Chang et al., 2025; Laban et al., 2025) have demonstrated the”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Lee et al., 2022 source

      Section 3.3 (page not available in our copy) reports that users changed their prompts over time, including adding answer choices and reformulating queries after unhelpful responses. It does not establish that users usually begin with ambiguous queries or then undertake a series of clarifying actions. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Chang et al., 2025 source

      The Introduction and §4.3 describe early user prompts that omit information and are often ambiguous, supporting the point that interaction differs from single-pass instruction following. But Appendix B.3 says the study’s conversations tended to be short and cites real-world data in which around 60% of conversations had only one user–AI turn; it does not support a usual series of clarifying actions. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Laban et al., 2025

      The Introduction (p. 2) says users might begin with an underspecified instruction and clarify their needs over subsequent turns; Background and Related Work (p. 3) describes underspecified instructions as common in real-world human–AI communication. The paper does not establish that general users usually follow this particular sequence or that they typically take a series of clarifying actions; its multi-turn evaluation uses simulated conversations. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 Background · p. 2

    Don-Yehiya et al. (2024) classifies implicit feedback into two categories: positive and negative.

    “Don-Yehiya et al. (2024) classifies implicit feedback into two categories: (1) positive feedback which praises the model's response (i.e., 'Great job!') and (2) negative feedback which signals the model's previous response was not satisfactory.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Don-Yehiya et al., 2024 source

      Section 3.1, “Feedback Taxonomy” (page not provided), distinguishes positive from negative feedback and describes praise or thanks as positive feedback. But it defines five categories—four negative and one positive—not just two categories. The citing sentence captures the broad positive–negative distinction while oversimplifying the paper’s taxonomy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 9

    Instead of framing user feedback as 'positive' and 'negative' feedback, Shaikh et al. provide a more finegrained ontology of multi-turn user responses (e.g., 'acknowledgement').

    “Instead of framing user feedback as 'positive”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Shaikh et al., 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 Background · p. 2

    The remaining two categories, 'Next Turns' and 'Follow-ups', do not belong in feedback.

    “The remaining two categories, 'Next Turns' and 'Follow-ups', do not belong in feedback.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Shaikh et al., 2025 source

      Section 3.3 (p. 4) places Next Turns and Follow-ups under “Advancing Acts,” supporting their classification as grounding acts. It does not say that they cannot belong in feedback; the claim’s exclusion from feedback is unsupported. (LLM, conf=0.75)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 9

    Shaikh et al. (2025) frames this interaction as a natural language grounding task, where both human and LLM initiate grounding acts in a multi-turn nature.

    “Another concurrent work (Shaikh et al., 2025) frames this interaction as a natural language grounding task, where both human and LLM initiate grounding acts in a multi-turn nature.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Shaikh et al., 2025 source

  • May combine two claims.

    Findingown data
    § Trends of Feedback across Conversation Turns · p. 4

    Later user turns frequently contain negative feedback, and positive feedback is rare.

    “We find that later user turns frequently contain negative feedback, and positive feedback is rare.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Trends of Feedback across Conversation Turns · p. 4

    In LMSYS, user prompts in interactions rendering negative feedback are slightly more toxic; in WildChat, no significant difference between negative feedback and random.

    “In LMSYS user prompts in interactions rendering negative feedback are slightly more toxic. In WildChat dataset, we do not see a significant difference between user utterances that invoke negative feedback vs. randomly sampled utterances.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 7.2 Evaluation · p. 7

    WildBench overall represents more challenging examples, with longer and more complex questions than MTBench.

    “WildBench overall represents more challenging examples, with longer and more complex questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Trends of Feedback across Conversation Turns · p. 5

    Model refusals are not common across all settings, always less than 3% of responses.

    “We find that model refusals are not common across all settings, always consisting less than 3% of responses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § E Comparison between MTBench and WildBench Prompts · p. 12

    WildBench contains longer and more complex questions compared to MTBench.

    “Wildbench contains longer and more complex questions compared to MTBench.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § G Statistical Significance Test · p. 13

    On MTBench, all comparisons are statistically significant, confirming that providing targeted fixes with feedback semantics can help models improve on MTBench.

    “On MTBench, all the comparisons are statistically significant, confirming that providing targeted fixes with feedback semantics can help models improve on MTBench.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Feedback in all positions of conversation is of equal importance.
    • The authors state this as an assumption: The feedback is for the most recent model responses.

  • May combine two claims.

    Findingown data
    § D Winrate of LLM-Regenerated Response on WildChat · p. 12

    Human feedback may not always help improve the model's response and the quality can vary across subsets and datasets.

    “3. Human feedback may not always help improve the model's response and the quality can vary across subsets and datasets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 7.2 Evaluation · p. 7

    MTBench Judge provides more comparable scores than WildBench Judge while relative model rankings stay unchanged.

    “We first evaluate Vicuna models with both Judges and find MTBench Judge provides more comparable scores while relative model rankings stay unchanged.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 9 Conclusion · p. 9

    Training on such feedback signals shows performance gains on MTBench but no gains on the harder benchmark.

    “Training on such feedback signals shows performance gains on MTBench but no gains on the harder benchmark.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Feedback in all positions of conversation is of equal importance.
    • The authors state this as an assumption: The feedback is for the most recent model responses.

  • May combine two claims.

    Findingnone given
    § 9 Conclusion · p. 9

    Strong LLMs can help improve on weak model's feedback, but this rewriting is not necessarily better than regeneration from the strong LLM alone.

    “We find that strong LLMs can help improve on weak model's feedback, but this rewriting is not necessarily better than regeneration from the strong LLM alone.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Feedback in all positions of conversation is of equal importance.
    • The authors state this as an assumption: The feedback is for the most recent model responses.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.1 Results · p. 6

    Results on WildChat dataset exhibit similar trends to LMSYS.

    “The results on WildChat dataset (Table 6 in the Appendix) exhibit similar trends.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § 3.2 Manually Annotated Feedback Dataset · p. 3

    Inter-annotator agreement: Cohen's kappa 0.70 binary, 0.74 three-way, 0.60 fine-grained.

    “We report substantial agreement measured by Cohen's kappa: 0.70 for binary classification, 0.74 for three-way classification and 0.60 for fine-grained classification.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § Adding feedback semantics doesn't always help. · p. 7

    The higher win rate for WildChat is because it used GPT models.

    “The number is higher for WildChat at 74%, as it used GPT models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § 3.1 Datasets · p. 3

    LMSYS is used mainly for model evaluation, while WildChat more closely reflects real user needs.

    “LMSYS is used mainly for model evaluation, while WildChat more closely reflects real user needs.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Zheng et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zhao et al., 2024 source

  • May combine two claims.

    Premisereasoning
    § 3.1 Datasets · p. 3

    LMSYS is shorter, containing more edge cases and ill-defined tasks; WildChat has longer interactions and more complex task instructions.

    “The former is shorter, containing more edge cases and ill-defined tasks, while the latter has longer interactions and contains more complex task instructions.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Zheng et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zhao et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 7.2 Evaluation · p. 7

    Both benchmarks use LLMs to rate the scores of model responses.

    “Both benchmarks use LLMs to rate the scores of model responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § Evaluating Multi-turn Human-LLM Collabora- · p. 8

    Random users interacting with LLMs differ significantly from professional educators, limiting the quality and complexity of the feedback provided.

    “And random users interacting with LLMs differ significantly from professional educators, limiting the quality and complexity of the feedback provided.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation
    § Evaluating Multi-turn Human-LLM Collabora- · p. 8

    Most LLMs benefit from GPT-simulated human feedback.

    “Similarly, (Wang et al.) proposes a benchmark to evaluate LLM's performance with GPT-simulated human feedback, claiming that most LLMs benefit from such signal.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang et al.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

31 of 31

  • The authors state this as an assumption.

    A stronger LLM, distinct from the LLM used in user interaction logs, exists and can be used for distillation.

    “We study a distillation setting, where we assume a stronger LLM, distinct from the LLM used in user interaction logs.”

  • The authors state this as an assumption.

    Feedback in all positions of conversation is of equal importance.

    “We also make an assumption that feedback in all positions of conversation is of equal importance.”

  • The authors state this as an assumption.

    The feedback is for the most recent model responses.

    “Finally, we treat the feedback to be for the most recent model responses, while there could be other cases when the user wants to revise earlier model answers.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Sparse evaluation has incompatible gold-label proportions

    Major

    In Table 11, true positives plus false negatives imply that 92.24% of examples are feedback under "Theirs," but only 61.15% are feedback under "Ours." These are presented as results on the same Sparse evaluation set, whose gold-label proportion cannot change with the detection method.

  • Positive and negative feedback are said both to overlap and never overlap

    Minor

    The annotation guidance says positive and negative labels never overlap, but the paper explicitly describes an utterance containing both positive and negative feedback.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Liu, Y., Zhang, M. J. Q., & Choi, E.
Affiliations
New York University (US)
ORCID
Choi, Eunsol
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
2666-2681
DOI
10.48550/arxiv.2507.23158
arXiv
2507.23158
OpenAlex
W4416036929
Cited by
3
References
30
Citations by year
2026: 3
FWCI
2.97
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2521091)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
implicit feedback, human-LLM interaction, feedback content, learning signals, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-16.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-16.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review