CoChat

Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data

Open access
Wei, Jing, Kim, Sungdong, Jung, Hyunhoon, Kim, Young-Ho · 2024 · Proceedings of the ACM on Human-Computer Interaction

The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index; 6 of the 60 sources its claims cite were checked for support.

Abstract

Large language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal such as collecting self-report data from users. We explore what design factors of prompts can help steer chatbots to talk naturally and collect data reliably. To this aim, we formulated four prompt designs with different structures and personas. Through an online study (N = 48) where participants conversed with chatbots driven by different designs of prompts, we assessed how prompt designs and conversation topics affected the conversation flows and users' perceptions of chatbots. Our chatbots covered 79% of the desired information slots during conversations, and the designs of prompts and topics significantly influenced the conversation flows and the data collection performance. We discuss the opportunities and challenges of building chatbots with LLMs.

Overview

CoChat Summary

TL;DR

This paper explores whether GPT-3-powered chatbots can collect self-reported health data through naturalistic conversation by varying prompt structure and persona. In an online study (N=48), zero-shot chatbots covered 79% of desired information slots, with prompt design, topic, and conversation path significantly affecting slot filling and dialogue style.

Key findings

  • Zero-shot GPT-3 chatbots covered 79% of desired information slots overall; slot-filling rates were SP 83%, SN 77%, DP 72%, DN 83%, with the highest SP-Exercise condition at 93% and the lowest SN-Work condition at 64%.
  • Prompt design affected question-asking: structured format increased required-question turn ratio (p = .03) and decreased secondary-question turn ratio (p = .002); descriptive format produced more acknowledging and appreciating turns; the personality modifier increased sympathizing turns (p = .002).
  • Topic and conversation path mattered: Exercise had the highest slot-filling rate (88.4%), significantly higher than Sleep (p = .01), Work (p < .0001), and Food (p = .02); the Positive path had higher slot filling (p = .01) and required-question ratio (p = .01), while the Negative path had higher acknowledging (p < .001) and sympathizing (p < .0001) turn ratios.
  • Errors were present: 6.7% of chatbot turns (257/3,916) were erroneous, including repetition (3.8%), self-talk (1.1%), incorrect phrases (0.79%), and miscellaneous errors (0.9%); 75.7% of conversations ended organically. In exit surveys, 31.3% rated 5 and 50.0% rated 4 on understanding; 22.9% rated 5 and 64.6% rated 4 on acknowledging; empathy ratings had a median of 3, with no significant differences among conditions.

Methods

  • Built 16 GPT-3 (davinci-text-002) chatbots: 4 topics (sleep, food intake, work/productivity, exercise) × 2 information formats (structured vs. descriptive) × 2 personality modifiers (present vs. absent); prompts included job identity and information slots, and ran on a web chat interface.
  • Online between-subject study: 48 participants (aged 19–56, 18 females; 12 per condition) each assigned one prompt design and completed 8 conversations (4 topics × positive/negative paths), yielding 374 dialogues and 7,442 turns.
  • Analysis: manual slot-filling coding; dialogue-act coding with Cohen’s Kappa 0.96 for essential acts and 0.935 for empathy/engagement; mixed-effect models with backward elimination and post-hoc least-squared means; Kruskal-Wallis tests for exit survey ratings.

Limitations

  • Paper-stated: limited participants and imperfect counterbalancing; Food and Sleep always came after Work and Exercise, and Positive always preceded Negative, so fatigue/order effects may remain; some participants may not have followed paths; slots covered only time, scale, binary, and open-ended types; topic differences may reflect lexicon and data-type composition; participant conversation style was uncontrolled.
  • Paper-stated: only GPT-3 davinci-text-002 and zero-shot prompts were tested, so prompts may not generalize to other LLMs; the study focused on chatbot performance/behaviors rather than user experience.
  • Evident from the text: single LLM and small sample limit generalizability; no neutral path was tested; no ending-detection algorithm was implemented (participants clicked “Next”); the system went offline 5 times due to API calls, and repetition/errors may affect data quality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3637364

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

3 Valid · 9 Not valid · 37 Not checked · 251 Own results and interpretations

87 more claims

Create a free account to see all 300 claims and to verify the 54 references still open.

See all claims — free account
  • Probably not a substantial claim. 154
  • It may state two separate claims as one. 91

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 90 references

Research works printed but not linked: 23

  1. Apple Inc. 2022. SIRI shortcuts boost health and fitness routines - Apple News Room. Retrieved Dec 04, 2022 from https://www.apple.com/newsroom/2019/03/siri-shortcuts-boost-health-and-fitness-routines/
  2. Florian Bemmann, Ramona Schödel, Niels Van Berkel, and Daniel Buschek. 2021. Chatbots for Experience Sampling-Initial Opportunities and Challenges.. In IUI Workshops.
  3. Susan E Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. The art of human-computer interface design (1990), 393--404.
  4. John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. 1--19.
  5. Google. 2022. Google Assistant, Your Own Personal Google. Retrieved Dec 04, 2022 from https://assistant.google.com
  6. IBM. 2022. IBM Watson. Retrieved Dec 04, 2022 from https://www.ibm.com/watson
  7. Jae Ho Jeon. 2016. OmniTrack: Designing Flexible and Highly Customizable Quantified-Self Tool. MS thesis. Seoul National University Seoul Korea. http://www.riss.kr/link?id=T14226449
  8. Juju, inc. 2022. Cognitive AI Chatbot. Retrieved Dec 04, 2022 from https://juji.io/
  9. Min Kyung Lee, Sara Kiesler, Jodi Forlizzi, Siddhartha Srinivasa, and Paul Rybski. 2010. Gracefully mitigating breakdowns in robotic services. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 203--210.
  10. Yuhan Luo, Young-Ho Kim, Bongshin Lee, Naeemul Hassan, and Eun Kyoung Choe. 2021. FoodScrap: Promoting Rich Data Capture and Reflective Food Journaling Through Speech Input. In Designing Interactive Systems Conference 2021. 606--618.
  11. Michael McTear. 2018. Conversational modelling for chatbots: current approaches and future directions. Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung (2018), 175--185.
  12. Shikib Mehri, Yasemin Altun, and Maxine Eskenazi. 2022. LAD: Language Models as Data for Zero-Shot Dialog. arXiv preprint arXiv:2207.14393 (2022). · arXiv
  13. Meta Platforms, Inc. 2022. React -- A JavaScript library for building user interfaces. Retrieved Dec 04, 2022 from https://reactjs.org/
  14. Microsoft. 2022. TypeScript. Retrieved Dec 04, 2022 from https://www.typescriptlang.org
  15. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv preprint arXiv:2202.12837 (2022). · arXiv
  16. Elliot Mitchell, Noemie Elhadad, and Lena Mamykina. 2022. Examining AI Methods for Micro-Coaching Dialogs. In CHI Conference on Human Factors in Computing Systems. 1--24.
  17. Ryan Morrison. 2022. GPT-3 developer OpenAI releases new Davinci generative text model. Retrieved Dec 04, 2022 from https://techmonitor.ai/technology/ai-and-automation/gpt-3-openai-davinci-generative-text
  18. OpenAI. 2022. OpenAI API. Retrieved Dec 04, 2022 from https://openai.com/api/
  19. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207 (2018). · arXiv
  20. Kathryn Roulston and Myungweon Choi. 2018. Qualitative interviews. The SAGE handbook of qualitative data collection (2018), 233--249.
  21. Woosuk Seo Chanmo Yang and Young-Ho Kim. 2023. ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events. arXiv:2309.12244 [cs.HC] · arXiv
  22. The OpenJS Foundation. 2022. Node.js. Retrieved Dec 04, 2022 from https://nodejs.org
  23. Anuradha Welivita and Pearl Pu. 2020. A taxonomy of empathetic response intents in human social conversations. arXiv preprint arXiv:2012.04080 (2020). · arXiv

Cited by

See all 86

Metadata

Authors
Wei, J., Kim, S., Jung, H., & Kim, Y.-H.
Affiliations
  • The University of Melbourne (AU)
  • NAVER Cloud (South Korea) (KR)
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2024-04-17
Volume
8
Issue
CSCW1
Pages
1-35
ISSN
2573-0142 (ISSN-L)
OpenAlex
W4396219010
Cited by
84
References
74
Citations by year
2026: 17 · 2025: 37 · 2024: 25
FWCI
11.70
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
large language models, chatbots, prompt engineering, naturalistic conversation, personas, conversation flow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data. https://app.cochat.ai/papers/10.1145/3637364
The paper
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024). Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3637364

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.