Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Open accessThe DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index; 6 of the 60 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal such as collecting self-report data from users. We explore what design factors of prompts can help steer chatbots to talk naturally and collect data reliably. To this aim, we formulated four prompt designs with different structures and personas. Through an online study (N = 48) where participants conversed with chatbots driven by different designs of prompts, we assessed how prompt designs and conversation topics affected the conversation flows and users' perceptions of chatbots. Our chatbots covered 79% of the desired information slots during conversations, and the designs of prompts and topics significantly influenced the conversation flows and the data collection performance. We discuss the opportunities and challenges of building chatbots with LLMs.
Overview
CoChat Summary
TL;DR
This paper explores whether GPT-3-powered chatbots can collect self-reported health data through naturalistic conversation by varying prompt structure and persona. In an online study (N=48), zero-shot chatbots covered 79% of desired information slots, with prompt design, topic, and conversation path significantly affecting slot filling and dialogue style.
Key findings
- Zero-shot GPT-3 chatbots covered 79% of desired information slots overall; slot-filling rates were SP 83%, SN 77%, DP 72%, DN 83%, with the highest SP-Exercise condition at 93% and the lowest SN-Work condition at 64%.
- Prompt design affected question-asking: structured format increased required-question turn ratio (p = .03) and decreased secondary-question turn ratio (p = .002); descriptive format produced more acknowledging and appreciating turns; the personality modifier increased sympathizing turns (p = .002).
- Topic and conversation path mattered: Exercise had the highest slot-filling rate (88.4%), significantly higher than Sleep (p = .01), Work (p < .0001), and Food (p = .02); the Positive path had higher slot filling (p = .01) and required-question ratio (p = .01), while the Negative path had higher acknowledging (p < .001) and sympathizing (p < .0001) turn ratios.
- Errors were present: 6.7% of chatbot turns (257/3,916) were erroneous, including repetition (3.8%), self-talk (1.1%), incorrect phrases (0.79%), and miscellaneous errors (0.9%); 75.7% of conversations ended organically. In exit surveys, 31.3% rated 5 and 50.0% rated 4 on understanding; 22.9% rated 5 and 64.6% rated 4 on acknowledging; empathy ratings had a median of 3, with no significant differences among conditions.
Methods
- Built 16 GPT-3 (davinci-text-002) chatbots: 4 topics (sleep, food intake, work/productivity, exercise) × 2 information formats (structured vs. descriptive) × 2 personality modifiers (present vs. absent); prompts included job identity and information slots, and ran on a web chat interface.
- Online between-subject study: 48 participants (aged 19–56, 18 females; 12 per condition) each assigned one prompt design and completed 8 conversations (4 topics × positive/negative paths), yielding 374 dialogues and 7,442 turns.
- Analysis: manual slot-filling coding; dialogue-act coding with Cohen’s Kappa 0.96 for essential acts and 0.935 for empathy/engagement; mixed-effect models with backward elimination and post-hoc least-squared means; Kruskal-Wallis tests for exit survey ratings.
Limitations
- Paper-stated: limited participants and imperfect counterbalancing; Food and Sleep always came after Work and Exercise, and Positive always preceded Negative, so fatigue/order effects may remain; some participants may not have followed paths; slots covered only time, scale, binary, and open-ended types; topic differences may reflect lexicon and data-type composition; participant conversation style was uncontrolled.
- Paper-stated: only GPT-3 davinci-text-002 and zero-shot prompts were tested, so prompts may not generalize to other LLMs; the study focused on chatbot performance/behaviors rather than user experience.
- Evident from the text: single LLM and small sample limit generalizability; no neutral path was tested; no ending-detection algorithm was implemented (participants clicked “Next”); the system went offline 5 times due to API calls, and repetition/errors may affect data quality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 9 Not valid · 37 Not checked · 251 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Recent Large Language Models (LLMs; e.g. , GPT-3 [17], PaLM [23], OPT [118], HyperCLOVA [41]), with billions of parameters pre-trained on a large amount of language corpora, provide new opportunities for conversational agents.”
What the sources say
- Not checked yetcites [17] [17] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901. -
- Supportedcites [23] [23] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, H source
The citation is an identification one: PaLM is listed as an example of a recent LLM with billions of parameters pre-trained on a large language corpus. Our copy states exactly that in the Introduction (p. 3): a 540 billion parameter, densely activated autoregressive Transformer trained on 780 billion tokens of high-quality text, with state-of-the-art few-shot results. The related-work section (p. 47) also ties scale to conversational/dialogue capability ('dialogue applications have advanced through scale'), which covers the framing half of the citing sentence; the 'new opportunities for conversational agents' conclusion itself is the citing authors' inference rather than PaLM's claim. (LLM, conf=0.85)
- Supportedcites [118] source
OPT is exactly the kind of thing the citing sentence lists it as: the Abstract (p. 1) and Section 2.1 (p. 2) describe OPT as decoder-only transformers from 125M to 175B parameters, and Section 2.3 (p. 2) documents a pre-training corpus of roughly 180B tokens drawn from RoBERTa, the Pile and Reddit data. The conversational-agent part is treated in Section 3.2 Dialogue (p. 5), which states LLMs 'are known to be an integral component of modern dialogue models' and shows unsupervised OPT-175B competitive with the fully supervised BlenderBot 1 on ConvAI2 and other dialogue datasets. The paper does not use the phrase 'new opportunities' — that framing is the citing authors' — and it adds safety caveats (Section 4.5, p. 8), but nothing contradicts the attribution. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While the HCI and CSCW community have actively explored the use of LLMs in various domains ( e.g. , [24, 53, 114]), research that leverages LLMs for powering chatbots, particularly taskoriented ones [12, 72, 103], is still sparse.”
What the sources say
- Not checked yetcites [12] [12] Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang, Sang-Woo Lee, and Woomyoung Park. 2022. Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models. arXiv preprint arXiv:2205.00176 (2022). Jing Wei, Sungdong Kim, Hyunhoon Jung, and Young-Ho Kim -
- Not checked yetcites [72]
- Partly supportedcites [103] sourceExperimental — accuracy not yet measured
Our copy supports only the existence/description of LLM-powered chatbot work with task-oriented capability: the Abstract (p. 1) shows a Codex (GPT-3 finetuned on GitHub) agent that holds multi-turn conversations and answers questions about structured Minecraft data, and the Conclusion (p. 7/9) states 'Codex-powered NPCs can integrate both conversational and task-oriented language interactions almost seamlessly with code generation.' Nothing in our copy asserts that such research is 'still sparse'; the closest field-level statement, in Related Work (p. 3), runs the other way — that prompting for dialog generation 'is rapidly being mainstreamed' — and the related-work survey (pp. 2-3) lists several dialogue/prompting systems (DialoGPT, Meena, PLATO-XL, LaMDA, Madotto et al. 2021, Zheng and Huang 2021) rather than a scarcity of task-oriented chatbot work. So the sparsity characterisation attributed here is the citing authors' own judgement, not this paper's claim. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In terms of utilizing mainstream chatbot platforms to build voice applications, it is of great human effort to come up with user intents and example phrases as well as design conversation flows and logic [27, 68].”
What the sources say
- Could not verifycites [27] [27] Tilman Dingler, Dominika Kwasnicka, Jing Wei, Enying Gong, and Brian Oldenburg. 2021. The use and promise of conversational agents in digital health. Yearbook of Medical Informatics 30, 01 (2021), 191-199. - sourceExperimental — accuracy not yet measured
Section 2 describes tools such as Dialogflow and says they do not require specific programming knowledge. Section 3.3 says custom agents built with open-source frameworks require expertise and development resources, but neither passage addresses whether mainstream platforms require substantial human effort to devise intents, example phrases, and conversation flows. Searches for those terms and for manual design effort found no supporting passage. (LLM, conf=0.72)
- Not checked yetcites [68]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Despite drawbacks discussed above, LLMs-based chatbots can become a valuable and scalable tool for researchers to collect data for personal informatics [35].”
What the sources say
- Could not verifycites [35] sourceExperimental — accuracy not yet measured
Our copy of Homewood et al. (2020) is a fourth-wave HCI methods paper: it proposes 'removal as a method' built on autoethnography, interviews and participant reflection on menstrual-cycle tracking apps (INTRODUCTION p. 1; 'How to use the method' pp. 9-10), and discusses personal informatics data collection only via existing approaches such as photo elicitation, surveys, diaries and autoethnographies (p. 2). No passage mentions LLMs, chatbots, conversational agents or scalability of data collection (searched: chatbot/LLM/conversational agent, artificial intelligence, scalable data collection, valuable method for researchers). The closest statement even cautions against such a generalisation: 'this method would have to be applied to many different cases before we could make generalized statements about its value, limitations and implications as a fourth-wave method' (EXPANDING AND APPLYING REMOVAL AS A METHOD, p. 9). So the claim about LLM-based chatbots cannot be checked against this source. (LLM, conf=0.80)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, 1,482 out of 7,442 turns ( 𝑎𝑣𝑔. 18.45% per dialogue) were classified as Statement (See Table 5).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
87 more claims
Create a free account to see all 300 claims and to verify the 54 references still open.
- Probably not a substantial claim. 154
- It may state two separate claims as one. 91
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As pointed out by existing work in LLMs and natural language processing, there are restrictions to the number of tokens input to the LLMs [6], limiting the length of conversation history texts that could be included in LLM prompting [12,70,116].”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [116]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2 Slot Filling Rate · p. 12
“On the other hand, there were significant randomeffects of the topics( 𝑝 < . 0001), the conversation paths ( 𝑝 = . 01), and the interaction between the information formats and personality modifiers ( 𝑝 < . 001).”
§ 3.4 Data Analysis · p. 11
“Starting from a full model containing participants as a random effect and the four main study factors-information format, personality modifier, topic, and path-and their interactions as fixed effects”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In total, 6.7% of the chatbot turns (257 out of 3,916) were tagged erroneous”
Reported 6.7 · recomputed 6.562819
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4 Data Analysis · p. 11
“To assess the difference among the experimental conditions, we conducted Kruskal-Wallis tests over the four rating questions.”
§ 3.2 Online Study · p. 9
“The survey consisted of three 5-point Likert scale questions and one open-ended feedback textfield.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“in 31 dialogues (SP: 3, SN: 9, DP: 8, DN:10).”
Reported 31 · recomputed 30
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 10×
“LLMs work differently from traditional task-specific models as they accept prompts written in natural language as input to perform various tasks [14, 60].”
BackgroundMethodInfluentialMixed-Effects Models in Sand S-PLUS
Pinheiro, J. C., & Bates, D. M. (2000) · Statistics and Computing · doi:10.1007/978-1-4419-0318-1 · Cited by 11,187
emmeans: Estimated Marginal Means, aka Least-Squares Means
Lenth, R. V., & Piaskowski, J. (2017) · CRAN: Contributed Packages · doi:10.32614/cran.package.emmeans · Cited by 8,143
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Self-citationLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“, [55]), given that LLMs can be sensitive to prompt designs [60], we reckon that our proposed prompts may not yield similar performance on other LLMs due to the differences in the training corpora and the model architecture.”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“, [55]), given that LLMs can be sensitive to prompt designs [60], we reckon that our proposed prompts may not yield similar performance on other LLMs due to the differences in the training corpora and the model architecture.”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1×
“to help people develop long-term adoptions for health monitoring [23, 26, 72].”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2×
“, GPT-3 [17], PaLM [23], OPT [118], HyperCLOVA [41]), with billions of parameters pre-trained on a large amount of language corpora, provide new opportunities for conversational agents.”
BackgroundSelf-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 1×
“We kept the temperature and the presence penalty unchanged based on OpenAI’s suggestions and increased the frequency penalty to reduce the re-use of words.”
MethodLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Research works printed but not linked: 23
- Apple Inc. 2022. SIRI shortcuts boost health and fitness routines - Apple News Room. Retrieved Dec 04, 2022 from https://www.apple.com/newsroom/2019/03/siri-shortcuts-boost-health-and-fitness-routines/
- Florian Bemmann, Ramona Schödel, Niels Van Berkel, and Daniel Buschek. 2021. Chatbots for Experience Sampling-Initial Opportunities and Challenges.. In IUI Workshops.
- Susan E Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. The art of human-computer interface design (1990), 393--404.
- John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. 1--19.
- Google. 2022. Google Assistant, Your Own Personal Google. Retrieved Dec 04, 2022 from https://assistant.google.com
- IBM. 2022. IBM Watson. Retrieved Dec 04, 2022 from https://www.ibm.com/watson
- Jae Ho Jeon. 2016. OmniTrack: Designing Flexible and Highly Customizable Quantified-Self Tool. MS thesis. Seoul National University Seoul Korea. http://www.riss.kr/link?id=T14226449
- Juju, inc. 2022. Cognitive AI Chatbot. Retrieved Dec 04, 2022 from https://juji.io/
- Min Kyung Lee, Sara Kiesler, Jodi Forlizzi, Siddhartha Srinivasa, and Paul Rybski. 2010. Gracefully mitigating breakdowns in robotic services. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 203--210.
- Yuhan Luo, Young-Ho Kim, Bongshin Lee, Naeemul Hassan, and Eun Kyoung Choe. 2021. FoodScrap: Promoting Rich Data Capture and Reflective Food Journaling Through Speech Input. In Designing Interactive Systems Conference 2021. 606--618.
- Michael McTear. 2018. Conversational modelling for chatbots: current approaches and future directions. Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung (2018), 175--185.
- Shikib Mehri, Yasemin Altun, and Maxine Eskenazi. 2022. LAD: Language Models as Data for Zero-Shot Dialog. arXiv preprint arXiv:2207.14393 (2022). · arXiv
- Meta Platforms, Inc. 2022. React -- A JavaScript library for building user interfaces. Retrieved Dec 04, 2022 from https://reactjs.org/
- Microsoft. 2022. TypeScript. Retrieved Dec 04, 2022 from https://www.typescriptlang.org
- Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv preprint arXiv:2202.12837 (2022). · arXiv
- Elliot Mitchell, Noemie Elhadad, and Lena Mamykina. 2022. Examining AI Methods for Micro-Coaching Dialogs. In CHI Conference on Human Factors in Computing Systems. 1--24.
- Ryan Morrison. 2022. GPT-3 developer OpenAI releases new Davinci generative text model. Retrieved Dec 04, 2022 from https://techmonitor.ai/technology/ai-and-automation/gpt-3-openai-davinci-generative-text
- OpenAI. 2022. OpenAI API. Retrieved Dec 04, 2022 from https://openai.com/api/
- Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207 (2018). · arXiv
- Kathryn Roulston and Myungweon Choi. 2018. Qualitative interviews. The SAGE handbook of qualitative data collection (2018), 233--249.
- Woosuk Seo Chanmo Yang and Young-Ho Kim. 2023. ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events. arXiv:2309.12244 [cs.HC] · arXiv
- The OpenJS Foundation. 2022. Node.js. Retrieved Dec 04, 2022 from https://nodejs.org
- Anuradha Welivita and Pearl Pu. 2020. A taxonomy of empathetic response intents in human social conversations. arXiv preprint arXiv:2012.04080 (2020). · arXiv
Cited by
A Systematic Review of Generative AI for Teaching and Learning Practice
Ogunleye, B., Zakariyyah, K. I., Ajao, O., Olayinka, O., & Sharma, H. (2024) · Education Sciences · doi:10.3390/educsci14060636 · Cited by 251
Zhong, W., Luo, J., & Zhang, H. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.04.057 · Cited by 173
MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling
Kim, T., Bae, S., Kim, H. A., Lee, S. H., Hong, H., Yang, C.-M., & Kim, Y. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642937 · Cited by 222
“Recent LLMs have suggested a new paradigm of bootstrapping chatbots [93, 94].”
BackgroundInfluentialSelf-citationYang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120
Pandey, S., & Sharma, S. (2023) · Healthcare Analytics · doi:10.1016/j.health.2023.100198 · Cited by 118
Metadata
- Authors
- Wei, J., Kim, S., Jung, H., & Kim, Y.-H.
- Affiliations
- The University of Melbourne (AU)
- NAVER Cloud (South Korea) (KR)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2024-04-17
- Volume
- 8
- Issue
- CSCW1
- Pages
- 1-35
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4396219010
- Cited by
- 84
- References
- 74
- Citations by year
- 2026: 17 · 2025: 37 · 2024: 25
- FWCI
- 11.70
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- large language models, chatbots, prompt engineering, naturalistic conversation, personas, conversation flow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data. https://app.cochat.ai/papers/10.1145/3637364
- The paper
- Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024). Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3637364
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.