Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Open accessThe DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index; 6 of the 60 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal such as collecting self-report data from users. We explore what design factors of prompts can help steer chatbots to talk naturally and collect data reliably. To this aim, we formulated four prompt designs with different structures and personas. Through an online study (N = 48) where participants conversed with chatbots driven by different designs of prompts, we assessed how prompt designs and conversation topics affected the conversation flows and users' perceptions of chatbots. Our chatbots covered 79% of the desired information slots during conversations, and the designs of prompts and topics significantly influenced the conversation flows and the data collection performance. We discuss the opportunities and challenges of building chatbots with LLMs.
Overview
CoChat Summary
TL;DR
This paper explores whether GPT-3-powered chatbots can collect self-reported health data through naturalistic conversation by varying prompt structure and persona. In an online study (N=48), zero-shot chatbots covered 79% of desired information slots, with prompt design, topic, and conversation path significantly affecting slot filling and dialogue style.
Key findings
- Zero-shot GPT-3 chatbots covered 79% of desired information slots overall; slot-filling rates were SP 83%, SN 77%, DP 72%, DN 83%, with the highest SP-Exercise condition at 93% and the lowest SN-Work condition at 64%.
- Prompt design affected question-asking: structured format increased required-question turn ratio (p = .03) and decreased secondary-question turn ratio (p = .002); descriptive format produced more acknowledging and appreciating turns; the personality modifier increased sympathizing turns (p = .002).
- Topic and conversation path mattered: Exercise had the highest slot-filling rate (88.4%), significantly higher than Sleep (p = .01), Work (p < .0001), and Food (p = .02); the Positive path had higher slot filling (p = .01) and required-question ratio (p = .01), while the Negative path had higher acknowledging (p < .001) and sympathizing (p < .0001) turn ratios.
- Errors were present: 6.7% of chatbot turns (257/3,916) were erroneous, including repetition (3.8%), self-talk (1.1%), incorrect phrases (0.79%), and miscellaneous errors (0.9%); 75.7% of conversations ended organically. In exit surveys, 31.3% rated 5 and 50.0% rated 4 on understanding; 22.9% rated 5 and 64.6% rated 4 on acknowledging; empathy ratings had a median of 3, with no significant differences among conditions.
Methods
- Built 16 GPT-3 (davinci-text-002) chatbots: 4 topics (sleep, food intake, work/productivity, exercise) × 2 information formats (structured vs. descriptive) × 2 personality modifiers (present vs. absent); prompts included job identity and information slots, and ran on a web chat interface.
- Online between-subject study: 48 participants (aged 19–56, 18 females; 12 per condition) each assigned one prompt design and completed 8 conversations (4 topics × positive/negative paths), yielding 374 dialogues and 7,442 turns.
- Analysis: manual slot-filling coding; dialogue-act coding with Cohen’s Kappa 0.96 for essential acts and 0.935 for empathy/engagement; mixed-effect models with backward elimination and post-hoc least-squared means; Kruskal-Wallis tests for exit survey ratings.
Limitations
- Paper-stated: limited participants and imperfect counterbalancing; Food and Sleep always came after Work and Exercise, and Positive always preceded Negative, so fatigue/order effects may remain; some participants may not have followed paths; slots covered only time, scale, binary, and open-ended types; topic differences may reflect lexicon and data-type composition; participant conversation style was uncontrolled.
- Paper-stated: only GPT-3 davinci-text-002 and zero-shot prompts were tested, so prompts may not generalize to other LLMs; the study focused on chatbot performance/behaviors rather than user experience.
- Evident from the text: single LLM and small sample limit generalizability; no neutral path was tested; no ending-detection algorithm was implemented (participants clicked “Next”); the system went offline 5 times due to API calls, and repetition/errors may affect data quality.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 9 Not valid · 37 Not checked · 251 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Recent Large Language Models (LLMs; e.g. , GPT-3 [17], PaLM [23], OPT [118], HyperCLOVA [41]), with billions of parameters pre-trained on a large amount of language corpora, provide new opportunities for conversational agents.”
What the sources say
- Not checked yetcites [17] [17] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901. -
- Supportedcites [23] [23] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, H source
The citation is an identification one: PaLM is listed as an example of a recent LLM with billions of parameters pre-trained on a large language corpus. Our copy states exactly that in the Introduction (p. 3): a 540 billion parameter, densely activated autoregressive Transformer trained on 780 billion tokens of high-quality text, with state-of-the-art few-shot results. The related-work section (p. 47) also ties scale to conversational/dialogue capability ('dialogue applications have advanced through scale'), which covers the framing half of the citing sentence; the 'new opportunities for conversational agents' conclusion itself is the citing authors' inference rather than PaLM's claim. (LLM, conf=0.85)
- Supportedcites [118] source
OPT is exactly the kind of thing the citing sentence lists it as: the Abstract (p. 1) and Section 2.1 (p. 2) describe OPT as decoder-only transformers from 125M to 175B parameters, and Section 2.3 (p. 2) documents a pre-training corpus of roughly 180B tokens drawn from RoBERTa, the Pile and Reddit data. The conversational-agent part is treated in Section 3.2 Dialogue (p. 5), which states LLMs 'are known to be an integral component of modern dialogue models' and shows unsupervised OPT-175B competitive with the fully supervised BlenderBot 1 on ConvAI2 and other dialogue datasets. The paper does not use the phrase 'new opportunities' — that framing is the citing authors' — and it adds safety caveats (Section 4.5, p. 8), but nothing contradicts the attribution. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While the HCI and CSCW community have actively explored the use of LLMs in various domains ( e.g. , [24, 53, 114]), research that leverages LLMs for powering chatbots, particularly taskoriented ones [12, 72, 103], is still sparse.”
What the sources say
- Not checked yetcites [12] [12] Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang, Sang-Woo Lee, and Woomyoung Park. 2022. Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models. arXiv preprint arXiv:2205.00176 (2022). Jing Wei, Sungdong Kim, Hyunhoon Jung, and Young-Ho Kim -
- Not checked yetcites [72]
- Partly supportedcites [103] sourceExperimental — accuracy not yet measured
Our copy supports only the existence/description of LLM-powered chatbot work with task-oriented capability: the Abstract (p. 1) shows a Codex (GPT-3 finetuned on GitHub) agent that holds multi-turn conversations and answers questions about structured Minecraft data, and the Conclusion (p. 7/9) states 'Codex-powered NPCs can integrate both conversational and task-oriented language interactions almost seamlessly with code generation.' Nothing in our copy asserts that such research is 'still sparse'; the closest field-level statement, in Related Work (p. 3), runs the other way — that prompting for dialog generation 'is rapidly being mainstreamed' — and the related-work survey (pp. 2-3) lists several dialogue/prompting systems (DialoGPT, Meena, PLATO-XL, LaMDA, Madotto et al. 2021, Zheng and Huang 2021) rather than a scarcity of task-oriented chatbot work. So the sparsity characterisation attributed here is the citing authors' own judgement, not this paper's claim. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In terms of utilizing mainstream chatbot platforms to build voice applications, it is of great human effort to come up with user intents and example phrases as well as design conversation flows and logic [27, 68].”
What the sources say
- Could not verifycites [27] [27] Tilman Dingler, Dominika Kwasnicka, Jing Wei, Enying Gong, and Brian Oldenburg. 2021. The use and promise of conversational agents in digital health. Yearbook of Medical Informatics 30, 01 (2021), 191-199. - sourceExperimental — accuracy not yet measured
Section 2 describes tools such as Dialogflow and says they do not require specific programming knowledge. Section 3.3 says custom agents built with open-source frameworks require expertise and development resources, but neither passage addresses whether mainstream platforms require substantial human effort to devise intents, example phrases, and conversation flows. Searches for those terms and for manual design effort found no supporting passage. (LLM, conf=0.72)
- Not checked yetcites [68]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Despite drawbacks discussed above, LLMs-based chatbots can become a valuable and scalable tool for researchers to collect data for personal informatics [35].”
What the sources say
- Could not verifycites [35] sourceExperimental — accuracy not yet measured
Our copy of Homewood et al. (2020) is a fourth-wave HCI methods paper: it proposes 'removal as a method' built on autoethnography, interviews and participant reflection on menstrual-cycle tracking apps (INTRODUCTION p. 1; 'How to use the method' pp. 9-10), and discusses personal informatics data collection only via existing approaches such as photo elicitation, surveys, diaries and autoethnographies (p. 2). No passage mentions LLMs, chatbots, conversational agents or scalability of data collection (searched: chatbot/LLM/conversational agent, artificial intelligence, scalable data collection, valuable method for researchers). The closest statement even cautions against such a generalisation: 'this method would have to be applied to many different cases before we could make generalized statements about its value, limitations and implications as a fourth-wave method' (EXPANDING AND APPLYING REMOVAL AS A METHOD, p. 9). So the claim about LLM-based chatbots cannot be checked against this source. (LLM, conf=0.80)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, 1,482 out of 7,442 turns ( 𝑎𝑣𝑔. 18.45% per dialogue) were classified as Statement (See Table 5).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
87 more claims
Create a free account to see all 300 claims and to verify the 54 references still open.
- Probably not a substantial claim. 154
- It may state two separate claims as one. 91
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As pointed out by existing work in LLMs and natural language processing, there are restrictions to the number of tokens input to the LLMs [6], limiting the length of conversation history texts that could be included in LLM prompting [12,70,116].”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [116]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2 Slot Filling Rate · p. 12
“On the other hand, there were significant randomeffects of the topics( 𝑝 < . 0001), the conversation paths ( 𝑝 = . 01), and the interaction between the information formats and personality modifiers ( 𝑝 < . 001).”
§ 3.4 Data Analysis · p. 11
“Starting from a full model containing participants as a random effect and the four main study factors-information format, personality modifier, topic, and path-and their interactions as fixed effects”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In total, 6.7% of the chatbot turns (257 out of 3,916) were tagged erroneous”
Reported 6.7 · recomputed 6.562819
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4 Data Analysis · p. 11
“To assess the difference among the experimental conditions, we conducted Kruskal-Wallis tests over the four rating questions.”
§ 3.2 Online Study · p. 9
“The survey consisted of three 5-point Likert scale questions and one open-ended feedback textfield.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“in 31 dialogues (SP: 3, SN: 9, DP: 8, DN:10).”
Reported 31 · recomputed 30
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 10×
“LLMs work differently from traditional task-specific models as they accept prompts written in natural language as input to perform various tasks [14, 60].”
BackgroundMethodInfluentialMixed-Effects Models in Sand S-PLUS
Pinheiro, J. C., & Bates, D. M. (2000) · Statistics and Computing · doi:10.1007/978-1-4419-0318-1 · Cited by 11,187
emmeans: Estimated Marginal Means, aka Least-Squares Means
Lenth, R. V., & Piaskowski, J. (2017) · CRAN: Contributed Packages · doi:10.32614/cran.package.emmeans · Cited by 8,143
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Self-citationLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“, [55]), given that LLMs can be sensitive to prompt designs [60], we reckon that our proposed prompts may not yield similar performance on other LLMs due to the differences in the training corpora and the model architecture.”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“, [55]), given that LLMs can be sensitive to prompt designs [60], we reckon that our proposed prompts may not yield similar performance on other LLMs due to the differences in the training corpora and the model architecture.”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1×
“to help people develop long-term adoptions for health monitoring [23, 26, 72].”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2×
“, GPT-3 [17], PaLM [23], OPT [118], HyperCLOVA [41]), with billions of parameters pre-trained on a large amount of language corpora, provide new opportunities for conversational agents.”
BackgroundSelf-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 1×
“We kept the temperature and the presence penalty unchanged based on OpenAI’s suggestions and increased the frequency penalty to reduce the re-use of words.”
MethodLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 1×
“We describe our prompt designs and the experimental design of the user study.”
MethodFlow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi
Csikszentmihalyi, M. (2014) · doi:10.1007/978-94-017-9088-8 · Cited by 1,653 · cited 1×
“[10] combined a chatbot with the experience sampling method (ESM, [47]) and found that personalized chatbots have the potential to collect data on sensitive or personal topics.”
BackgroundWhy Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274 · cited 1×
“As suggested by earlier work, even NLP experts adopt trial and error and iterative experimentation when designing prompts before applying prompts in large datasets [117].”
BackgroundLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
A stage-based model of personal informatics systems
Li, I., Dey, A., & Forlizzi, J. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753409 · Cited by 1,100 · cited 1×
“Personal informatics systems have commonly incorporated data collection techniques to track personal health and activity [17, 53].”
BackgroundMixed-Effects Models in S and S-PLUS
(2000) · Statistics and Computing · doi:10.1007/b98882 · Cited by 930
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777 · cited 1×
“Many open-domain chatbots that engage and entertain people socially are predominantly dependent on large datasets [4, 119].”
BackgroundThe Design and Implementation of XiaoIce, an Empathetic Social Chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2018) · arXiv · doi:10.48550/arxiv.1812.08989 · Cited by 777 · cited 1×
“Many open-domain chatbots that engage and entertain people socially are predominantly dependent on large datasets [4, 119].”
BackgroundUnderstanding quantified-selfers' practices in collecting and exploring personal data
Choe, E. K., Lee, N. B., Lee, B., Pratt, W., & Kientz, J. A. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557372 · Cited by 766 · cited 1×
“However, repeated manual input on a computer or smartphone screen is burdensome and may gradually disengage people from tracking [17, 18].”
BackgroundRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635 · cited 4×
“Despite these potentials, it is yet fully understood how LLMs read the prompt and use pretrained knowledge [17, 64], the development of prompts is usually conducted through iterative trial and error [66].”
BackgroundInfluentialTowards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · cited 3×
“While the mechanism enabling such few-shot abilities behind LLMs is still veiled [17, 76], some prompting techniques are found to improve the model performance.”
BackgroundInfluentialRetrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
(2018) · doi:10.1145/3173574 · Cited by 569
User Experiences of Social Support From Companion Chatbots in Everyday Contexts: Thematic Analysis
Ta, V., Griffith, C., Boatfield, C., Wang, X., Civitello, M., Bader, H., DeCero, E., & Loggarakis, A. (2020) · Journal of Medical Internet Research · doi:10.2196/16235 · Cited by 561 · cited 1×
“Suggested by its name, CAs interact with people in natural languages, thereby providing customer services [22] and companionship [88].”
BackgroundWu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483
Understanding my data, myself: supporting self-reflection with ubicomp technologies
Li, I., Dey, A. K., & Forlizzi, J. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030166 · Cited by 459
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 2×
“We describe our prompt designs and the experimental design of the user study.”
MethodEvaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376 · cited 3×
“[34] revealed that many first-time chatbot users had disappointment and frustration with the selected chatbots: most chatbots lacked the ability to fully comprehend user messages or intentions.”
BackgroundInfluentialSuperAgent: A Customer Service Chatbot for E-commerce Websites
Cui, L., Huang, S., Wei, F., Tan, C., Duan, C., & Zhou, M. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4017 · Cited by 347 · cited 1×
“In this section, we cover related work in the areas of (1) self-report data collection through chatbots, (2) design considerations for chatbots, (3) chatbot platforms, and (4) designing LLM prompts for chatbots.”
BackgroundLee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345
Communication Breakdowns Between Families and Alexa
Beneteau, E., Richards, O. K., Zhang, M., Kientz, J. A., Yip, J., & Hiniker, A. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300473 · Cited by 313 · cited 1×
“Current conversational interfaces such as Google Assistant and Amazon Alexa shows certain abilities in maintaining contexts [66]; however, most of them are still criticized for not detecting contextual details [11, 16].”
BackgroundKorngiebel, D. M., & Mooney, S. D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00464-x · Cited by 259 · cited 3×
“Due to the inherent characteristics of LLMs, LLM-driven chatbots may be error-prone [44] or digress from their tasks [94].”
BackgroundRoss, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258 · cited 1×
“, programmers [86], socially-isolated people [39], and children [90]) as well as tasks (e.”
BackgroundHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Neural Text Generation with Unlikelihood Training
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2019) · arXiv · doi:10.48550/arxiv.1908.04319 · Cited by 254 · cited 1×
“In addition, comparing user perception of LLM-driven chatbots with other mainstream chatbot frameworks ( c.f. , [77]) would provide holistic design implications for self-reporting chatbots with balanced data collection performance and user perception.”
BackgroundBarriers and Negative Nudges: Exploring Challenges in Food Journaling
Cordeiro, F., Epstein, D. A., Thomaz, E., Bales, E., Jagannathan, A. K., Abowd, G. D., & Fogarty, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702155 · Cited by 249
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · cited 1×
“However, creating more dynamic conversations usually requires programming skills.”
BackgroundResilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns
Ashktorab, Z., Jain, M., Liao, Q. V., & Weisz, J. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300484 · Cited by 224
Patterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 2×
“Although existing chatbot frameworks also have error recovery features, their features are not only limited but often cannot allow quick repairs [66, 74, 98].”
BackgroundSocial Simulacra: Creating Populated Prototypes for Social Computing Systems
Park, J. S., Popowski, L., Cai, C., Morris, M. R., Liang, P., & Bernstein, M. S. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545616 · Cited by 212 · cited 1×
“LLMs are trained on an existing corpus that may contain biased and problematic information [27, 77].”
BackgroundToss 'n' turn: smartphone as sleep and sleep quality detector
Min, J.-K., Doryab, A., Wiese, J., Amini, S., Zimmerman, J., & Hong, J. I. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557220 · Cited by 201
Lab And/Or Field? Measuring Personality Processes and Their Social Consequences
Wrzus, C., & Mehl, M. R. (2015) · European Journal of Personality · doi:10.1002/per.1986 · Cited by 200 · cited 1×
“With more focus on self-reported data, prior studies leveraged chatbots to collect self-reports such as emotion ( e.g. , [13]), pain level ( e.g. , [113]), and food intake ( e.g. , [77]).”
BackgroundKim, S., Lee, J., & Gweon, G. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300316 · Cited by 190
Self-citationXiao, Z., Zhou, M. X., Liao, Q. V., Mark, G., Chi, C., Chen, W., & Yang, H. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3381804 · Cited by 185
At Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces
Braun, M., Mainz, A., Chadowitz, R., Pfleging, B., & Alt, F. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300270 · Cited by 182
Jo, E., Epstein, D. A., Jung, H., & Kim, Y.-H. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581503 · Cited by 174 · cited 1×
“, programmers [86], socially-isolated people [39], and children [90]) as well as tasks (e.”
BackgroundSelf-citationSemi-Automated Tracking: A Balanced Approach for Self-Monitoring Applications
Choe, E. K., Abdullah, S., Rabbi, M., Thomaz, E., Epstein, D. A., Cordeiro, F., Kay, M., Abowd, G. D., Choudhury, T., Fogarty, J., et al. (2017) · IEEE Pervasive Computing · doi:10.1109/mprv.2017.18 · Cited by 170 · cited 3×
“However, repeated manual input on a computer or smartphone screen is burdensome and may gradually disengage people from tracking [17, 18].”
BackgroundDialogue State Tracking with a Language Model using Schema-Driven Prompting
Lee, C., Cheng, H., & Mari Ostendorf, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.404 · Cited by 165 · cited 1×
“We kept the temperature and the presence penalty unchanged based on OpenAI’s suggestions and increased the frequency penalty to reduce the re-use of words.”
MethodEnabling Conversational Interaction with Mobile UI using Large Language Models
Wang, B., Li, G., & Li, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580895 · Cited by 156 · cited 2×
“We believe that well-designed prompts can effectively drive chatbots to perform specified tasks [104].”
BackgroundWhat Makes Good In-Context Examples for GPT-$3$?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2101.06804 · Cited by 148 · cited 1×
“Despite these potentials, it is yet fully understood how LLMs read the prompt and use pretrained knowledge [17, 64], the development of prompts is usually conducted through iterative trial and error [66].”
BackgroundTimeAware: Leveraging Framing Effects to Enhance Personal Productivity
Kim, Y.-H., Jeon, J. H., Choe, E. K., Lee, B., Kim, K., & Seo, J. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858428 · Cited by 139
Self-citationRhim, J., Kwak, M., Gong, Y., & Gweon, G. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107034 · Cited by 138 · cited 1×
“Further, incorporating more humanized traits, such as casual conversation styles [38], self-introduction, and echoing [80], led to not only a higher level of user engagement and satisfaction but also more self-disclosure in responses.”
BackgroundA Review on Implementation Issues of Rule-based Chatbot Systems
Thorat, S. A., & Jadhav, V. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3567047 · Cited by 119 · cited 6×
“Lastly, other commercial chatbot frameworks, such as Dialogflow [33] and IBM Watson [36], also allow developers to build rule-based chatbots with GUIs [98].”
BackgroundInfluentialOmniTrack: A Flexible Self-Tracking Approach Leveraging Semi-Automated Tracking
Kim, Y.-H., Jeon, J. H., Lee, B., Choe, E. K., & Seo, J. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3130930 · Cited by 118
Self-citationIf I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills
Xiao, Z., Zhou, M. X., Chen, W., Yang, H., & Chi, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376131 · Cited by 113 · cited 1×
“Studies have found that people are willing to engage with chatbots and provide valuable information, such as self-reports [96] and survey responses [15], to chatbots [105, 106].”
BackgroundEffects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
Shi, W., Wang, X., Oh, Y. J., Zhang, J., Sahay, S., & Yu, Z. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376843 · Cited by 93 · cited 1×
“Referring to some existing taxonomies of dialogue acts [85, 92, 100], three researchers independently coded one participant’s dialogues (132 turns; 1.”
BackgroundA model of user adoption of interface agents for email notification
Serenko, A. (2008) · Interacting with Computers · doi:10.1016/j.intcom.2008.04.004 · Cited by 92 · cited 2×
“Given the importance of social features such as chit-chat for positive user experience [57, 84], our results show that we can easily equip GPT-3 with such social aspects.”
BackgroundLiao, Q. V., Davis, M., Geyer, W., Muller, M., & Shami, N. S. (2016) · Proceedings of the 2016 ACM Conference on Designing Interactive Systems · doi:10.1145/2901790.2901842 · Cited by 92
Removal as a Method: A Fourth Wave HCI Approach to Understanding the Experience of Self-Tracking
Homewood, S., Karlsson, A., & Vallgårda, A. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395425 · Cited by 87
Simply asking questions about health behaviors increases both healthy and unhealthy behaviors
Williams, P., Block, L. G., & Fitzsimons, G. J. (2006) · Social Influence · doi:10.1080/15534510600630850 · Cited by 82 · cited 1×
“Towards this goal, we first chose four health-related topics—Sleep [70], Food intake [25, 63],Work and productivity [39, 41, 54], and Exercise [64]—that retain interests not only by the research community and are also personal use in daily life [102].”
Kim, Y.-H., Choe, E. K., Lee, B., & Seo, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300845 · Cited by 81
Self-citationThe Use and Promise of Conversational Agents in Digital Health
Dingler, T., Kwasnicka, D., Wei, J., Gong, E., & Oldenburg, B. (2021) · Yearbook of Medical Informatics · doi:10.1055/s-0041-1726510 · Cited by 81 · cited 4×
“This feature can be especially useful for people without programming experiences, such as medical practitioners [23].”
BackgroundInfluentialSelf-citation20 Questions on Dialogue Act Taxonomies
TRAUM, D. R. (2000) · Journal of Semantics · doi:10.1093/jos/17.1.7 · Cited by 78 · cited 1×
“Referring to some existing taxonomies of dialogue acts [85, 92, 100], three researchers independently coded one participant’s dialogues (132 turns; 1.”
BackgroundKim, B., Kim, H., Lee, S.-W., Lee, G., Kwak, D., Dong Hyeon, J., Park, S., Kim, S., Kim, S., Seo, D., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.274 · Cited by 72 · cited 2×
“In this section, we cover related work in the areas of (1) self-report data collection through chatbots, (2) design considerations for chatbots, (3) chatbot platforms, and (4) designing LLM prompts for chatbots.”
BackgroundMethodSelf-citationEpstein, D. A., Avrahami, D., & Biehl, J. T. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858066 · Cited by 65
Völkel, S. T., Schödel, R., Buschek, D., Stachl, C., Winterhalter, V., Bühner, M., & Hussmann, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376210 · Cited by 65 · cited 1×
“In other words, future chatbot platforms should allow developers to easily build personalized chatbots with different personalities [93] and conversation styles [12, 45].”
BackgroundCelino, I., & Re Calegari, G. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102410 · Cited by 63 · cited 10×
“Lastly, other commercial chatbot frameworks, such as Dialogflow [33] and IBM Watson [36], also allow developers to build rule-based chatbots with GUIs [98].”
BackgroundMethodResultInfluentialA Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot
Li, C.-H., Yeh, S.-F., Chang, T.-J., Tsai, M.-H., Chen, K., & Chang, Y.-J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376209 · Cited by 57 · cited 2×
“For instance, a banking chatbot can provide services such as currencyexchange conversion and introduction of credit cards [52].”
BackgroundZero-Shot Dialogue State Tracking via Cross-Task Transfer
Lin, Z., Liu, B., Madotto, A., Moon, S., Zhou, Z., Crook, P., Wang, Z., Zhou, Y., Cho, E., Subba, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.622 · Cited by 57 · cited 1×
“Such approaches will require incorporating dialog state tracking techniques ( e.g. , [63]) for automated slot filling.”
MethodSvenningsson, N., & Faraon, M. (2019) · Proceedings of the 2019 2nd Artificial Intelligence and Cloud Computing Conference · doi:10.1145/3375959.3375973 · Cited by 56 · cited 1×
“Follow-up questions are commonly used in human-administered interviews to increase interactivity [81] and many studies suggest that chatbots that can ask on-topic follow-up questions are considered more human-like [62, 87].”
BackgroundVölkel, S. T., Schödel, R., Buschek, D., Stachl, C., Winterhalter, V., Bühner, M., & Hußmann, H. (2020) · Alexandria (UniSG) (University of St.Gallen) · Cited by 55
The Role of Conversational Grounding in Supporting Symbiosis Between People and Digital Assistants
Cho, J., & Rader, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392838 · Cited by 49 · cited 2×
“In particular, the error handling is tricky to design as conversation breakdowns can be common and even unexpected in the wild [16, 74, 98].”
BackgroundWei, J., Tag, B., Trippas, J. R., Dingler, T., & Kostakos, V. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517432 · Cited by 46 · cited 2×
“Although existing chatbot frameworks also have error recovery features, their features are not only limited but often cannot allow quick repairs [66, 74, 98].”
BackgroundSelf-citationLuo, Y., Lee, B., & Choe, E. K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376616 · Cited by 44 · cited 11×
“As such, we picked different job identities for each topic— sleep expert (Sleep), dietitian (Food), life coach (Work), and fitness coach (Exercise).”
BackgroundInfluentialMyMove: Facilitating Older Adults to Collect In-Situ Activity Labels on a Smartwatch with Speech
Kim, Y.-H., Chou, D., Lee, B., Danilovich, M., Lazar, A., Conroy, D. E., Kacorri, H., & Choe, E. K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517457 · Cited by 39 · cited 1×
“Prior research found that when people are allowed to insert data in free-form natural language, they tend to provide detailed answers with surrounding contexts [40, 63].”
BackgroundSelf-citationLet's Talk It Out: A Chatbot for Effective Study Habit Behavioral Change
Tian, X., Risha, Z., Ahmed, I., Lekshmi Narayanan, A. B., & Biehl, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449171 · Cited by 37
The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues
Tack, A., & Piech, C. (2022) · arXiv · doi:10.48550/arxiv.2205.07540 · Cited by 36 · cited 1×
“We describe our prompt designs and the experimental design of the user study.”
MethodInvestigating Preferred Food Description Practices in Digital Food Journaling
Silva, L. M., & Epstein, D. A. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462145 · Cited by 35
Understanding User Perceptions of Proactive Smart Speakers
Wei, J., Dingler, T., & Kostakos, V. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3494965 · Cited by 33 · cited 3×
“Lastly, we hope future research can investigate the user perceptions of LLM-driven chatbots, or even voice-based ones like smart speakers [96].”
BackgroundSelf-citationChatClimate: Grounding conversational AI in climate sciencePreprint
Vaghefi, S. A., Stammbach, D., Muccione, V., Bingler, J., Ni, J., Kraus, M., Allen, S., Senni, C. C., Werkhof, T., Schimanski, T., et al. (2023) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.3929/ethz-b-000649578 · Cited by 32
Volum, R., Rao, S., Xu, M., DesGarennes, G., Brockett, C., Van Durme, B., Deng, O., Malhotra, A., & Dolan, B. (2022) · Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022) · doi:10.18653/v1/2022.wordplay-1.3 · Cited by 30 · cited 5×
“Compared to other frameworks, LLMs show great potential in scaffolding chatbots that are sensible of contexts and even respond to off-topic user messages [94].”
BackgroundInfluentialAn Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
Wang, L., Mujib, M. I., Williams, J., Demiris, G., & Huh-Yoo, J. (2021) · arXiv · doi:10.48550/arxiv.2107.13115 · Cited by 21 · cited 1×
“We describe our prompt designs and the experimental design of the user study.”
MethodBuilding a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language ModelsPreprint
Bae, S., Kwak, D., Kim, S., Ham, D., Kang, S., Lee, S., & Park, W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.00176 · Cited by 20 · cited 3×
“, [20, 49, 104]), research that leverages LLMs for powering chatbots, particularly task-oriented ones [8, 67, 94], is still sparse.”
BackgroundSelf-citationEffects of Language Variety on Personality Perception in Embodied Conversational Agents
Krenn, B., Endrass, B., Kistler, F., & André, E. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-07230-2_41 · Cited by 13 · cited 1×
“Nevertheless, it is unclear how to design prompts for LLMs to steer chatbots that can effectively ask questions around desired information and have different conversational styles.”
BackgroundUnderstanding How to Administer Voice Surveys through Smart Speakers
Wei, J., Jiang, W., Wang, C., Yu, D., Goncalves, J., Dingler, T., & Kostakos, V. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555606 · Cited by 9 · cited 1×
“As such, compared to rule-based chatbots that can almost 100% ask pre-defined scripts [97], we can see that not all specified information slots were asked by our chatbots during the study.”
BackgroundSelf-citationOlder Adults' Satisfaction and Compliance of Smartwatches Providing Ecological Momentary
Laborde, C., Cenko, E., Mardini, M., Ranka, S., Rashidi, P., & Manini, T. M. (2020) · Innovation in Aging · doi:10.1093/geroni/igaa057.2898 · Cited by 3 · cited 1×
“While various physiological or physical activity data—such as step count, heart rate, and sleep duration—can be captured automatically by sensors and wearable devices [46], various types of personal data still demand self-reporting by the person who self-tracks [17].”
BackgroundAtkinson, D. (2022) · Manufacturing Management · doi:10.12968/s2514-9768(23)90304-2 · Cited by 1
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068 · cited 1×
“In this section, we cover related work in the areas of (1) self-report data collection through chatbots, (2) design considerations for chatbots, (3) chatbot platforms, and (4) designing LLM prompts for chatbots.”
BackgroundRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2009.11462 · cited 1×
“In this study, we explored how we can use GPT-3 for powering chatbots that can reliably ask people health-related questions through natural conversations.”
Research works printed but not linked: 23
- Apple Inc. 2022. SIRI shortcuts boost health and fitness routines - Apple News Room. Retrieved Dec 04, 2022 from https://www.apple.com/newsroom/2019/03/siri-shortcuts-boost-health-and-fitness-routines/
- Florian Bemmann, Ramona Schödel, Niels Van Berkel, and Daniel Buschek. 2021. Chatbots for Experience Sampling-Initial Opportunities and Challenges.. In IUI Workshops.
- Susan E Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. The art of human-computer interface design (1990), 393--404.
- John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. 1--19.
- Google. 2022. Google Assistant, Your Own Personal Google. Retrieved Dec 04, 2022 from https://assistant.google.com
- IBM. 2022. IBM Watson. Retrieved Dec 04, 2022 from https://www.ibm.com/watson
- Jae Ho Jeon. 2016. OmniTrack: Designing Flexible and Highly Customizable Quantified-Self Tool. MS thesis. Seoul National University Seoul Korea. http://www.riss.kr/link?id=T14226449
- Juju, inc. 2022. Cognitive AI Chatbot. Retrieved Dec 04, 2022 from https://juji.io/
- Min Kyung Lee, Sara Kiesler, Jodi Forlizzi, Siddhartha Srinivasa, and Paul Rybski. 2010. Gracefully mitigating breakdowns in robotic services. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 203--210.
- Yuhan Luo, Young-Ho Kim, Bongshin Lee, Naeemul Hassan, and Eun Kyoung Choe. 2021. FoodScrap: Promoting Rich Data Capture and Reflective Food Journaling Through Speech Input. In Designing Interactive Systems Conference 2021. 606--618.
- Michael McTear. 2018. Conversational modelling for chatbots: current approaches and future directions. Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung (2018), 175--185.
- Shikib Mehri, Yasemin Altun, and Maxine Eskenazi. 2022. LAD: Language Models as Data for Zero-Shot Dialog. arXiv preprint arXiv:2207.14393 (2022). · arXiv
- Meta Platforms, Inc. 2022. React -- A JavaScript library for building user interfaces. Retrieved Dec 04, 2022 from https://reactjs.org/
- Microsoft. 2022. TypeScript. Retrieved Dec 04, 2022 from https://www.typescriptlang.org
- Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv preprint arXiv:2202.12837 (2022). · arXiv
- Elliot Mitchell, Noemie Elhadad, and Lena Mamykina. 2022. Examining AI Methods for Micro-Coaching Dialogs. In CHI Conference on Human Factors in Computing Systems. 1--24.
- Ryan Morrison. 2022. GPT-3 developer OpenAI releases new Davinci generative text model. Retrieved Dec 04, 2022 from https://techmonitor.ai/technology/ai-and-automation/gpt-3-openai-davinci-generative-text
- OpenAI. 2022. OpenAI API. Retrieved Dec 04, 2022 from https://openai.com/api/
- Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207 (2018). · arXiv
- Kathryn Roulston and Myungweon Choi. 2018. Qualitative interviews. The SAGE handbook of qualitative data collection (2018), 233--249.
- Woosuk Seo Chanmo Yang and Young-Ho Kim. 2023. ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events. arXiv:2309.12244 [cs.HC] · arXiv
- The OpenJS Foundation. 2022. Node.js. Retrieved Dec 04, 2022 from https://nodejs.org
- Anuradha Welivita and Pearl Pu. 2020. A taxonomy of empathetic response intents in human social conversations. arXiv preprint arXiv:2012.04080 (2020). · arXiv
Cited by
A Systematic Review of Generative AI for Teaching and Learning Practice
Ogunleye, B., Zakariyyah, K. I., Ajao, O., Olayinka, O., & Sharma, H. (2024) · Education Sciences · doi:10.3390/educsci14060636 · Cited by 251
Zhong, W., Luo, J., & Zhang, H. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.04.057 · Cited by 173
MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling
Kim, T., Bae, S., Kim, H. A., Lee, S.-W., Hong, H., Yang, C., & Kim, Y.-H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642937 · Cited by 112
“Recent LLMs have suggested a new paradigm of bootstrapping chatbots [93, 94].”
BackgroundInfluentialSelf-citationYang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120
Pandey, S., & Sharma, S. (2023) · Healthcare Analytics · doi:10.1016/j.health.2023.100198 · Cited by 77
Metadata
- Authors
- Wei, J., Kim, S., Jung, H., & Kim, Y.-H.
- Affiliations
- The University of Melbourne (AU)
- NAVER Cloud (South Korea) (KR)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2024-04-17
- Volume
- 8
- Issue
- CSCW1
- Pages
- 1-35
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4396219010
- Cited by
- 84
- References
- 74
- Citations by year
- 2026: 17 · 2025: 37 · 2024: 25
- FWCI
- 11.70
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- large language models, chatbots, prompt engineering, naturalistic conversation, personas, conversation flow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.