Do We Still Need Human Assessors? Prompt-Based GPT-3 User Simulation in Conversational AI
Open accessThe DOI matches its Crossref record; 16 of the 18 references it cites were found in a registry or an index, and 2 are not checked yet.
Abstract
Scarcity of user data continues to be a problem in research on conversational user interfaces and often hinders or slows down technical innovation. In the past, different ways of synthetically generating data, such as data augmentation techniques have been explored. With the rise of ever improving pre-trained language models, we ask if we can go beyond such methods by simply providing appropriate prompts to these general purpose models to generate data. We explore the feasibility and cost-benefit trade-offs of using non fine-tuned synthetic data to train classification algorithms for conversational agents. We compare this synthetically generated data with real user data and evaluate the performance of classifiers trained on different combinations of synthetic and real data. We come to the conclusion that, although classifiers trained on such synthetic data perform much better than random baselines, they do not compare to the performance of classifiers trained on even very small amounts of real user data, largely because such data is lacking much of the variability found in user generated data. Nevertheless, we show that in situations where very little data and resources are available, classifiers trained on such synthetically generated data might be preferable to the collection and annotation of naturalistic data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Revision is essential to the writing process [29,42,62,101].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [62]
Experimental — accuracy not yet measuredIn the citing paper
“granular [62,81]: happens at the word, sentence, or paragraph-level”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [62]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“GPT-3 has been praised for its few-shot and zero-shot learning capabilities [5].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 1×
“In recent years, Natural Language Generation (NLG) approaches have become ever more powerful and are by now capable of outputting very human sounding and seemingly coherent text [4].”
BackgroundPrompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1×
“Reynolds and McDonell explored prompt programming for zero-shot learning, showing how important carefully crafted prompts are to achieve relevance of the output and outlining various tweaks that can be applied to prompts to improve results [23].”
BackgroundParaphrasing with bilingual parallel corpora
Bannard, C., & Callison-Burch, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219914 · Cited by 533 · cited 1×
“Furthermore, we intend to explore other data augmentation techniques [14] such as back-translation [2] or available paraphrasing technologies 3 , which were out of the scope of this specific project.”
BackgroundAll That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · cited 1×
“Other research found that even after being trained and seeing examples, human evaluators were not able to distinguish GPT-3 created stories, news articles and recipes from those written by humans [7].”
BackgroundBudzianowski, P., & Vulić, I. (2019) · Proceedings of the 3rd Workshop on Neural Generation and Translation · doi:10.18653/v1/d19-5602 · Cited by 279 · cited 1×
“There are a number of approaches that have been used to mitigate the costs of data collection [15] and data scarcity in the past, mainly focused on distant supervision, which uses weakly labeled examples gathered using noisy techniques, and data augmentation, which exploits different techniques to create new data from…”
BackgroundPretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2022) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-02181-7 · Cited by 156
A Short Survey of Pre-trained Language Models for Conversational AI-A New Age in NLP
Zaib, M., Sheng, Q. Z., & Emma Zhang, W. (2020) · Proceedings of the Australasian Computer Science Week Multiconference · doi:10.1145/3373017.3373028 · Cited by 69 · cited 1×
“A major problem in chatbot research, past and present, has been a scarcity of user data, especially for domain-specific Conversational Agents (CAs) focused on specific strategies or outcomes and for CAs that are constructed for non-English speakers [3, 9, 28].”
BackgroundŞahin, G. G. (2021) · Computational Linguistics · doi:10.1162/coli_a_00425 · Cited by 65 · cited 1×
“There are a number of approaches that have been used to mitigate the costs of data collection [15] and data scarcity in the past, mainly focused on distant supervision, which uses weakly labeled examples gathered using noisy techniques, and data augmentation, which exploits different techniques to create new data from…”
BackgroundEvaluating Mixed-initiative Conversational Search Systems via User Simulation
Sekulić, I., Aliannejadi, M., & Crestani, F. (2022) · Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3488560.3498440 · Cited by 55 · cited 2×
“In the context of responses to clarifying questions, texts written by humans were perceived as more natural but not more useful than GPT-generated texts [25].”
BackgroundMethod
Research works printed but not linked: 11
- Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics 34, 4 (2008), 555–596.
- Ms Aayushi Bansal Dr Rewa Sharma and Dr Mamta Kathuria. 2020. A Systematic Review on Data Scarcity Problem in Deep Learning: Solution and Applications. ACM Computing Surveys (CSUR)(2020).
- 10.18653/v1 · doi:10.18653/v1
- Alexander Frummet, David Elsweiler, and Bernd Ludwig. 2022. “What Can I Cook with these Ingredients?”-Understanding Cooking-Related Information Needs in Conversational Search. ACM Transactions on Information Systems (TOIS) 40, 4 (2022), 1–32.
- ARDB Landim, AM Pereira, Thales Vieira, E de B. Costa, JAB Moura, V Wanick, and Eirini Bazaki. 2021. Chatbot design approaches for fashion E-commerce: an interdisciplinary review. International Journal of Fashion Design, Technology and Education (2021), 1–11.
- Antonis Maronikolakis Mark Stevenson and Hinrich Schütze. 2020. Transformers Are Better Than Humans at Identifying Generated Text. ArXiv abs/2009.13375(2020). · arXiv
- Selina Meyer. 2022. “I’m at my wits’ end”-Anticipating Information Needs and Appropriate Support Strategies in Behaviour Change. In ACM SIGIR Conference on Human Information Interaction and Retrieval. 396–399.
- Selina Meyer and David Elsweiler. 2022. GLoHBCD: A Naturalistic German Dataset for Language of Health Behaviour Change on Online Support Forums. In Proceedings of the Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 2226–2235. https://aclanthology.org/2022.lrec-1.239
- William R Miller and Stephen Rollnick. 2002. Motivational interviewing: Preparing people for change. Book Review.(2002).
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Kang Min Yoo Dongju Park Jaewook Kang Sang-Woo Lee and Woomyeong Park. 2021. Gpt3mix: Leveraging large-scale language models for text augmentation. arXiv preprint arXiv:2104.08826(2021). · arXiv
Cited by
Alawida, M., Mejri, S., Mehmood, A., Chikhaoui, B., & Isaac Abiodun, O. (2023) · Information · doi:10.3390/info14080462 · Cited by 210
“Another study that may be relevant is a comparative study of GPT-3 and BERT language models [20,21].”
BackgroundReza, M., Laundry, N. M., Musabirov, I., Dushniku, P., Yu, Z. Y. “., Mittal, K., Grossman, T., Liut, M., Kuzminykh, A., & Williams, J. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641899 · Cited by 52
“Recent HCI studies on LLM-based tool design have mainly focused on prompt engineering [6, 16, 46, 55, 94, 98] and exploring the generative capabilities of LLMs [50, 58, 63, 67, 97].”
User Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000098 · Cited by 52
Do You Mind? User Perceptions of Machine Consciousness
Scott, A. E., Neumann, D., Niess, J., & Woźniak, P. W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581296 · Cited by 46
Hellwig, N. C., Fehle, J., & Wolff, C. G. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125514 · Cited by 67
Metadata
- Authors
- Meyer, S., Elsweiler, D., Ludwig, B., Fernandez-Pichel, M., & Losada, D. E.
- Affiliations
- University of Regensburg (DE)
- Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Spain
- Universidade de Santiago de Compostela (ES)
- Venue
- Proceedings of the 4th Conference on Conversational User Interfaces
- Publisher
- ACM
- Published
- 2022-07-26
- Pages
- 1-6
- OpenAlex
- W4295955200
- Cited by
- 34
- References
- 15
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 14 · 2023: 12
- FWCI
- 4.56
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Conversational User Interfaces (CUI) (publishedVersion, cc-by)
- University of Regensburg Publication Server (University of Regensburg) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- synthetic data generation, GPT-3, chatbots, user simulation, conversational interfaces, data scarcity, classifier training, cost-benefit trade-off
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Do We Still Need Human Assessors? Prompt-Based GPT-3 User Simulation in Conversational AI. https://app.cochat.ai/papers/10.1145/3543829.3544529
- The paper
- Meyer, S., Elsweiler, D., Ludwig, B., Fernandez-Pichel, M., & Losada, D. E. (2022). Do We Still Need Human Assessors? Prompt-Based GPT-3 User Simulation in Conversational AI. Proceedings of the 4th Conference on Conversational User Interfaces. https://doi.org/10.1145/3543829.3544529
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.