The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Sequence to Sequence Modeling for User Simulation in Dialog Systems
Abstract
User simulators are a principal offline method for training and evaluating human-computer dialog systems. In this paper, we examine simple sequence-to-sequence neural network architectures for training end-to-end, natural language to natural language, user simulators, using only raw logs of previous interactions without any additional human labelling. We compare the neural network-based simulators with a language model (LM)-based approach for creating natural language user simulators. Using both an automatic evaluation using LM perplexity and a human evaluation, we demonstrate that the sequence-to-sequence approaches outperform the LM-based method. We show correlation between LM perplexity and the human evaluation on this task, and discuss the benefits of different neural network architecture variations. — Example sessions that were generated when running the seq2seq user simulator models with Cortana.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
End-to-end models can be trained to contain internal states keeping track of the user model, learning these states from large amounts of dialogue examples.
“End-to-end models can be trained to contain internal states keeping track of the user model, learning these states simply from large amounts of dialogue examples (Crook and Marin, 2017; Boyd et al., 2020; Li et al., 2020).”
From Modeling Feedback in Interaction With Conversational Agents—A Review (2022) · cites this paper as Crook and Marin, 2017
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
(2004) · Cited by 9,718
SRILM - an extensible language modeling toolkit
Stolcke, A. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-303 · Cited by 4,385
(2005) · Cited by 4,152
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9883 · Cited by 1,643
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,367
Agenda-based user simulation for bootstrapping a POMDP dialogue system
Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., & Young, S. J. (2007) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1614108.1614146 · Cited by 337
User simulation for spoken dialogue systems: learning and evaluation
Georgila, K., Henderson, J., & Lemon, O. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-160 · Cited by 126
Gaussian Processes for POMDP-Based Dialogue Manager Optimization
Gašić, M., & Young, S. J. (2013) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2013.2282190 · Cited by 123
The Hidden Agenda User Simulation Model
Schatzmann, J., & Young, S. J. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2012071 · Cited by 94
Assessment of dialogue systems by means of a new simulation technique
López-Cózar, R., de la Torre, Á., Segura, J. C., & Rubio, A. J. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00126-7 · Cited by 74
Stochastic natural language generation for spoken dialog systems
Oh, A. H., & Rudnicky, A. I. (2002) · Computer Speech & Language · doi:10.1016/s0885-2308(02)00012-8 · Cited by 73
Sarikaya, R., Crook, P. A., Marin, A., Jeong, M., Robichaud, J. P., Çelikyılmaz, A., Kim, Y. B., Rochette, A., Khan, O., Liu, X., et al. (2016) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2016.7846294 · Cited by 72
Self-citationParameter estimation for agenda-based user simulation
Keizer, S., Gašić, M., Jurčíček, F., Mairesse, F., Thomson, B., Yu, K., & Young, S. J. (2010) · Cited by 38
User simulation for the evaluation of bus information systems
Götze, J., Scheffler, T., Roller, R., & Reithinger, N. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700895 · Cited by 6
SpeechEval: A Domain-Independent User Simulation Platform for Spoken Dialog System Evaluation
Scheffler, T., Roller, R., & Reithinger, N. (2011) · Springer eBooks · doi:10.1007/978-1-4614-1335-6_29 · Cited by 4
Flagged first, then by citations
Cited by
Sorted by citations
Recent advances and challenges in task-oriented dialog systems
Zhang, Z., Takanobu, R., Zhu, Q., Huang, M., & Zhu, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1692-3 · Cited by 216
Neural User Simulation for Corpus-based Policy Optimisation of Spoken Dialogue Systems
Kreyssig, F., Casanueva, I., Budzianowski, P., & Gašić, M. (2018) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w18-5007 · Cited by 83
User Modeling for Task Oriented Dialogues
Gur, I., Hakkani-Tur, D., Tur, G., & Shah, P. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639652 · Cited by 48
Modeling Feedback in Interaction With Conversational Agents—A Review
Axelsson, A., Buschmeier, H., & Skantze, G. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.744574 · Cited by 45
“End-to-end models can be trained to contain internal states keeping track of the user model, learning these states simply from large amounts of dialogue examples (Crook and Marin, 2017; Boyd et al., 2020; Li et al., 2020).”
BackgroundUser Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000098 · Cited by 52
Learning Robust Dialog Policies in Noisy EnvironmentsPreprint
Fazel-Zarandi, M., Li, S.-W. D., Cao, J. L., Casale, J., Henderson, P., Whitney, D., & Geramifard, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.04034 · Cited by 20
PictoBERT: Transformers for next pictogram prediction
Pereira, J., Macêdo, D., Zanchettin, C., Oliveira, A. L. I., & Fidalgo, R. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117231 · Cited by 19
User Modeling for Task Oriented DialoguesPreprint
Gür, İ., Hakkani-Tür, D. Z., Tür, G., & Shah, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04369 · Cited by 14
A Unified Dialogue User Simulator for Few-shot Data Augmentation
Wan, D., Zhang, Z., Zhu, Q., Liao, L., & Huang, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.277 · Cited by 14
Deep Learning in Spoken and Text-Based Dialog Systems
Çelikyılmaz, A., Deng, L., & Hakkani-Tür, D. Z. (2018) · Springer eBooks · doi:10.1007/978-981-10-5209-5_3 · Cited by 13
Deep Learning for Dialogue Systems
Chen, Y.-N., Çelikyılmaz, A., & Hakkani-Tür, D. Z. (2018) · International Conference on Computational Linguistics · Cited by 8
LAD: Language Models as Data for Zero-Shot Dialog
Mehri, S., Altün, Y., & Eskénazi, M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.55 · Cited by 11
Long Short-Term Memory Neural Networks for Artificial Dialogue Generation
Selouani, S., & Mohamed, S. Y. (2018) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2018.00113 · Cited by 6
Advising Agent for Service-Providing Live-Chat Operators
Aviv, A., Oshrat, Y., Assefa, S., Mustapha, T., Borrajo, D., Veloso, M., & Kraus, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20614-6_3 · Cited by 4
Towards a Formal Characterization of User Simulation Objectives in Conversational Information Access
Bernard, N., & Balog, K. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672529 · Cited by 5
Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented DialoguePreprint
Chen, M., Crook, P. A., & Roller, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06905 · Cited by 9
Self-citationA Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application
Hanafi, M. F., Katsis, Y., Cooper, M. S., & Li, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21541 · Cited by 4
Predictive authoring for Brazilian Portuguese augmentative and alternative communication
Pereira, J., Nogueira, R. F., Zanchettin, C., & Fidalgo, R. (2024) · Natural language processing. · doi:10.1017/nlp.2024.27 · Cited by 6
Automatic Task Completion Flows from Web APIs
Williams, K., Hashemi, S. H., & Zitouni, I. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331318 · Cited by 1
Towards a Universal NLG for Dialogue Systems and Simulators with Future BridgingPreprint
Ennen, P., Lin, Y.-T. R., Özbay, A. G., Insalata, F., Li, M., Tian, Y., Jalali, S., & Shiu, D.-S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.10267 · Cited by 2
Graph Neural Net-Based User Simulator
Nie, X., Lin, Z., Huang, X., & Zhang, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32381-3_51
Two Approaches to Building Collaborative, Task-Oriented Dialog Agents through Self-PlayPreprint
Arkhangorodsky, A., Fang, S., Knight, V., Nagesh, A., Ryskina, M., & Knight, K. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09597
BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems
Wang, G., Tan, S., Joty, S., Wu, G., Au, J., & Hoi, S. C. H. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.18
Few-shot task-oriented dialogue generation with in-context learning
Bi, Z., Hu, X., Zhang, W., & Wang, X. (2025) · International Journal of Web Information Systems · doi:10.1108/ijwis-01-2025-0007
Modeling User-Machine Interactions During The Information Retrieval Process
Mustar, A. (2023) · theses.fr (ABES) · doi:10.70675/bbdff803z7a52z450dz8f02z791c70371485
Metadata
- Authors
- Crook, P., & Marin, A.
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 1706-1710
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2745680449
- Cited by
- 25
- References
- 20
- Citations by year
- 2025: 1 · 2024: 3 · 2023: 1 · 2022: 7 · 2021: 3 · 2020: 1 · 2019: 2 · 2018: 6 · 2017: 1
- FWCI
- 1.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- user simulation, dialogue systems, sequence-to-sequence models, human evaluation, neural network architecture, Cortana, interaction logs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
