CoChat

Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions

Open access
Xiao, Ziang, Zhou, Michelle X., Liao, Q. Vera, Mark, Gloria, Chi, Changyan, Chen, Wenxi, Yang, Huahai · 2020 · ACM Transactions on Computer-Human Interaction
Cited by 185

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

Abstract

The rise of increasingly more powerful chatbots offers a new way to collect information through conversational surveys, where a chatbot asks open-ended questions, interprets a user’s free-text responses, and probes answers whenever needed. To investigate the effectiveness and limitations of such a chatbot in conducting surveys, we conducted a field study involving about 600 participants. In this study with mostly open-ended questions, half of the participants took a typical online survey on Qualtrics and the other half interacted with an AI-powered chatbot to complete a conversational survey. Our detailed analysis of over 5,200 free-text responses revealed that the chatbot drove a significantly higher level of participant engagement and elicited significantly better quality responses measured by Gricean Maxims in terms of their informativeness, relevance, specificity, and clarity. Based on our results, we discuss design implications for creating AI-powered chatbots to conduct effective surveys and beyond.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3381804

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 73

  1. David A. Aaker . 1976 . Design and analysis: A researcher’s handbook . Journal of Marketing Research 13 , 000003 (1976), 318 . David A. Aaker. 1976. Design and analysis: A researcher’s handbook. Journal of Marketing Research 13, 000003 (1976), 318.
  2. James F. Allen , Donna K. Byron , Myroslava Dzikovska , George Ferguson , Lucian Galescu , and Amanda Stent . 2001 . Toward conversational human-computer interaction . AI Magazine 22 , 4 (2001), 27 . James F. Allen, Donna K. Byron, Myroslava Dzikovska, George Ferguson, Lucian Galescu, and Amanda Stent. 2001. Toward conversational human-computer interaction. AI Magazine 22, 4 (2001), 27.
  3. Irwin Altman and Dalmas A . Taylor . 1973 . Social Penetration : The Development of Interpersonal Relationships. Holt, Rinehart 8 Winston . Irwin Altman and Dalmas A. Taylor. 1973. Social Penetration: The Development of Interpersonal Relationships. Holt, Rinehart 8 Winston.
  4. 10.1093/poq/nfw088 · DOI
  5. C. Bauer , K. Figl , and R. Motschnig-Pitrik . 2010 . Introducing ‘active listening’ to instant messaging and e-mail: Benefits and limitations . IADIS International Journal on WWW/Internet 7 , 2 (2010), 1 -- 17 . C. Bauer, K. Figl, and R. Motschnig-Pitrik. 2010. Introducing ‘active listening’ to instant messaging and e-mail: Benefits and limitations. IADIS International Journal on WWW/Internet 7, 2 (2010), 1--17.
  6. 10.1177/0894439311435305 · DOI
  7. Pazit Ben-Nun . 2008 . Respondent fatigue . Encyclopedia of Survey Research Methods 2 (2008), 742 -- 743 . Pazit Ben-Nun. 2008. Respondent fatigue. Encyclopedia of Survey Research Methods 2 (2008), 742--743.
  8. 10.1007/978-3-642-23974-8_7 · DOI
  9. Dan Bohus and Alexander I . Rudnicky . 2009 . The ravenclaw dialog management framework: Architecture and systems. Computer Speech 8 Language 23, 3 (2009), 332--361. Dan Bohus and Alexander I. Rudnicky. 2009. The ravenclaw dialog management framework: Architecture and systems. Computer Speech 8 Language 23, 3 (2009), 332--361.
  10. 10.1007/BF02763643 · DOI
  11. 10.1145/355112.355123 · DOI · Paper page
  12. Frederick G. Conrad , Mick P. Couper , Roger Tourangeau , Mirta Galesic , and T. Yan . 2005. Interactive feedback can improve the quality of responses in web surveys . In Proceedings of the Section on Survey Research Methods. 3835--3840 . Frederick G. Conrad, Mick P. Couper, Roger Tourangeau, Mirta Galesic, and T. Yan. 2005. Interactive feedback can improve the quality of responses in web surveys. In Proceedings of the Section on Survey Research Methods. 3835--3840.
  13. 10.18653/v1/P17-4017 · DOI · Paper page
  14. 10.1177/0894439315573744 · DOI
  15. David DeVault , Ron Artstein , Grace Benn , Teresa Dey , Ed Fast , Alesia Gainer , Kallirroi Georgila , Jon Gratch , Arno Hartholt , Margaux Lhommet , 2014 . SimSensei kiosk: A virtual human interviewer for healthcare decision support . In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 1061--1068 . David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014. SimSensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 1061--1068.
  16. 10.3115/992628.992686 · DOI
  17. 10.1016/j.specom.2004.02.001 · DOI
  18. S. Egelman E. Chi and S. Dow. 2014. Crowdsourcing in HCI research. In Ways of Knowing in HCI J. Olson and W. Kellogg (Eds.). Springer 267--289. S. Egelman E. Chi and S. Dow. 2014. Crowdsourcing in HCI research. In Ways of Knowing in HCI J. Olson and W. Kellogg (Eds.). Springer 267--289. · DOI
  19. 10.1108/10662240510590360 · DOI · Paper page
  20. 10.1016/j.chb.2018.02.025 · DOI · Paper page
  21. Eric N. Forsythand and Craig H. Martell. 2007. Lexical and discourse analysis of online chat dialog . In Proceedings of the International Conference on Semantic Computing. IEEE, 19--26 . Eric N. Forsythand and Craig H. Martell. 2007. Lexical and discourse analysis of online chat dialog. In Proceedings of the International Conference on Semantic Computing. IEEE, 19--26.
  22. 10.1177/152582202237725 · DOI · Paper page
  23. 10.1145/2702123.2702443 · DOI · Paper page
  24. 10.1109/TE.2005.856149 · DOI · Paper page
  25. Herbert P. Grice . 1975. Logic and conversation . In Speech Acts . Brill , 41–58. Herbert P. Grice. 1975. Logic and conversation. In Speech Acts. Brill, 41–58.
  26. Elisabeth Hayes and Maryellen Ohrnberger . 2013 . The gamer generation teaches school: The gaming practices and attitudes towards technology of pre-service teachers . Journal of Technology and Teacher Education 21 , 2 (2013), 154 -- 177 . Elisabeth Hayes and Maryellen Ohrnberger. 2013. The gamer generation teaches school: The gaming practices and attitudes towards technology of pre-service teachers. Journal of Technology and Teacher Education 21, 2 (2013), 154--177.
  27. 10.1093/ijpor/edl028 · DOI
  28. Charles T. Hemphill , John J. Godfrey , and George R. Doddington . 1990. The ATIS spoken language systems pilot corpus . In Proceedings of the workshop on Speech and Natural Language. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Proceedings of the workshop on Speech and Natural Language.
  29. 10.1177/0956797611436349 · DOI · Paper page
  30. Knut Hofland and Stig Johansson . 1982. Word Frequencies in British and American English . Norwegian Computing Centre for the Humanities. Knut Hofland and Stig Johansson. 1982. Word Frequencies in British and American English. Norwegian Computing Centre for the Humanities.
  31. 10.1017/s1351324900002497 · DOI · Paper page
  32. 10.1145/2736277.2741669 · DOI · Paper page
  33. Douglas Samuel Jones . 1979. Elementary Information Theory . Clarendon Press . Douglas Samuel Jones. 1979. Elementary Information Theory. Clarendon Press.
  34. Jie Kang , Kyle Condiff , Shuo Chang , Joseph A. Konstan , Loren Terveen , and F. Maxwell Harper . 2017. Understanding how people use natural language to ask for recommendations . In Proceedings of the 11th ACM Conference on Recommender Systems. ACM, 229--237 . Jie Kang, Kyle Condiff, Shuo Chang, Joseph A. Konstan, Loren Terveen, and F. Maxwell Harper. 2017. Understanding how people use natural language to ask for recommendations. In Proceedings of the 11th ACM Conference on Recommender Systems. ACM, 229--237.
  35. 10.1145/3290605.3300316 · DOI · Paper page
  36. 10.1146/annurev.psych.50.1.537 · DOI · Paper page
  37. Paul J. Lavrakas . 2008. Encyclopedia of Survey Research Methods . Sage Publications . Paul J. Lavrakas. 2008. Encyclopedia of Survey Research Methods. Sage Publications.
  38. 10.3389/fpsyg.2016.01771 · DOI
  39. 10.1145/3025171.3025206 · DOI
  40. Vera Q. Liao , Muhammed Masud Hussain , Praveen Chandar , Matthew Davis , Marco Crasso , Dakuo Wang , Michael Muller , Sadat N. Shami , and Werner Geyer . 2018 . All work and no play? conversations with a question-and-answer chatbot in the wild . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18) . ACM, New York, NY , Vol. 13 . Vera Q. Liao, Muhammed Masud Hussain, Praveen Chandar, Matthew Davis, Marco Crasso, Dakuo Wang, Michael Muller, Sadat N. Shami, and Werner Geyer. 2018. All work and no play? conversations with a question-and-answer chatbot in the wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). ACM, New York, NY, Vol. 13.
  41. 10.18653/v1/D16-1230 · DOI · Paper page
  42. S. Louw , R. W. Todd , and P. Jimakorn . 2010. Active listening in qualitative research interviews . In Proceedings of the Doing Research in Applied Linguistics. 71--82 . S. Louw, R. W. Todd, and P. Jimakorn. 2010. Active listening in qualitative research interviews. In Proceedings of the Doing Research in Applied Linguistics. 71--82.
  43. 10.1177/0894439313485201 · DOI · Paper page
  44. 10.3115/974499.974549 · doi:10.3115/974499.974549
  45. Hendrik Muller , Aaron Sedley , and Elizabeth Ferrall-Nunge . 2014. Survey research in HCI . In Ways of Knowing in HCI . Springer , 229--266. Hendrik Muller, Aaron Sedley, and Elizabeth Ferrall-Nunge. 2014. Survey research in HCI. In Ways of Knowing in HCI. Springer, 229--266.
  46. 10.1007/978-3-319-07230-2_45 · DOI
  47. Marije Oudejans and Leah Melani Christian. 2010. Using interactive features to motivate and probe responses to open-ended questions. In Social and Behavioral Research and the Internet. 215--244. Marije Oudejans and Leah Melani Christian. 2010. Using interactive features to motivate and probe responses to open-ended questions. In Social and Behavioral Research and the Internet. 215--244. · doi:10.4324/9780203844922-9
  48. W. K. Park. 2005. Mobile phone addiction. In Mobile Communications. 253–272. W. K. Park. 2005. Mobile phone addiction. In Mobile Communications. 253–272. · DOI · Paper page
  49. Richard Paul and Linda Elder . 2006. The Thinker’s Guide to the Art of Socratic Questioning . Foundation for Critical Thinking Dillon Beach , CA. Richard Paul and Linda Elder. 2006. The Thinker’s Guide to the Art of Socratic Questioning. Foundation for Critical Thinking Dillon Beach, CA.
  50. 10.1037/0022-3514.41.5.847 · doi:10.1037/0022-3514.41.5.847 · Paper page
  51. 10.1002/ir.97 · DOI · Paper page
  52. 10.1002/ir.101 · DOI · Paper page
  53. 10.1145/3020165.3020183 · DOI · Paper page
  54. Ursa Reja , Katja Lozar Manfreda , Valentina Hlebec , and Vasja Vehovar . 2003 . Open-ended vs. close-ended questions in web questionnaires . Developments in Applied Statistics 19 , 1 (2003), 159 -- 177 . Ursa Reja, Katja Lozar Manfreda, Valentina Hlebec, and Vasja Vehovar. 2003. Open-ended vs. close-ended questions in web questionnaires. Developments in Applied Statistics 19, 1 (2003), 159--177.
  55. Francesco Ricci , Lior Rokach , and Bracha Shapira . 2015. Recommender systems: Introduction and challenges . In Recommender Systems Handbook . Springer , 1--34. Francesco Ricci, Lior Rokach, and Bracha Shapira. 2015. Recommender systems: Introduction and challenges. In Recommender Systems Handbook. Springer, 1--34.
  56. P. H. Rossi J. D. Wright and A. B. Anderson (Eds.). 2013. Handbook of Survey Research. Academic Press. P. H. Rossi J. D. Wright and A. B. Anderson (Eds.). 2013. Handbook of Survey Research. Academic Press.
  57. Shruti Sannon , Brett Stoll , Dominic DiFranzo , Malte Jung , and Natalya N. Bazarova . 2018. How personification and interactivity influence stress-related disclosures to conversational agents . In Proceedings of the Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 285--288 . Shruti Sannon, Brett Stoll, Dominic DiFranzo, Malte Jung, and Natalya N. Bazarova. 2018. How personification and interactivity influence stress-related disclosures to conversational agents. In Proceedings of the Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 285--288.
  58. Jessica Schroeder , Chelsey Wilkes , Kael Rowan , Arturo Toledo , Ann Paradiso , Mary Czerwinski , Gloria Mark , and Marsha M. Linehan . 2018. Pocket skills: A conversational mobile web app to support dialectical behavioral therapy . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 398 . Jessica Schroeder, Chelsey Wilkes, Kael Rowan, Arturo Toledo, Ann Paradiso, Mary Czerwinski, Gloria Mark, and Marsha M. Linehan. 2018. Pocket skills: A conversational mobile web app to support dialectical behavioral therapy. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 398.
  59. 10.1145/1541948.1541983 · DOI · Paper page
  60. 10.1145/3173574.3173965 · DOI
  61. Eleanor Singer and Mick P. Couper . 2017 . Some methodological uses of responses to open questions and other verbatim comments in quantitative surveys. Methods, Data, Analyses : A Journal for Quantitative Methods and Survey Methodology 11 , 2 (2017), 115 -- 134 . Eleanor Singer and Mick P. Couper. 2017. Some methodological uses of responses to open questions and other verbatim comments in quantitative surveys. Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology 11, 2 (2017), 115--134.
  62. 10.1145/3173574.3174178 · DOI
  63. 10.1145/3176349.3176388 · DOI · Paper page
  64. 10.1016/j.artint.2007.09.009 · DOI · Paper page
  65. David Traum . 2017. Computational approaches to dialogue . In The Routledge Handbook of Language and Dialogue, Edda Weigand (Ed.). Routledge , 143–161. David Traum. 2017. Computational approaches to dialogue. In The Routledge Handbook of Language and Dialogue, Edda Weigand (Ed.). Routledge, 143–161.
  66. 10.1145/3176349.3176387 · DOI
  67. Marilyn A. Walker , Diane J. Litman , Candace A. Kamm , and Alicia Abella . 1997 . PARADISE: A framework for evaluating spoken dialogue agents . In Proceedings of the 8th Conference of the European Chapter of the Association for Computational Linguistics. Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proceedings of the 8th Conference of the European Chapter of the Association for Computational Linguistics.
  68. 10.1145/2702123.2702293 · DOI
  69. 10.1177/0894439313505829 · DOI · Paper page
  70. 10.1145/3173574.3173662 · DOI · Paper page
  71. 10.1145/3301275.3302264 · DOI
  72. 10.1145/3025453.3025496 · DOI · Paper page
  73. 10.1145/3172944.3172998 · doi:10.1145/3172944.3172998

Cited by

See all 185

Metadata

Authors
Xiao, Z., Zhou, M. X., Liao, Q. V., Mark, G., Chi, C., Chen, W., & Yang, H.
Venue
ACM Transactions on Computer-Human Interaction
Published
2020-06-13
Volume
27
Issue
3
Pages
1-37
ISSN
  • 1073-0516 (ISSN-L)
  • 1557-7325
OpenAlex
W4288080268
Cited by
185
References
16
FWCI
23.47
Type
journal article
Language
en
Open access
bronze
Funders
Air Force Office of Scientific Research (FA9550-15-C-0032)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.