Understanding How to Administer Voice Surveys through Smart Speakers
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
Smart speakers have become exceedingly popular and entered many people's homes due to their ability to engage users with natural conversations. Researchers have also looked into using smart speakers as an interface to collect self-reported health data through conversations. Responding to surveys prompted by smart speakers requires users to listen to questions and answer in voice without any visual stimuli. Compared to traditional web-based surveys, where users can see questions and answers visually, voice surveys may be more cognitively challenging. Therefore, to collect reliable survey data, it is important to understand what types of questions are suitable to be administered by smart speakers. We selected five common survey questionnaires and deployed them as voice surveys and web surveys in a within-subject study. Our 24 participants answered questions using voice and web questionnaires in one session. They then repeated the same study session after 1 week to provide a "retest'' response. Our results suggest that voice surveys have comparable reliability to web surveys. We find that, when using 5-point or 7-point scales, voice surveys take about twice as long as web surveys. Based on objective measurements, such as response agreement and test-retest reliability, and subjective evaluations of user experience, we recommend that researchers consider adopting the binary scale and 5-point numerical scales for voice surveys on smart speakers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As such, compared to rule-based chatbots that can almost 100% ask pre-defined scripts [107], we can see that not all specified information slots were asked by our chatbots during the study.”
From Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data (2024) · cites this paper as [107]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“Before the recent advances in ASR technologies [16], a study dating back to 1989 implemented IVR surveys that allow voice inputs [13].”
BackgroundDawes, J. (2008) · International Journal of Market Research · doi:10.1177/147078530805000106 · Cited by 2,107
Markland, D., & Tobin, V. (2004) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.26.2.191 · Cited by 1,444 · cited 1×
“…1) Positive Thinking Scale (PTS, 22 yes/no items) [17], 2) The Behavioural Regulation In Exercise Questionnaire-3 (BREQ-3, 24 5-point scale items) [48, 87], 3) Problematic Use of Mobile Phones Scale (PUMP, 20 5-point scale items) [50], and 4) Fear of Spiders Questionnaire (FSQ, 18 7-point scale…”
BackgroundThe SAGE Encyclopedia of Communication Research Methods
Allen, M. (2017) · doi:10.4135/9781483381411 · Cited by 1,103
Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires
Holbrook, A. L., Green, M. C., & Krosnick, J. A. (2003) · Public Opinion Quarterly · doi:10.1086/346010 · Cited by 916
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765 · cited 1×
“Therefore, before mass-deploying voice surveys, researchers should consider measuring users’ home network conditions [60], minimizing ASR errors by designing better surveys [29], and evaluating voice questionnaire designs with people with different accents [33, 54].”
BackgroundMeasures of Political Attitudes.
Maranell, G. M., Robinson, J. P., Rusk, J. G., & Head, K. B. (1970) · American Sociological Review · doi:10.2307/2093023 · Cited by 754 · cited 1×
“On the other hand, close-ended questions, where respondents are given a set of options to choose from [67], are easier to administer.”
BackgroundVoice Interfaces in Everyday Life
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174214 · Cited by 717 · cited 3×
“Recent studies have further suggested that interactions with speakers can be error-prone and subject to network delays [58, 85].”
Background“It's Who I Am … Really!’ The Importance of Integrated Regulation in Exercise Contexts1
Wilson, P. M., Rodgers, W. M., Loitz, C. C., & Scime, G. (2006) · Journal of Applied Biobehavioral Research · doi:10.1111/j.1751-9861.2006.tb00021.x · Cited by 457
Fear of spiders’ questionnaire
Szymanski, J., & O'Donohue, W. (1995) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)00072-t · Cited by 450 · cited 4×
“Previous studies on our selected questionnaires have already confirmed that those questionnaires have great split-half reliability, i.e., any halves of those questionnaires are equal, and participants should score similarly in both halves [17, 24, 77, 86].”
BackgroundMethodResultInfluential
Research works printed but not linked: 22
- Duane F Alwin . 1992. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. Sociological methodology ( 1992 ), 83--118. Duane F Alwin. 1992. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. Sociological methodology (1992), 83--118.
- Richard L Clayton and Debbie LS Winter . 1992 . Speech data entry: results of a test of voice recognition for survey data collection . JOURNAL OF OFFICIAL STATISTICS-STOCKHOLM- 8 (1992), 377 -- 377 . Richard L Clayton and Debbie LS Winter. 1992. Speech data entry: results of a test of voice recognition for survey data collection. JOURNAL OF OFFICIAL STATISTICS-STOCKHOLM- 8 (1992), 377--377.
- Don C Des Jarlais , Denise Paone , Judith Milliken , Charles F Turner , Heather Miller , James Gribble , Qiuhu Shi , Holly Hagan , and Samuel R Friedman . 1999. Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. The Lancet 353, 9165 ( 1999 ), 1657--1661. Don C Des Jarlais, Denise Paone, Judith Milliken, Charles F Turner, Heather Miller, James Gribble, Qiuhu Shi, Holly Hagan, and Samuel R Friedman. 1999. Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. The Lancet 353, 9165 (1999), 1657--1661.
- Ed Diener , Derrick Wirtz , Robert Biswas-Diener , William Tov , Chu Kim-Prieto , Dong-won Choi, and Shigehiro Oishi . 2009. New measures of well-being . In Assessing well-being . Springer , 247--266. Ed Diener, Derrick Wirtz, Robert Biswas-Diener, William Tov, Chu Kim-Prieto, Dong-won Choi, and Shigehiro Oishi. 2009. New measures of well-being. In Assessing well-being. Springer, 247--266.
- Don A Dillman and Leah Melani Christian . 2005. Survey mode as a source of instability in responses across surveys. Field methods 17, 1 ( 2005 ), 30--52. Don A Dillman and Leah Melani Christian. 2005. Survey mode as a source of instability in responses across surveys. Field methods 17, 1 (2005), 30--52.
- Don A Dillman , Glenn Phelps , Robert Tortora , Karen Swift , Julie Kohrell , Jodi Berck , and Benjamin L Messer . 2009. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social science research 38, 1 ( 2009 ), 1--18. Don A Dillman, Glenn Phelps, Robert Tortora, Karen Swift, Julie Kohrell, Jodi Berck, and Benjamin L Messer. 2009. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social science research 38, 1 (2009), 1--18.
- Khalil G Ghanem , Heidi E Hutton , Jonathan M Zenilman , Rebecca Zimba , and Emily J Erbelding . 2005. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sexually transmitted infections 81, 5 ( 2005 ), 421--425. Khalil G Ghanem, Heidi E Hutton, Jonathan M Zenilman, Rebecca Zimba, and Emily J Erbelding. 2005. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sexually transmitted infections 81, 5 (2005), 421--425.
- Katharina Graben , Bettina K Doering , Franziska Jeromin , and Antonia Barke . 2020. Problematic mobile phone use: Validity and reliability of the Problematic Use of Mobile Phone (PUMP) Scale in a German sample. Addictive behaviors reports 12 ( 2020 ), 100297. Katharina Graben, Bettina K Doering, Franziska Jeromin, and Antonia Barke. 2020. Problematic mobile phone use: Validity and reliability of the Problematic Use of Mobile Phone (PUMP) Scale in a German sample. Addictive behaviors reports 12 (2020), 100297.
- Jiepu Jiang , Wei Jeng , and Daqing He . 2013 . How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search . In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. Association for Computing Machinery , New York, NY, USA, 143--152. Jiepu Jiang, Wei Jeng, and Daqing He. 2013. How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. Association for Computing Machinery, New York, NY, USA, 143--152.
- Aman Khullar , Priyadarshi Hitesh , Shoaib Rahman , Deepak Kumar , Rachit Pandey , Praveen Kumar , Rajeshwari Tripathi , Prince Prince , Ankit Akash Jha , Himanshu Himanshu, et al. 2021 . Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response Systems. In ACM SIGCAS Conference on Computing and Sustainable Societies. Association for Computing Machinery , New York, NY, USA, 288--298. Aman Khullar, Priyadarshi Hitesh, Shoaib Rahman, Deepak Kumar, Rachit Pandey, Praveen Kumar, Rajeshwari Tripathi, Prince Prince, Ankit Akash Jha, Himanshu Himanshu, et al. 2021. Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response Systems. In ACM SIGCAS Conference on Computing and Sustainable Societies. Association for Computing Machinery, New York, NY, USA, 288--298.
- Bret Kinsella . 2019. Loup Ventures says 75% of U.S. households will have smart speakers by 2025 , Google to surpass Amazon in market share. https://voicebot.ai/2019/06/18/loup-ventures-says-75-of-u-s-households-will-have-smartspeakers-by-2025-google-to-surpass-amazon-in-market-share/ Bret Kinsella. 2019. Loup Ventures says 75% of U.S. households will have smart speakers by 2025, Google to surpass Amazon in market share. https://voicebot.ai/2019/06/18/loup-ventures-says-75-of-u-s-households-will-have-smartspeakers-by-2025-google-to-surpass-amazon-in-market-share/
- Jon A Krosnick . 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology 5, 3 ( 1991 ), 213--236. Jon A Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology 5, 3 (1991), 213--236.
- Jon A Krosnick and Matthew K Berent . 1993. Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science ( 1993 ), 941--964. Jon A Krosnick and Matthew K Berent. 1993. Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science (1993), 941--964.
- Jon A Krosnick , Sowmya Narayan , and Wendy R Smith . 1996. Satisficing in surveys: Initial evidence. New directions for evaluation 1996 , 70 (1996), 29--44. Jon A Krosnick, Sowmya Narayan, and Wendy R Smith. 1996. Satisficing in surveys: Initial evidence. New directions for evaluation 1996, 70 (1996), 29--44.
- J Richard Landis and Gary G Koch . 1977. The measurement of observer agreement for categorical data. biometrics ( 1977 ), 159--174. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159--174.
- Rensis Likert . 1932. A technique for the measurement of attitudes. Archives of psychology ( 1932 ). Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology (1932).
- Raju Maharjan , Per Bækgaard , and Jakob E Bardram . 2019. " Hear me out" smart speaker based conversational agent to monitor symptoms in mental health. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers . Association for Computing Machinery , New York, NY, USA , 929--933. Raju Maharjan, Per Bækgaard, and Jakob E Bardram. 2019. " Hear me out" smart speaker based conversational agent to monitor symptoms in mental health. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA, 929--933.
- Lisa J Merlo , Amanda M Stone , and Alex Bibbey . 2013. Measuring problematic mobile phone use: development and preliminary psychometric properties of the PUMP scale. Journal of addiction 2013 ( 2013 ). Lisa J Merlo, Amanda M Stone, and Alex Bibbey. 2013. Measuring problematic mobile phone use: development and preliminary psychometric properties of the PUMP scale. Journal of addiction 2013 (2013).
- Charles Egerton Osgood , George J Suci , and Percy H Tannenbaum . 1957. The measurement of meaning. Number 47 . University of Illinois press. Charles Egerton Osgood, George J Suci, and Percy H Tannenbaum. 1957. The measurement of meaning. Number 47. University of Illinois press.
- Alisha Pradhan , Leah Findlater , and Amanda Lazar . 2019 . " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults . Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--21. Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--21.
- Roger Tourangeau and Kenneth A Rasinski . 1988. Cognitive processes underlying context effects in attitude measurement. Psychological bulletin 103, 3 ( 1988 ), 299. Roger Tourangeau and Kenneth A Rasinski. 1988. Cognitive processes underlying context effects in attitude measurement. Psychological bulletin 103, 3 (1988), 299.
- Charles F Turner , Leighton Ku , Susan M Rogers , Laura D Lindberg , Joseph H Pleck , and Freya L Sonenstein . 1998. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 280, 5365 ( 1998 ), 867--873. Charles F Turner, Leighton Ku, Susan M Rogers, Laura D Lindberg, Joseph H Pleck, and Freya L Sonenstein. 1998. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 280, 5365 (1998), 867--873.
Cited by
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637364 · Cited by 84
“As such, compared to rule-based chatbots that can almost 100% ask pre-defined scripts [97], we can see that not all specified information slots were asked by our chatbots during the study.”
BackgroundSelf-citationLim, J., Koh, Y., Kim, A., & Lee, U. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642846 · Cited by 38
Designing the Conversational Agent: Asking Follow-up Questions for Information Elicitation
Hu, J., Guo, J., Tang, N., Ma, X., Yao, Y., Yang, C., & Xu, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637320 · Cited by 19
“The user study in this work involved human participants as the previous works on the human-CA interaction [13, 39, 67], which extended the findings’ generalizability compared to the work that only evaluated the follow-up question generation results with automatic metrics [62].”
BackgroundMethodResultInfluentialSpeaking of accent: A content analysis of accent misconceptions in ASR research
Prinos, K., Patwari, N., & Power, C. A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658969 · Cited by 12
Fusang: Graph-inspired Robust and Accurate Object Recognition on Commodity mmWave Devices
He, G., Chen, S., Xu, D., Chen, X., Xie, Y., Wang, X., & Fang, D. (2023) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3581791.3596849 · Cited by 14
Metadata
- Authors
- Wei, J., Jiang, W., Wang, C., Yu, D., Goncalves, J., Dingler, T., & Kostakos, V.
- Affiliations
- The University of Melbourne (AU)
- University of Melbourne, Melbourne, VIC, Australia
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2022-11-07
- Volume
- 6
- Issue
- CSCW2
- Pages
- 1-32
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4309619903
- Cited by
- 9
- References
- 65
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 3
- FWCI
- 3.24
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- Funders
- National Health and Medical Research Council (1170937, 2004316)
- Australian Research Council (DP190102627)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- smart speakers, speech interfaces, questionnaire development, test-retest reliability, reaction time (psychology), scale types, user experience, self-reported health data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Understanding How to Administer Voice Surveys through Smart Speakers. https://app.cochat.ai/papers/10.1145/3555606
- The paper
- Wei, J., Jiang, W., Wang, C., Yu, D., Goncalves, J., Dingler, T., & Kostakos, V. (2022). Understanding How to Administer Voice Surveys through Smart Speakers. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3555606
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.