CoChat

Understanding How to Administer Voice Surveys through Smart Speakers

Wei, Jing, Jiang, Weiwei, Wang, Chaofan, Yu, Difeng, Goncalves, Jorge, Dingler, Tilman, Kostakos, Vassilis · 2022 · Proceedings of the ACM on Human-Computer Interaction

The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.

Abstract

Smart speakers have become exceedingly popular and entered many people's homes due to their ability to engage users with natural conversations. Researchers have also looked into using smart speakers as an interface to collect self-reported health data through conversations. Responding to surveys prompted by smart speakers requires users to listen to questions and answer in voice without any visual stimuli. Compared to traditional web-based surveys, where users can see questions and answers visually, voice surveys may be more cognitively challenging. Therefore, to collect reliable survey data, it is important to understand what types of questions are suitable to be administered by smart speakers. We selected five common survey questionnaires and deployed them as voice surveys and web surveys in a within-subject study. Our 24 participants answered questions using voice and web questionnaires in one session. They then repeated the same study session after 1 week to provide a "retest'' response. Our results suggest that voice surveys have comparable reliability to web surveys. We find that, when using 5-point or 7-point scales, voice surveys take about twice as long as web surveys. Based on objective measurements, such as response agreement and test-retest reliability, and subjective evaluations of user experience, we recommend that researchers consider adopting the binary scale and 5-point numerical scales for voice surveys on smart speakers.

Overview

Integrity signals

Registry

Crossref · 10.1145/3555606

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 22

  1. Duane F Alwin . 1992. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. Sociological methodology ( 1992 ), 83--118. Duane F Alwin. 1992. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. Sociological methodology (1992), 83--118.
  2. Richard L Clayton and Debbie LS Winter . 1992 . Speech data entry: results of a test of voice recognition for survey data collection . JOURNAL OF OFFICIAL STATISTICS-STOCKHOLM- 8 (1992), 377 -- 377 . Richard L Clayton and Debbie LS Winter. 1992. Speech data entry: results of a test of voice recognition for survey data collection. JOURNAL OF OFFICIAL STATISTICS-STOCKHOLM- 8 (1992), 377--377.
  3. Don C Des Jarlais , Denise Paone , Judith Milliken , Charles F Turner , Heather Miller , James Gribble , Qiuhu Shi , Holly Hagan , and Samuel R Friedman . 1999. Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. The Lancet 353, 9165 ( 1999 ), 1657--1661. Don C Des Jarlais, Denise Paone, Judith Milliken, Charles F Turner, Heather Miller, James Gribble, Qiuhu Shi, Holly Hagan, and Samuel R Friedman. 1999. Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. The Lancet 353, 9165 (1999), 1657--1661.
  4. Ed Diener , Derrick Wirtz , Robert Biswas-Diener , William Tov , Chu Kim-Prieto , Dong-won Choi, and Shigehiro Oishi . 2009. New measures of well-being . In Assessing well-being . Springer , 247--266. Ed Diener, Derrick Wirtz, Robert Biswas-Diener, William Tov, Chu Kim-Prieto, Dong-won Choi, and Shigehiro Oishi. 2009. New measures of well-being. In Assessing well-being. Springer, 247--266.
  5. Don A Dillman and Leah Melani Christian . 2005. Survey mode as a source of instability in responses across surveys. Field methods 17, 1 ( 2005 ), 30--52. Don A Dillman and Leah Melani Christian. 2005. Survey mode as a source of instability in responses across surveys. Field methods 17, 1 (2005), 30--52.
  6. Don A Dillman , Glenn Phelps , Robert Tortora , Karen Swift , Julie Kohrell , Jodi Berck , and Benjamin L Messer . 2009. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social science research 38, 1 ( 2009 ), 1--18. Don A Dillman, Glenn Phelps, Robert Tortora, Karen Swift, Julie Kohrell, Jodi Berck, and Benjamin L Messer. 2009. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social science research 38, 1 (2009), 1--18.
  7. Khalil G Ghanem , Heidi E Hutton , Jonathan M Zenilman , Rebecca Zimba , and Emily J Erbelding . 2005. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sexually transmitted infections 81, 5 ( 2005 ), 421--425. Khalil G Ghanem, Heidi E Hutton, Jonathan M Zenilman, Rebecca Zimba, and Emily J Erbelding. 2005. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sexually transmitted infections 81, 5 (2005), 421--425.
  8. Katharina Graben , Bettina K Doering , Franziska Jeromin , and Antonia Barke . 2020. Problematic mobile phone use: Validity and reliability of the Problematic Use of Mobile Phone (PUMP) Scale in a German sample. Addictive behaviors reports 12 ( 2020 ), 100297. Katharina Graben, Bettina K Doering, Franziska Jeromin, and Antonia Barke. 2020. Problematic mobile phone use: Validity and reliability of the Problematic Use of Mobile Phone (PUMP) Scale in a German sample. Addictive behaviors reports 12 (2020), 100297.
  9. Jiepu Jiang , Wei Jeng , and Daqing He . 2013 . How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search . In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. Association for Computing Machinery , New York, NY, USA, 143--152. Jiepu Jiang, Wei Jeng, and Daqing He. 2013. How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. Association for Computing Machinery, New York, NY, USA, 143--152.
  10. Aman Khullar , Priyadarshi Hitesh , Shoaib Rahman , Deepak Kumar , Rachit Pandey , Praveen Kumar , Rajeshwari Tripathi , Prince Prince , Ankit Akash Jha , Himanshu Himanshu, et al. 2021 . Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response Systems. In ACM SIGCAS Conference on Computing and Sustainable Societies. Association for Computing Machinery , New York, NY, USA, 288--298. Aman Khullar, Priyadarshi Hitesh, Shoaib Rahman, Deepak Kumar, Rachit Pandey, Praveen Kumar, Rajeshwari Tripathi, Prince Prince, Ankit Akash Jha, Himanshu Himanshu, et al. 2021. Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response Systems. In ACM SIGCAS Conference on Computing and Sustainable Societies. Association for Computing Machinery, New York, NY, USA, 288--298.
  11. Bret Kinsella . 2019. Loup Ventures says 75% of U.S. households will have smart speakers by 2025 , Google to surpass Amazon in market share. https://voicebot.ai/2019/06/18/loup-ventures-says-75-of-u-s-households-will-have-smartspeakers-by-2025-google-to-surpass-amazon-in-market-share/ Bret Kinsella. 2019. Loup Ventures says 75% of U.S. households will have smart speakers by 2025, Google to surpass Amazon in market share. https://voicebot.ai/2019/06/18/loup-ventures-says-75-of-u-s-households-will-have-smartspeakers-by-2025-google-to-surpass-amazon-in-market-share/
  12. Jon A Krosnick . 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology 5, 3 ( 1991 ), 213--236. Jon A Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology 5, 3 (1991), 213--236.
  13. Jon A Krosnick and Matthew K Berent . 1993. Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science ( 1993 ), 941--964. Jon A Krosnick and Matthew K Berent. 1993. Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science (1993), 941--964.
  14. Jon A Krosnick , Sowmya Narayan , and Wendy R Smith . 1996. Satisficing in surveys: Initial evidence. New directions for evaluation 1996 , 70 (1996), 29--44. Jon A Krosnick, Sowmya Narayan, and Wendy R Smith. 1996. Satisficing in surveys: Initial evidence. New directions for evaluation 1996, 70 (1996), 29--44.
  15. J Richard Landis and Gary G Koch . 1977. The measurement of observer agreement for categorical data. biometrics ( 1977 ), 159--174. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159--174.
  16. Rensis Likert . 1932. A technique for the measurement of attitudes. Archives of psychology ( 1932 ). Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology (1932).
  17. Raju Maharjan , Per Bækgaard , and Jakob E Bardram . 2019. " Hear me out" smart speaker based conversational agent to monitor symptoms in mental health. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers . Association for Computing Machinery , New York, NY, USA , 929--933. Raju Maharjan, Per Bækgaard, and Jakob E Bardram. 2019. " Hear me out" smart speaker based conversational agent to monitor symptoms in mental health. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA, 929--933.
  18. Lisa J Merlo , Amanda M Stone , and Alex Bibbey . 2013. Measuring problematic mobile phone use: development and preliminary psychometric properties of the PUMP scale. Journal of addiction 2013 ( 2013 ). Lisa J Merlo, Amanda M Stone, and Alex Bibbey. 2013. Measuring problematic mobile phone use: development and preliminary psychometric properties of the PUMP scale. Journal of addiction 2013 (2013).
  19. Charles Egerton Osgood , George J Suci , and Percy H Tannenbaum . 1957. The measurement of meaning. Number 47 . University of Illinois press. Charles Egerton Osgood, George J Suci, and Percy H Tannenbaum. 1957. The measurement of meaning. Number 47. University of Illinois press.
  20. Alisha Pradhan , Leah Findlater , and Amanda Lazar . 2019 . " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults . Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--21. Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--21.
  21. Roger Tourangeau and Kenneth A Rasinski . 1988. Cognitive processes underlying context effects in attitude measurement. Psychological bulletin 103, 3 ( 1988 ), 299. Roger Tourangeau and Kenneth A Rasinski. 1988. Cognitive processes underlying context effects in attitude measurement. Psychological bulletin 103, 3 (1988), 299.
  22. Charles F Turner , Leighton Ku , Susan M Rogers , Laura D Lindberg , Joseph H Pleck , and Freya L Sonenstein . 1998. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 280, 5365 ( 1998 ), 867--873. Charles F Turner, Leighton Ku, Susan M Rogers, Laura D Lindberg, Joseph H Pleck, and Freya L Sonenstein. 1998. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 280, 5365 (1998), 867--873.

Cited by

See all 10

Metadata

Authors
Wei, J., Jiang, W., Wang, C., Yu, D., Goncalves, J., Dingler, T., & Kostakos, V.
Affiliations
  • The University of Melbourne (AU)
  • University of Melbourne, Melbourne, VIC, Australia
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2022-11-07
Volume
6
Issue
CSCW2
Pages
1-32
ISSN
2573-0142 (ISSN-L)
OpenAlex
W4309619903
Cited by
9
References
65
Citations by year
2025: 2 · 2024: 4 · 2023: 3
FWCI
3.24
Type
journal article
Language
en
Open access
closed
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion)
Funders
  • National Health and Medical Research Council (1170937, 2004316)
  • Australian Research Council (DP190102627)
Topics
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Keywords
smart speakers, speech interfaces, questionnaire development, test-retest reliability, reaction time (psychology), scale types, user experience, self-reported health data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.