Understanding How to Administer Voice Surveys through Smart Speakers
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
Smart speakers have become exceedingly popular and entered many people's homes due to their ability to engage users with natural conversations. Researchers have also looked into using smart speakers as an interface to collect self-reported health data through conversations. Responding to surveys prompted by smart speakers requires users to listen to questions and answer in voice without any visual stimuli. Compared to traditional web-based surveys, where users can see questions and answers visually, voice surveys may be more cognitively challenging. Therefore, to collect reliable survey data, it is important to understand what types of questions are suitable to be administered by smart speakers. We selected five common survey questionnaires and deployed them as voice surveys and web surveys in a within-subject study. Our 24 participants answered questions using voice and web questionnaires in one session. They then repeated the same study session after 1 week to provide a "retest'' response. Our results suggest that voice surveys have comparable reliability to web surveys. We find that, when using 5-point or 7-point scales, voice surveys take about twice as long as web surveys. Based on objective measurements, such as response agreement and test-retest reliability, and subjective evaluations of user experience, we recommend that researchers consider adopting the binary scale and 5-point numerical scales for voice surveys on smart speakers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As such, compared to rule-based chatbots that can almost 100% ask pre-defined scripts [107], we can see that not all specified information slots were asked by our chatbots during the study.”
From Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data (2024) · cites this paper as [107]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“Before the recent advances in ASR technologies [16], a study dating back to 1989 implemented IVR surveys that allow voice inputs [13].”
BackgroundDawes, J. (2008) · International Journal of Market Research · doi:10.1177/147078530805000106 · Cited by 2,107
Markland, D., & Tobin, V. (2004) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.26.2.191 · Cited by 1,444 · cited 1×
“…1) Positive Thinking Scale (PTS, 22 yes/no items) [17], 2) The Behavioural Regulation In Exercise Questionnaire-3 (BREQ-3, 24 5-point scale items) [48, 87], 3) Problematic Use of Mobile Phones Scale (PUMP, 20 5-point scale items) [50], and 4) Fear of Spiders Questionnaire (FSQ, 18 7-point scale…”
BackgroundThe SAGE Encyclopedia of Communication Research Methods
Allen, M. (2017) · doi:10.4135/9781483381411 · Cited by 1,103
Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires
Holbrook, A. L., Green, M. C., & Krosnick, J. A. (2003) · Public Opinion Quarterly · doi:10.1086/346010 · Cited by 916
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765 · cited 1×
“Therefore, before mass-deploying voice surveys, researchers should consider measuring users’ home network conditions [60], minimizing ASR errors by designing better surveys [29], and evaluating voice questionnaire designs with people with different accents [33, 54].”
BackgroundMeasures of Political Attitudes.
Maranell, G. M., Robinson, J. P., Rusk, J. G., & Head, K. B. (1970) · American Sociological Review · doi:10.2307/2093023 · Cited by 754 · cited 1×
“On the other hand, close-ended questions, where respondents are given a set of options to choose from [67], are easier to administer.”
BackgroundVoice Interfaces in Everyday Life
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174214 · Cited by 717 · cited 3×
“Recent studies have further suggested that interactions with speakers can be error-prone and subject to network delays [58, 85].”
Background“It's Who I Am … Really!’ The Importance of Integrated Regulation in Exercise Contexts1
Wilson, P. M., Rodgers, W. M., Loitz, C. C., & Scime, G. (2006) · Journal of Applied Biobehavioral Research · doi:10.1111/j.1751-9861.2006.tb00021.x · Cited by 457
Fear of spiders’ questionnaire
Szymanski, J., & O'Donohue, W. (1995) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)00072-t · Cited by 450 · cited 4×
“Previous studies on our selected questionnaires have already confirmed that those questionnaires have great split-half reliability, i.e., any halves of those questionnaires are equal, and participants should score similarly in both halves [17, 24, 77, 86].”
BackgroundMethodResultInfluentialThe Experience Sampling Method on Mobile Devices
van Berkel, N., Ferreira, D., & Kostakos, V. (2018) · ACM Computing Surveys · doi:10.1145/3123988 · Cited by 432
Self-citationUnderstanding the Long-Term Use of Smart Speaker Assistants
Bentley, F., Luvogt, C., Silverman, M., Wirasinghe, R., White, B., & Lottridge, D. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264901 · Cited by 316 · cited 1×
“We speculate that although all of our participants are experienced smart speaker users, they still lack the experience of engaging in multi-turn conversations with their speakers [5].”
BackgroundThe impact of administration mode on response effects in survey measurement
Schwarz, N., Strack, F., Hippler, H., & Bishop, G. (1991) · Applied Cognitive Psychology · doi:10.1002/acp.2350050304 · Cited by 278 · cited 1×
“In particular, some participants struggled with the direction of the rating scale, which may be due to the lack of visual presentation [19, 63, 70].”
BackgroundPerlis, T. E., Des Jarlais, D. C., Friedman, S. R., Arasteh, K., & Turner, C. F. (2004) · Addiction · doi:10.1111/j.1360-0443.2004.00740.x · Cited by 223 · cited 1×
“With the presence of an observer, participants may opt to choose more “positive” answers without thinking more deeply, or they may attempt to give longer answers to open-ended questions [22, 57].”
BackgroundPatterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 2×
“If open-ended questions are administered through smart speakers, respondents may encounter interaction errors [52] and the recorded answers may also contain transcription errors [28, 84].”
BackgroundKim, S., Lee, J., & Gweon, G. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300316 · Cited by 190 · cited 4×
“Kim et al. [30] implemented both chatbot surveys and web surveys that administer mostly close-ended questions.”
BackgroundMethodInfluentialSimmons, A. D., & Bobo, L. D. (2015) · Sociological Methodology · doi:10.1177/0081175015570096 · Cited by 187
Using the Theory of Satisficing to Evaluate the Quality of Survey Data
Barge, S., & Gehlbach, H. (2012) · Research in Higher Education · doi:10.1007/s11162-011-9251-2 · Cited by 187 · cited 2×
“Conversely, as web questionnaires do not trigger any interaction errors, the high variance may actually indicate that some participants were rushing to complete the surveys [4].”
BackgroundXiao, Z., Zhou, M. X., Liao, Q. V., Mark, G., Chi, C., Chen, W., & Yang, H. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3381804 · Cited by 185
HealthLine: Speech-based access to health information by low-literate users
Sherwani, J., Ali, N., Mirza, S., Fatma, A., Memon, Y., Karim, M., Tongia, R., & Rosenfeld, R. (2007) · 2007 International Conference on Information and Communication Technologies and Development · doi:10.1109/ictd.2007.4937399 · Cited by 181 · cited 1×
“These works, together with previous studies, have shown that respondents from low-literate populations and developing regions benefit from voice-based IVR surveys [56, 71].”
The Palgrave Handbook of Survey Research
(2018) · Springer eBooks · doi:10.1007/978-3-319-54395-6 · Cited by 177
The Measurement of Personal Values in Survey Research
McCarty, J. A., & Shrum, L. J. (2000) · Public Opinion Quarterly · doi:10.1086/317989 · Cited by 170
Examining the Psychometric Properties of the Behavioral Regulation in Exercise Questionnaire
Wilson, P. M., Rodgers, W. M., & Fraser, S. N. (2002) · Measurement in Physical Education and Exercise Science · doi:10.1207/s15327841mpee0601_1 · Cited by 154 · cited 2×
“Previous studies on our selected questionnaires have already confirmed that those questionnaires have great split-half reliability, i.e., any halves of those questionnaires are equal, and participants should score similarly in both halves [17, 24, 77, 86].”
MethodResultHe Is Just Like Me: A Study of the Long-Term Use of Smart Speakers by Parents and Children
Garg, R., & Sengupta, S. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3381002 · Cited by 138 · cited 1×
“Researchers have investigated how smart speakers can be used to educate children [21], accompany older adults [59], and provide mental support for those in needs [62, 83].”
BackgroundRhim, J., Kwak, M., Gong, Y., & Gweon, G. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107034 · Cited by 138 · cited 1×
“Further, Rhim et al. [65] compared a baseline chatbot with a humanized chatbot that used humanizing traits like self-introduction, adaptive response speed, and echoing.”
BackgroundThe Oxford Handbook of American Elections and Political Behavior
(2010) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199235476.001.0001 · Cited by 128
Designing for Workplace Reflection: A Chat and Voice-Based Conversational Agent
Kocielnik, R., Avrahami, D., Marlow, J., Lu, D., & Hsieh, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196784 · Cited by 119
USA, U. '. T. 2. A. I. J. C. O. P. A. U. C. 2. V., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Computer-Human Interaction, ACM Special Interest Group on Mobility of Systems, U., ACM Special Interest Group on Spatial Information, & SIGs, A. (2021) · doi:10.1145/3460418 · Cited by 110
Call centre management: responsibilities and performance
Robinson, G., & Morley, C. L. (2006) · International Journal of Service Industry Management · doi:10.1108/09564230610667122 · Cited by 104 · cited 1×
“Although voice-based surveys have been widely implemented through IVR systems, especially the use of customer service call centers [66], most of our participants seem to consider voice surveys on smart speakers to be a new “thing”.”
BackgroundPyae, A., & Joelsson, T. N. (2018) · Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct · doi:10.1145/3236112.3236130 · Cited by 99 · cited 1×
“Therefore, before mass-deploying voice surveys, researchers should consider measuring users’ home network conditions [60], minimizing ASR errors by designing better surveys [29], and evaluating voice questionnaire designs with people with different accents [33, 54].”
BackgroundGivon, M. M., & Shapira, Z. (1984) · Journal of Marketing Research · doi:10.1177/002224378402100406 · Cited by 91 · cited 1×
“Existing literature suggests that moderately long scale lengths (i.e., more points) can increase both the reliability [23, 75] and the validity [68] of questionnaires.”
BackgroundA comparative study of speech and dialed input voice interfaces in rural India
Patel, N., Agarwal, S., Rajput, N., Nanavati, A., Dave, P., & Parikh, T. S. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518709 · Cited by 89
Focused, Aroused, but so Distractible: Temporal Perspectives on Multitasking and Communications
Mark, G., Iqbal, S., Czerwinski, M., & Johns, P. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675221 · Cited by 86 · cited 1×
“In the Computer-Supported Cooperative Work (CSCW) and Human-Computer Interaction (HCI) research community, surveys are widely used to acquire self-reported opinions from the crowd [11], gather data for intervention systems [82], collect frequent ESM self-reports of user sociometrics, such as creativity [79], mood, and…”
BackgroundShin, E., Johnson, T. P., & Rao, K. (2012) · Social Science Computer Review · doi:10.1177/0894439311404508 · Cited by 85
The ADL Taxonomy - from Individual Categorical Data to Ordinal Categorical Data
Sonn, U., Törnquist, K. B., & Svensson, E. (1999) · Scandinavian Journal of Occupational Therapy · doi:10.1080/110381299443807 · Cited by 81
The Use and Promise of Conversational Agents in Digital Health
Dingler, T., Kwasnicka, D., Wei, J., Gong, E., & Oldenburg, B. (2021) · Yearbook of Medical Informatics · doi:10.1055/s-0041-1726510 · Cited by 81
Self-citationEngaging Solidarity in Data Collection Practices for Community Health
Ismail, A., & Kumar, N. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274345 · Cited by 78 · cited 1×
“However, it is unclear whether data from such voice surveys can be as reliable and valid as pen&paper surveys or web/mobile surveys [27].”
BackgroundAlexa as Coach: Leveraging Smart Speakers to Build Social Agents that Reduce Public Speaking Anxiety
Wang, J., Yang, H., Shao, R., Abdullah, S., & Sundar, S. S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376561 · Cited by 75 · cited 2×
“Researchers have investigated how smart speakers can be used to educate children [21], accompany older adults [59], and provide mental support for those in needs [62, 83].”
BackgroundCha, N., Kim, A., Park, C. Y., Kang, S., Park, M., Lee, J.-G., Lee, S., & Lee, U. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3411810 · Cited by 74
Capturing the mood: facebook and face-to-face encounters in the workplace
Mark, G., Iqbal, S., Czerwinski, M., & Johns, P. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531673 · Cited by 72 · cited 1×
“In the Computer-Supported Cooperative Work (CSCW) and Human-Computer Interaction (HCI) research community, surveys are widely used to acquire self-reported opinions from the crowd [11], gather data for intervention systems [82], collect frequent ESM self-reports of user sociometrics, such as creativity [79], mood, and…”
BackgroundL’Engle, K., Sefa, E., Adimazoya, E. A., Yartey, E., Lenzi, R., Tarpo, C., Heward-Mills, N. L., Lew, K., & Ampeh, Y. (2018) · PLOS ONE · doi:10.1371/journal.pone.0190902 · Cited by 72
Four Years in Review: Statistical Practices of Likert Scales in Human-Robot Interaction Studies
Schrum, M. L., Johnson, M., Ghuy, M., & Gombolay, M. C. (2020) · Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3371382.3380739 · Cited by 72
Cavusgil, S. T., & Elvey, L. A. (1998) · European Journal of Marketing · doi:10.1108/03090569810243776 · Cited by 69 · cited 1×
“Extended survey completion time negatively impacts respondents’ motivation [7], especially for ESM-style surveys [81].”
BackgroundMore positive or More Extreme? A Meta-Analysis of Mode Differences in Response Choice
Ye, C., Fulton, J., & Tourangeau, R. (2011) · Public Opinion Quarterly · doi:10.1093/poq/nfr009 · Cited by 64
Celino, I., & Re Calegari, G. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102410 · Cited by 63 · cited 2×
“Similarly, Celino and Calegari [8] showed that submitting responses to conversational surveys can achieve the same reliability and a higher response quality with respect to traditional web surveys.”
BackgroundUser Experience with Smart Voice Assistants: The Accent Perspective
Pal, D., Arpnikanondt, C., Funilkul, S., & Varadarajan, V. (2019) · 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt45670.2019.8944754 · Cited by 60 · cited 1×
“Therefore, before mass-deploying voice surveys, researchers should consider measuring users’ home network conditions [60], minimizing ASR errors by designing better surveys [29], and evaluating voice questionnaire designs with people with different accents [33, 54].”
BackgroundActionable UI Design Guidelines for Smartphone Applications Inclusive of Low-Literate Users
Srivastava, A., Kapania, S., Tuli, A., & Singh, P. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449210 · Cited by 58 · cited 1×
“Prior work in developing countries has suggested that IVR voice surveys can include a short, straightforward introductory message of the survey to improve the completion rate, and use shorter question prompts (without response options) to improve data quality [43, 76].”
BackgroundSpeech Versus Touch: A Comparative Study of the Use of Speech and DTMF Keypad for Navigation
Lee, K. M., & Lai, J. (2005) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1903_4 · Cited by 50 · cited 1×
“Similarly, another early work in 2005 found that most users preferred the voice inputs to keypad inputs despite encountering fairly high recognition errors [39].”
BackgroundThe Metric Quality of Ordered Categorical Data
Srinivasan, V., & Basu, A. K. (1989) · Marketing Science · doi:10.1287/mksc.8.3.205 · Cited by 50 · cited 1×
“Existing literature suggests that moderately long scale lengths (i.e., more points) can increase both the reliability [23, 75] and the validity [68] of questionnaires.”
BackgroundTesting the Use of Voice Input in a Smartphone Web Survey
Revilla, M., Couper, M. P., Bosch, O. J., & Asensio, M. (2018) · Social Science Computer Review · doi:10.1177/0894439318810715 · Cited by 46
Wei, J., Tag, B., Trippas, J. R., Dingler, T., & Kostakos, V. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517432 · Cited by 46 · cited 1×
“Recent studies have further suggested that interactions with speakers can be error-prone and subject to network delays [58, 85].”
BackgroundSelf-citationIntegrating Cultural Relevance into a Behavioral mHealth Intervention for Native American Youth
Vigil-Hayes, M., Collier, A. F., Hagemann, S., Castillo, G., Mikkelson, K., Dingman, J., Muñoz, A., Luther, J., & McLaughlin, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449239 · Cited by 45 · cited 1×
“In the Computer-Supported Cooperative Work (CSCW) and Human-Computer Interaction (HCI) research community, surveys are widely used to acquire self-reported opinions from the crowd [11], gather data for intervention systems [82], collect frequent ESM self-reports of user sociometrics, such as creativity [79], mood, and…”
BackgroundLuo, Y., Lee, B., & Choe, E. K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376616 · Cited by 44 · cited 6×
“Regular check-ins with patients and the collection of self-reports (e.g., medical or mental condition and activities) require robust interactions [11, 20, 42].”
BackgroundInfluentialMeasuring Change in Personal Economic Well-Being
Rosenstone, S. J., Hansen, J. M., & Kinder, D. R. (1986) · Public Opinion Quarterly · doi:10.1086/268973 · Cited by 41
Evaluation of IVR data collection UIs for untrained rural users
Lerer, A. K., Ward, M., & Amarasinghe, S. P. (2010) · ACM Symposium on Computing for Development (ACM DEV) · doi:10.1145/1926180.1926183 · Cited by 39 · cited 2×
“In fact, IVR surveys are mostly advantageous as they can be easily distributed in regions where the access to smartphones and stable internet is limited, through regular phone calls [10, 40].”
BackgroundResultPredicting creativity in the wild: experience sample and sociometric modeling of teams
Tripathi, P., & Burleson, W. (2012) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2145204.2145386 · Cited by 36 · cited 1×
“In the Computer-Supported Cooperative Work (CSCW) and Human-Computer Interaction (HCI) research community, surveys are widely used to acquire self-reported opinions from the crowd [11], gather data for intervention systems [82], collect frequent ESM self-reports of user sociometrics, such as creativity [79], mood, and…”
BackgroundChung, J., Bleich, M., Wheeler, D. C., Winship, J. M., McDowell, B., Baker, D., & Parsons, P. (2021) · Gerontology and Geriatric Medicine · doi:10.1177/23337214211005869 · Cited by 35
Understanding User Perceptions of Proactive Smart Speakers
Wei, J., Dingler, T., & Kostakos, V. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3494965 · Cited by 33 · cited 7×
“In the end, we provide participants with twelve in-home scenarios proposed in previous studies [9, 74, 84] (see Table 2), and ask them to choose whether they prefer the voice or the web-browser in each scenario.”
BackgroundMethodInfluentialSelf-citationFrownOnError: Interrupting Responses from Smart Speakers by Facial Expressions
Yan, Y., Yu, C., Zheng, W., Tang, R., Xu, X., & Shi, Y. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376810 · Cited by 27 · cited 1×
“Currently, the smart speaker cannot receive user input when it is speaking [89].”
BackgroundMaharjan, R., Rohani, D. A., Bækgaard, P., Bardram, J., & Doherty, K. (2021) · CUI 2021 - 3rd Conference on Conversational User Interfaces · doi:10.1145/3469595.3469600 · Cited by 25 · cited 9×
“Maharjan et al. [45] used smart speakers to administer a 5-item health-related questionnaire and suggested that smart speakers could be a viable platform to administer surveys and support health and wellbeings.”
BackgroundResultInfluentialQiu, L., Kanski, B., Doerksen, S., Winkels, R., Schmitz, K. H., & Abdullah, S. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451827 · Cited by 23 · cited 1×
“Moreover, other than providing information to users, smart speakers are poised to become a practical tool for collecting user data via conversations [20, 25, 42, 61].”
Background"Hi! I am the Crowd Tasker" Crowdsourcing through Digital Voice Assistants
Hettiachchi, D., Sarsenbayeva, Z., Allison, F., van Berkel, N., Dingler, T., Marini, G., Kostakos, V., & Goncalves, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376320 · Cited by 22 · cited 1×
“Moreover, other than providing information to users, smart speakers are poised to become a practical tool for collecting user data via conversations [20, 25, 42, 61].”
BackgroundSelf-citationKaramad: A Voice-based Crowdsourcing Platform for Underserved Populations
Randhawa, S., Ahmad, T., Chen, J., & Raza, A. A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445417 · Cited by 22 · cited 8×
“Previous works on voice-based IVR systems have shed light on the use of voice surveys and provided some evidence on the reliability of MCQ questions and binary questions in voice surveys [63].”
BackgroundMethodResultInfluential"I can show what I really like.": Eliciting Preferences via Quadratic Voting
Cheng, T.-C., Li, T. W., Chou, Y.-H., Karahalios, K. G., & Sundaram, H. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449281 · Cited by 15 · cited 4×
“Close-ended questions, such as 5-point and 7-point Likert scales, are commonly used to measure levels of agreement or user satisfaction in CSCW [11].”
BackgroundMethodInfluentialChakraborty, D., Medhi, I., Cutrell, E. B., & Thies, W. (2013) · ACM Symposium on Computing for Development (ACM DEV) · doi:10.1145/2442882.2442891 · Cited by 14
Miller, E. T., Neal, D. J., Roberts, L. J., Boer, J. S., Cresskr, S. O., Metrik, J., & Marlatt, G. A. (2009) · American Psychological Association eBooks · doi:10.1037/11855-013 · Cited by 7
Research works printed but not linked: 22
- Duane F Alwin . 1992. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. Sociological methodology ( 1992 ), 83--118. Duane F Alwin. 1992. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. Sociological methodology (1992), 83--118.
- Richard L Clayton and Debbie LS Winter . 1992 . Speech data entry: results of a test of voice recognition for survey data collection . JOURNAL OF OFFICIAL STATISTICS-STOCKHOLM- 8 (1992), 377 -- 377 . Richard L Clayton and Debbie LS Winter. 1992. Speech data entry: results of a test of voice recognition for survey data collection. JOURNAL OF OFFICIAL STATISTICS-STOCKHOLM- 8 (1992), 377--377.
- Don C Des Jarlais , Denise Paone , Judith Milliken , Charles F Turner , Heather Miller , James Gribble , Qiuhu Shi , Holly Hagan , and Samuel R Friedman . 1999. Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. The Lancet 353, 9165 ( 1999 ), 1657--1661. Don C Des Jarlais, Denise Paone, Judith Milliken, Charles F Turner, Heather Miller, James Gribble, Qiuhu Shi, Holly Hagan, and Samuel R Friedman. 1999. Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. The Lancet 353, 9165 (1999), 1657--1661.
- Ed Diener , Derrick Wirtz , Robert Biswas-Diener , William Tov , Chu Kim-Prieto , Dong-won Choi, and Shigehiro Oishi . 2009. New measures of well-being . In Assessing well-being . Springer , 247--266. Ed Diener, Derrick Wirtz, Robert Biswas-Diener, William Tov, Chu Kim-Prieto, Dong-won Choi, and Shigehiro Oishi. 2009. New measures of well-being. In Assessing well-being. Springer, 247--266.
- Don A Dillman and Leah Melani Christian . 2005. Survey mode as a source of instability in responses across surveys. Field methods 17, 1 ( 2005 ), 30--52. Don A Dillman and Leah Melani Christian. 2005. Survey mode as a source of instability in responses across surveys. Field methods 17, 1 (2005), 30--52.
- Don A Dillman , Glenn Phelps , Robert Tortora , Karen Swift , Julie Kohrell , Jodi Berck , and Benjamin L Messer . 2009. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social science research 38, 1 ( 2009 ), 1--18. Don A Dillman, Glenn Phelps, Robert Tortora, Karen Swift, Julie Kohrell, Jodi Berck, and Benjamin L Messer. 2009. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social science research 38, 1 (2009), 1--18.
- Khalil G Ghanem , Heidi E Hutton , Jonathan M Zenilman , Rebecca Zimba , and Emily J Erbelding . 2005. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sexually transmitted infections 81, 5 ( 2005 ), 421--425. Khalil G Ghanem, Heidi E Hutton, Jonathan M Zenilman, Rebecca Zimba, and Emily J Erbelding. 2005. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sexually transmitted infections 81, 5 (2005), 421--425.
- Katharina Graben , Bettina K Doering , Franziska Jeromin , and Antonia Barke . 2020. Problematic mobile phone use: Validity and reliability of the Problematic Use of Mobile Phone (PUMP) Scale in a German sample. Addictive behaviors reports 12 ( 2020 ), 100297. Katharina Graben, Bettina K Doering, Franziska Jeromin, and Antonia Barke. 2020. Problematic mobile phone use: Validity and reliability of the Problematic Use of Mobile Phone (PUMP) Scale in a German sample. Addictive behaviors reports 12 (2020), 100297.
- Jiepu Jiang , Wei Jeng , and Daqing He . 2013 . How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search . In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. Association for Computing Machinery , New York, NY, USA, 143--152. Jiepu Jiang, Wei Jeng, and Daqing He. 2013. How do users respond to voice input errors? Lexical and phonetic query reformulation in voice search. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. Association for Computing Machinery, New York, NY, USA, 143--152.
- Aman Khullar , Priyadarshi Hitesh , Shoaib Rahman , Deepak Kumar , Rachit Pandey , Praveen Kumar , Rajeshwari Tripathi , Prince Prince , Ankit Akash Jha , Himanshu Himanshu, et al. 2021 . Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response Systems. In ACM SIGCAS Conference on Computing and Sustainable Societies. Association for Computing Machinery , New York, NY, USA, 288--298. Aman Khullar, Priyadarshi Hitesh, Shoaib Rahman, Deepak Kumar, Rachit Pandey, Praveen Kumar, Rajeshwari Tripathi, Prince Prince, Ankit Akash Jha, Himanshu Himanshu, et al. 2021. Costs and Benefits of Conducting Voice-based Surveys Versus Keypress-based Surveys on Interactive Voice Response Systems. In ACM SIGCAS Conference on Computing and Sustainable Societies. Association for Computing Machinery, New York, NY, USA, 288--298.
- Bret Kinsella . 2019. Loup Ventures says 75% of U.S. households will have smart speakers by 2025 , Google to surpass Amazon in market share. https://voicebot.ai/2019/06/18/loup-ventures-says-75-of-u-s-households-will-have-smartspeakers-by-2025-google-to-surpass-amazon-in-market-share/ Bret Kinsella. 2019. Loup Ventures says 75% of U.S. households will have smart speakers by 2025, Google to surpass Amazon in market share. https://voicebot.ai/2019/06/18/loup-ventures-says-75-of-u-s-households-will-have-smartspeakers-by-2025-google-to-surpass-amazon-in-market-share/
- Jon A Krosnick . 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology 5, 3 ( 1991 ), 213--236. Jon A Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology 5, 3 (1991), 213--236.
- Jon A Krosnick and Matthew K Berent . 1993. Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science ( 1993 ), 941--964. Jon A Krosnick and Matthew K Berent. 1993. Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science (1993), 941--964.
- Jon A Krosnick , Sowmya Narayan , and Wendy R Smith . 1996. Satisficing in surveys: Initial evidence. New directions for evaluation 1996 , 70 (1996), 29--44. Jon A Krosnick, Sowmya Narayan, and Wendy R Smith. 1996. Satisficing in surveys: Initial evidence. New directions for evaluation 1996, 70 (1996), 29--44.
- J Richard Landis and Gary G Koch . 1977. The measurement of observer agreement for categorical data. biometrics ( 1977 ), 159--174. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159--174.
- Rensis Likert . 1932. A technique for the measurement of attitudes. Archives of psychology ( 1932 ). Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology (1932).
- Raju Maharjan , Per Bækgaard , and Jakob E Bardram . 2019. " Hear me out" smart speaker based conversational agent to monitor symptoms in mental health. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers . Association for Computing Machinery , New York, NY, USA , 929--933. Raju Maharjan, Per Bækgaard, and Jakob E Bardram. 2019. " Hear me out" smart speaker based conversational agent to monitor symptoms in mental health. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA, 929--933.
- Lisa J Merlo , Amanda M Stone , and Alex Bibbey . 2013. Measuring problematic mobile phone use: development and preliminary psychometric properties of the PUMP scale. Journal of addiction 2013 ( 2013 ). Lisa J Merlo, Amanda M Stone, and Alex Bibbey. 2013. Measuring problematic mobile phone use: development and preliminary psychometric properties of the PUMP scale. Journal of addiction 2013 (2013).
- Charles Egerton Osgood , George J Suci , and Percy H Tannenbaum . 1957. The measurement of meaning. Number 47 . University of Illinois press. Charles Egerton Osgood, George J Suci, and Percy H Tannenbaum. 1957. The measurement of meaning. Number 47. University of Illinois press.
- Alisha Pradhan , Leah Findlater , and Amanda Lazar . 2019 . " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults . Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--21. Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--21.
- Roger Tourangeau and Kenneth A Rasinski . 1988. Cognitive processes underlying context effects in attitude measurement. Psychological bulletin 103, 3 ( 1988 ), 299. Roger Tourangeau and Kenneth A Rasinski. 1988. Cognitive processes underlying context effects in attitude measurement. Psychological bulletin 103, 3 (1988), 299.
- Charles F Turner , Leighton Ku , Susan M Rogers , Laura D Lindberg , Joseph H Pleck , and Freya L Sonenstein . 1998. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 280, 5365 ( 1998 ), 867--873. Charles F Turner, Leighton Ku, Susan M Rogers, Laura D Lindberg, Joseph H Pleck, and Freya L Sonenstein. 1998. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 280, 5365 (1998), 867--873.
Cited by
Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
Wei, J., Kim, S., Jung, H., & Kim, Y.-H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637364 · Cited by 84
“As such, compared to rule-based chatbots that can almost 100% ask pre-defined scripts [97], we can see that not all specified information slots were asked by our chatbots during the study.”
BackgroundSelf-citationLim, J., Koh, Y., Kim, A., & Lee, U. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642846 · Cited by 38
Designing the Conversational Agent: Asking Follow-up Questions for Information Elicitation
Hu, J., Guo, J., Tang, N., Ma, X., Yao, Y., Yang, C., & Xu, Y. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637320 · Cited by 19
“The user study in this work involved human participants as the previous works on the human-CA interaction [13, 39, 67], which extended the findings’ generalizability compared to the work that only evaluated the follow-up question generation results with automatic metrics [62].”
BackgroundMethodResultInfluentialSpeaking of accent: A content analysis of accent misconceptions in ASR research
Prinos, K., Patwari, N., & Power, C. A. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658969 · Cited by 12
Fusang: Graph-inspired Robust and Accurate Object Recognition on Commodity mmWave Devices
He, G., Chen, S., Xu, D., Chen, X., Xie, Y., Wang, X., & Fang, D. (2023) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3581791.3596849 · Cited by 14
Metadata
- Authors
- Wei, J., Jiang, W., Wang, C., Yu, D., Goncalves, J., Dingler, T., & Kostakos, V.
- Affiliations
- The University of Melbourne (AU)
- University of Melbourne, Melbourne, VIC, Australia
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2022-11-07
- Volume
- 6
- Issue
- CSCW2
- Pages
- 1-32
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4309619903
- Cited by
- 9
- References
- 65
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 3
- FWCI
- 3.24
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- Funders
- National Health and Medical Research Council (1170937, 2004316)
- Australian Research Council (DP190102627)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- smart speakers, speech interfaces, questionnaire development, test-retest reliability, reaction time (psychology), scale types, user experience, self-reported health data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.