Just ask Siri? A pilot study comparing smartphone digital assistants and laptop Google searches for smoking cessation advice
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To compare voice-activated internet searches by smartphone (two digital assistants) with laptop ones for information and advice related to smoking cessation. DESIGN: Responses to 80 questions on a range of topics related to smoking cessation (including the FAQ from a NHS website), compared for quality. SETTING: Smartphone and internet searches as performed in New Zealand. MAIN OUTCOME MEASURES: Ranked responses to the questions. RESULTS: Google laptop internet searches came first (or first equal) for best quality smoking cessation advice for 83% (66/80) of the responses. Voiced questions to Google Assistant ("OK Google") came first/first equal 76% of the time vs Siri (Apple) at 28%. Google and Google Assistant were statistically significantly better than Siri searches (odds ratio 12.4 and 8.5 respectively, p<0.0001 in each comparison). When asked FAQs from the National Health Service website, or to find information the Centers for Disease Control has made videos on, the best search results used expert sources 59% (31/52) of the time, "some expertise" (eg, Wikipedia) 18% of the time, but also magazines and other low quality sources 19% of the time. Using all three methods failed to find relevant information 8% (6/80) of the time, with Siri having the most failed responses (53% of the time). CONCLUSION: Google internet searches and Google Assistant were found to be significantly superior to the Siri digital assistant for smoking cessation information. While expert content was returned over half the time, there is still substantial room for improvement in how these software systems deliver smoking cessation advice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“A total of 4 papers [61,66,68,70] did not explicitly specify the study design they used, whereas the other papers provided labels.”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [61]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One study [61] observed that the standard Google Search performs better than a voice-activated internet search performed with Google Assistant and Apple Siri.”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A total of 2 studies [61,70] qualitatively assessed the accuracy of the VCAs.”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Half (n=6) of the included studies [61,63,64,68-70] evaluated the accuracy of the system.”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of the included studies (n=8) also did not report controlling for participants’ previous experience with technology [59-63,66,69,70].”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [61]
Experimental — accuracy not yet measured
15 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Miner, A. S., Milstein, A., Schueller, S., Hegde, R., Mangurian, C., & Linos, E. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.0400 · Cited by 368
Higgins, O., Sixsmith, J., Barry, M. M., & Domegan, C. T. (2011) · Arrow@dit (Dublin Institute of Technology) · doi:10.2900/5788 · Cited by 172
Cited by
Bérubé, C., Schachner, T., Keller, R., Fleisch, E., v Wangenheim, F., Barata, F., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/25933 · Cited by 133
“Half of the studies (n=6) did not report on any system measures [59,60,62,65-67], whereas the other half reported either speech recognition performance measures (n=4) [63,64,68,69] or criterion-based evaluation of the goodness of the VCA’s response (n=2) [61,70].”
InfluentialCleofas, J. V., & Rocha, I. C. N. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10529-9 · Cited by 128
Sezgin, E., Militello, L. K., Huang, Y., & Lin, S. (2020) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz141 · Cited by 83
Kocaballi, A. B., Quiroz, J. C., Rezazadegan, D., Berkovsky, S., Magrabi, F., Coiera, E., & Laranjo, L. (2020) · Journal of Medical Internet Research · doi:10.2196/15823 · Cited by 73
“These studies focused on queries around physical health, mental health, personal violence [13], general health, medication, emergency health [14], and smoking cessation [15], having highlighted the inability of these CAs to respond in an appropriate manner.”
BackgroundResultDing, H., Simmich, J., Vaezipour, A., Andrews, N. E., & Russell, T. G. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad222 · Cited by 73
Nobles, A. L., Leas, E. C., Caputi, T. L., Zhu, S.-H., Strathdee, S. A., & Ayers, J. W. (2020) · npj Digital Medicine · doi:10.1038/s41746-019-0215-9 · Cited by 67
Alagha, E. C., & Helbing, R. R. (2019) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100075 · Cited by 64
Palanica, A., Thommandram, A., Lee, A., Li, M., & Fossat, Y. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0133-x · Cited by 63
Conversational AI and Vaccine Communication: Systematic Review of the Evidence
Passanante, A., Pertwee, E., Lin, L. K., Lee, K. Y., Wu, J. T. K., & Larson, H. J. (2023) · Journal of Medical Internet Research · doi:10.2196/42758 · Cited by 59
Ferrand, J. L., Hockensmith, R., Houghton, R. F., & Buhi, E. R. (2020) · Journal of Medical Internet Research · doi:10.2196/19018 · Cited by 51
Crane, D., Ubhi, H. K., Brown, J., & West, R. (2019) · F1000Research · doi:10.12688/f1000research.16148.2 · Cited by 39
Crane, D., Ubhi, H. K., Brown, J., & West, R. (2018) · F1000Research · doi:10.12688/f1000research.16148.1 · Cited by 33
Palanica, A., & Fossat, Y. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.669971 · Cited by 31
Kannampallil, T., Ajilore, O., Lv, N., Smyth, J. M., Wittels, N. E., Ronneberg, C. R., Kumar, V., Xiao, L., Dosala, S., Barve, A., et al. (2023) · Translational Psychiatry · doi:10.1038/s41398-023-02462-x · Cited by 31
Language Use in Conversational Agent–Based Health Communication: Systematic Review
Shan, Y., Ji, M., Xie, W., Qian, X., Li, R., Zhang, X., & Hao, T. (2022) · Journal of Medical Internet Research · doi:10.2196/37403 · Cited by 30
Kannampallil, T., Ronneberg, C. R., Wittels, N. E., Kumar, V., Lv, N., Smyth, J. M., Gerber, B. S., Kringle, E. A., Johnson, J. A., Yu, P. S., et al. (2022) · JMIR Formative Research · doi:10.2196/38092 · Cited by 26
A Brief Taxonomy of Hybrid Intelligence
Pescetelli, N. (2021) · Forecasting · doi:10.3390/forecast3030039 · Cited by 24
Eagle, T., Blau, C., Bales, S., Desai, N., Li, V., & Whittaker, S. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3488057 · Cited by 22
Bickmore, T., Ólafsson, S., & O’Leary, T. K. (2021) · Journal of Medical Internet Research · doi:10.2196/30704 · Cited by 21
Pediatrician Attitudes toward Digital Voice Assistant Technology Use in Clinical Practice
Wilder, J. L., Nadar, D., Gujral, N., Ortiz, B. D., Stevens, R., Holder-Niles, F. F., Lee, J., & Gaffin, J. M. (2019) · Applied Clinical Informatics · doi:10.1055/s-0039-1687863 · Cited by 20
Epstein, R., Lee, V., Mohr, R., & Zankich, V. R. (2022) · PLoS ONE · doi:10.1371/journal.pone.0268081 · Cited by 19
Picard, C. T., Smith, K. E., Picard, K., & Douma, M. J. (2020) · BMJ Innovations · doi:10.1136/bmjinnov-2018-000326 · Cited by 19
Evaluation of COVID-19 Information Provided by Digital Voice Assistants
Goh, A. S. Y., Wong, L. L., & Yap, K. Y. (2021) · International Journal of Digital Health · doi:10.29337/ijdh.25 · Cited by 19
A Survey of Conversational Agents and Their Applications for Self-Management of Chronic Conditions
Park, M. S., Upama, P. B., Anik, A. A., Ahamed, S. I., Luo, J., Tian, S., Rabbani, M., & Oh, H. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00162 · Cited by 18
Xiao, Z., Lee, J., Zeng, L., & Ni, L. (2020) · Psychology Health & Medicine · doi:10.1080/13548506.2020.1728348 · Cited by 17
Hills, O., & Shah, D. V. (2020) · Patient Education and Counseling · doi:10.1016/j.pec.2020.06.006 · Cited by 15
Bérubé, C., Kovacs, Z. F., Fleisch, E., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/32161 · Cited by 14
“The source type was categorized based on the work of Alagha and Helbing [29] and Boyd and Wilson [27] into expert, commercial, crowdsourced, or not stated.”
BackgroundMethodResultInfluentialMancone, S., Diotaiuti, P., Valente, G., Corrado, S., Bellizzi, F., Vilarino, G. T., & Andrade, A. (2023) · Behavioral Sciences · doi:10.3390/bs13070550 · Cited by 14
Why Do You Trust Siri?: The Factors Affecting Trustworthiness of Intelligent Personal Assistant
Wu, D., & Huang, Y. (2021) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.464 · Cited by 14
Joorabchi, A., Doherty, C., & Dawson, J. (2019) · Health Informatics Journal · doi:10.1177/1460458219892711 · Cited by 11
Threshy: supporting safe usage of intelligent web services
Cummaudo, A., Barnett, S., Vasa, R., & Grundy, J. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417919 · Cited by 8
Towards Profile and Domain Modelling in Agent-Based Applications for Behavior Change
Calbimonte, J.-P., Calvaresi, D., Dubosson, F., & Schumacher, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-24209-1_2 · Cited by 7
Voice Switching in Voice-Enabled Digital Assistants (VDAs)
Bilal, D., & Barfield, J. K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78462-1_39 · Cited by 6
Voice-based conversational agents for sensing and support: Examples from academia and industry
Bérubé, C., & Fleisch, E. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-90045-4.00017-4 · Cited by 5
Cabansag, J. N. (2022) · World Journal of English Language · doi:10.5430/wjel.v12n7p265 · Cited by 4
Frehmann, K., Ziegele, M., & Rosar, U. (2022) · Studies in Communication and Media · doi:10.5771/2192-4007-2022-2-278 · Cited by 3
Govier, E. (2018) · Journal of Posthuman Studies · doi:10.5325/jpoststud.2.2.0003 · Cited by 2
Radford, M. L., Floegel, D., Barriage, S., & Houli, D. (2019) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 2
Meryl, K. O. H., Xie, Q., Wong, L., & Yap, K. Y. (2021) · Minerva Psychiatry · doi:10.23736/s2724-6612.20.02073-7 · Cited by 1
Antwortverhalten von deutschsprachigen Sprachassistenten auf Gesundheitsfragen
Sander, L., Kühn, C., Bengel, J., & Krämer, L. V. (2019) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-019-02979-x · Cited by 1
Graduate and postgraduate education at a crossroads
Miller, S. M., Moos, W. H., Munk, B. H., Munk, S. A., Hart, C. P., & Spellmeyer, D. C. (2023) · Elsevier eBooks · doi:10.1016/b978-0-12-824304-6.00009-2
Makolkina, M., Shypota, N., & Koucheryavy, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66471-8_19
'Siri, should I take this pill?' — Assessing Which AI-driven Assistant is Best
Lessing, C., McKay, M., Foster, S., & Johnston, D. (2026) · Tuwhera (Auckland University of Technology)
(2019) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
المساعد الصوتي الرقمي: مراجعة علمية
حطب, آ. أ. (2024) · المجلة الدولية لعلوم المكتبات و المعلومات · doi:10.21608/ijlis.2024.347752
Metadata
- Authors
- Boyd, M., & Wilson, N.
- Affiliations
- University of Otago (NZ)
- Corresponding author
- Boyd, Matt
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-03-28
- Volume
- 13
- Issue
- 3
- Pages
- e0194811
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5874038
- OpenAlex
- W2794532037
- Cited by
- 40
- References
- 2
- Citations by year
- 2023: 6 · 2022: 6 · 2021: 10 · 2020: 7 · 2019: 9 · 2018: 2
- FWCI
- 11.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Smoking Behavior and Cessation
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Keywords
- smoking cessation, virtual assistants, Google Assistant, Siri, health information quality, search result quality, expert sources, health information seeking
- MeSH terms
- Smartphone, Humans, New Zealand, Pilot Projects, Therapy, Computer-Assisted, Smoking Cessation, Internet, Information Seeking Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.