Evolving Health Information–Seeking Behavior in the Context of Google AI Overviews, ChatGPT, and Alexa: Interview Study Using the Think-Aloud Protocol
Open accessPublished version of a preprint
The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index; 14 of the 44 sources its claims cite were checked for support.
Abstract
We conducted in-depth qualitative research with 27 participants (ages 19-80 years) using a think-aloud protocol. Participants searched for health information across 3 platforms—Google, ChatGPT, and Alexa—while verbalizing their thought processes. Participants integrated AI tools into their broader search routines rather than using them in isolation. ChatGPT was valued for its clarity, speed, and ability to offer broad overviews, but participants often questioned its source transparency and accuracy.
Overview
CoChat Summary
TL;DR
This qualitative think-aloud interview study with 27 participants found that people do not simply “trust” or “reject” AI health tools; instead, they mix Google AI Overviews, ChatGPT, and Alexa into broader search routines depending on context, urgency, and perceived usefulness. Across platforms, participants valued speed and summarization but were wary of missing sources, bias, and limited or inaccessible responses, especially for more serious health questions.
Key findings
- Of 27 participants, 23 (85%) reported favorably on some aspect of ChatGPT’s formatting or efficiency, while 14/27 already used it as part of their search routine.
- Google’s AI Overviews were often treated cautiously: about one-third of participants commented on them, 5/27 explicitly skipped them, and no participant reported stopping their search at the overview.
- Alexa was seen as useful mainly for low-stakes tasks: 17/27 (63%) said they would not typically use it for the study questions, while 8/27 (30%) said short voice responses were convenient for simple questions.
- Utility often outweighed trust: 15/27 (56%) said they would still use AI-generated information if it helped direct further investigation, and participants frequently described AI tools as “starting points” rather than final authorities.
Methods
- Qualitative interview study using a think-aloud protocol with 27 participants aged 19-80 years, conducted in summer and fall 2024.
- Participants searched for two health prompts on three platforms: Google, ChatGPT 3.5, and Alexa; one prompt was a standardized St. John’s wort question and the other was a personally relevant health query.
- Transcripts (370 pages) were analyzed by the three researchers using reflexive thematic analysis, with 34 codes developed and organized around two research questions.
Limitations
- The paper states the sample was small, recruited through loose personal-network connections, and not longitudinal; it warns the findings cannot be generalized to a wider population.
- The think-aloud protocol may have produced a Hawthorne effect, with participants changing behavior because they knew they were being observed.
- The paper notes that the technologies changed during data collection, so findings may quickly become outdated; this is an evident limitation given platform updates and evolving AI Overviews/ChatGPT source-linking behavior.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 2 Not valid · 17 Not checked · 109 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“More recent perspectives discuss how trust is iterative in online health contexts, but AI tools seem to be making this overlap between exploration and evaluation central to the search process [42].”
§ Evolving Health Information–Seeking Behavior in the Context of Google AI Overviews, ChatGPT, and Alexa: Interview Study Using the Think-Aloud Protocol > Discussion > Principal Findings
What the sources say
- Could not verifycites [42] sourceExperimental — accuracy not yet measured
The Background and Principal Findings discuss trust judgments and corroboration during online health-information searches. I searched for AI tools, iterative trust, and overlap between exploration and evaluation, but found no passage addressing whether AI makes that overlap central to the search process. (LLM, conf=0.75)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Second, our use of the think-aloud protocol to observe search behavior has the potential for producing the Hawthorne effect [63], in which participants change their behavior because they know they are being observed.”
§ Evolving Health Information–Seeking Behavior in the Context of Google AI Overviews, ChatGPT, and Alexa: Interview Study Using the Think-Aloud Protocol > Discussion > Limitations This study had several important limitations. First, our sample was small and generated via loose connections in personal networks. Second, our use of the think-aloud protocol to observe search behavior has the potential for producing the Hawthorne effect [63], in which participants change their behavior because they know they are being observed. Finally, this study was not longitudinal. It provides a snapshot of experiences with AI- and voice-assisted technologies, with a small sample of people, at one particular point in time. Notably, 10 of the 27 (37%) participants had never previously used a stand-alone LLM; therefore, we were able to observe their first impressions of using ChatGPT rather than discuss existing experiences or attitudes. Unfortunately, undertaking research on these types of technologies means that results quickly become out of date. In the context of this research, the functionality of Google’s AI Overviews and ChatGPT changed during the short time frame of the study. Google’s AI Overviews now appear in response to more search queries and are longer in length compared to the period when the interviews were being conducted. Additionally, both Google Overviews and ChatGPT now link out to sources. A majority of the participants had noted the absence of sources in both tools as a serious drawback, which suggests that interviews conducted today would likely sound very different in relation to this issue. While we are describing this issue as a limitation of our research, we also, as a wider field, need to grapple with the challenges of conducting research on technologies that are evolving at such speed. Authors need to focus on which of their findings are likely to withstand platform design changes and model updates and which will not. More research is also needed to explore whether users understand these design changes.
What the sources say
- Could not verifycites [63] sourceExperimental — accuracy not yet measured
The Materials and methods section describes clinical scenarios entered into ChatGPT and graded by physicians; the Discussion addresses limitations of that evaluation. Neither discusses a think-aloud protocol, observed search behavior, or the Hawthorne effect (searched those terms and the claim's description of behavior changing under observation). Our copy therefore does not verify the point attributed to this source. (LLM, conf=0.76)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, 74% (n=20) of the sample had at least an undergraduate degree”
§ Evolving Health Information–Seeking Behavior in the Context of Google AI Overviews, ChatGPT, and Alexa: Interview Study Using the Think-Aloud Protocol > Methods > Recruitment
plainn = 20- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants also appreciated that the platform allowed them to ask questions more precisely and that the conversational flow of the platform was responsive to how they worded their searches:”
§ Evolving Health Information–Seeking Behavior in the Context of Google AI Overviews, ChatGPT, and Alexa: Interview Study Using the Think-Aloud Protocol > Results > Observations Specifically Related to the 3 Technologies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants in this study selectively engaged with AI- and voice-assisted tools based on perceived usefulness, not just trustworthiness”
§ Evolving Health Information–Seeking Behavior in the Context of Google AI Overviews, ChatGPT, and Alexa: Interview Study Using the Think-Aloud Protocol > Abstract > Conclusions
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
76 more claims
Create a free account to see all 139 claims and to verify the 8 references still open.
- Probably not a substantial claim. 36
- It may state two separate claims as one. 29
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245 · cited 1×
“There is significant research on the ways in which algorithms produce search results that mirror people’s worldviews, but the majority of that work has been carried out on social networks [83,84].”
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921
McCambridge, J., Witton, J., & Elbourne, D. R. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.08.015 · Cited by 2,748
The Persuasiveness of Source Credibility: A Critical Review of Five Decades' Evidence
Pornpitakpan, C. (2004) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2004.tb02547.x · Cited by 2,205
Filter Bubbles, Echo Chambers, and Online News Consumption
Flaxman, S., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw006 · Cited by 1,992 · cited 1×
“There is significant research on the ways in which algorithms produce search results that mirror people’s worldviews, but the majority of that work has been carried out on social networks [83,84].”
Eysenbach, G. (2002) · BMJ · doi:10.1136/bmj.324.7337.573 · Cited by 1,915 · cited 1×
“For over a decade, surveys have demonstrated that search engines are the most common platform for online information seeking [2], and this is supported in studies with a range of people, from everyday users [3,4] to patients with chronic health conditions [5-7].”
Hesse, B. W., Nelson, D. E., Kreps, G. L., Croyle, R. T., Arora, N. K., Rimer, B. K., & Viswanath, K. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.22.2618 · Cited by 1,523 · cited 1×
“Existing models emphasize credibility, trustworthiness, and usefulness as primary drivers of HISB [75-77].”
Social and Heuristic Approaches to Credibility Evaluation Online
Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010) · Journal of Communication · doi:10.1111/j.1460-2466.2010.01488.x · Cited by 1,394
Models of ecological rationality: The recognition heuristic.
Goldstein, D. G., & Gigerenzer, G. (2002) · Psychological Review · doi:10.1037/0033-295x.109.1.75 · Cited by 1,352 · cited 1×
“Psychologists have described this phenomenon as the illusory truth effect, explaining that people are more likely to believe information they have heard before, even if that information is false [78-80].”
Lumsdaine, A. A. (1954) · Educational Technology Research and Development · doi:10.1007/bf02713272 · Cited by 1,081
Trust in AI and Its Role in the Acceptance of AI Technologies
Choung, H., David, P., & Ross, A. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2050543 · Cited by 963
Kontos, E., Blake, K. D., Chou, W.-Y. S., & Prestin, A. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3117 · Cited by 917 · cited 1×
“In addition, we oversampled people with university-level education due to research that has demonstrated people with lower levels of education are less likely to go online to seek health information [10,72].”
(2012) · American Psychological Association eBooks · doi:10.1037/13620-000 · Cited by 889
Schaefer, K. E., Chen, J. Y. C., Szalma, J. L., & Hancock, P. A. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816634228 · Cited by 768
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715
How do patients evaluate and make use of online health information?
Sillence, E., Briggs, P., Harris, P. R., & Fishwick, L. (2007) · Social Science & Medicine · doi:10.1016/j.socscimed.2007.01.012 · Cited by 676
Begg, I. M., Anas, A., & Farinacci, S. (1992) · Journal of Experimental Psychology: General · doi:10.1037//0096-3445.121.4.446 · Cited by 650
Jacobs, W., Amuta, A. O., & Jeon, K. C. (2017) · Cogent Social Sciences · doi:10.1080/23311886.2017.1302785 · Cited by 635 · cited 1×
“Furthermore, the design of voice assistants can inadvertently perpetuate inequalities in information access, which we know has been a serious problem with all forms of new technological adoption [12].”
Knowledge does not protect against illusory truth.
Fazio, L. K., Brashier, N. M., Payne, B. K., & Marsh, E. J. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000098 · Cited by 598 · cited 1×
“Psychologists have described this phenomenon as the illusory truth effect, explaining that people are more likely to believe information they have heard before, even if that information is false [78-80].”
How do users evaluate the credibility of Web sites?: a study with over 2,500 participants
Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003) · Proceedings of the 2003 conference on Designing for user experiences · doi:10.1145/997078.997097 · Cited by 565
Online Health Information Seeking Behavior: A Systematic Review
Jia, X., Pang, Y., & Liu, L. S. (2021) · Healthcare · doi:10.3390/healthcare9121740 · Cited by 547 · cited 2×
“Two recent systematic reviews of online HISB [26,27] have demonstrated key influencing factors, including age, sex, income, employment status, literacy (or education) level, country of origin, place of residence, and caregiving role.”
Trust and Credibility in Web-Based Health Information: A Review and Agenda for Future Research
Sbaffi, L., & Rowley, J. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7579 · Cited by 537
Powell, J., Inglis, N., Ronnie, J., & Large, S. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1600 · Cited by 493 · cited 1×
“A reliance on search engines is particularly the case for populations less able to access a primary care provider [8-15].”
Finney Rutten, L. J., Blake, K. D., Greenberg-Worisek, A. J., Allen, S. V., Moser, R. P., & Hesse, B. W. (2019) · Public Health Reports® · doi:10.1177/0033354919874074 · Cited by 426
Effects of Online Health Sources on Credibility and Behavioral Intentions
Hu, Y., & Shyam Sundar, S. (2010) · Communication Research · doi:10.1177/0093650209351512 · Cited by 413 · cited 1×
“The source credibility theory [35] has been applied to studies comparing doctors to online health sources [36], credibility judgments on social media [37], and user-generated content related to health [38], demonstrating how people combine considerations around trustworthiness and perceived…”
Online Health Information Seeking: A Review and Meta-Analysis
Wang, X., Shi, J., & Kong, H. (2021) · Health Communication · doi:10.1080/10410236.2020.1748829 · Cited by 370 · cited 2×
“Two recent systematic reviews of online HISB [26,27] have demonstrated key influencing factors, including age, sex, income, employment status, literacy (or education) level, country of origin, place of residence, and caregiving role.”
Algorithmic Colonization of Africa
Birhane, A. (2020) · SCRIPTed: A Journal of Law, Technology & Society · doi:10.2966/scrip.170220.389 · Cited by 349 · cited 1×
“Studies have documented the challenges of using US-owned technology for people who speak languages other than English [89-91], but the adoption of LLMs raises further questions.”
Who Searches the Internet for Health Information?
Bundorf, M. K., Wagner, T. H., Singer, S. J., & Baker, L. C. (2006) · Health Services Research · doi:10.1111/j.1475-6773.2006.00510.x · Cited by 313
Talk to me: Exploring user interactions with the Amazon Alexa
Lopatovska, I., Rink, K., Knight, I., Raines, K., Cosenza, K., Williams, H., Sorsche, P., Hirsch, D., Li, Q., & Martinez, A. (2019) · Journal of Librarianship and Information Science · doi:10.1177/0961000618759414 · Cited by 298
Lee, K., Hoti, K., Hughes, J. D., & Emmerton, L. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3706 · Cited by 284 · cited 1×
“For over a decade, surveys have demonstrated that search engines are the most common platform for online information seeking [2], and this is supported in studies with a range of people, from everyday users [3,4] to patients with chronic health conditions [5-7].”
Using cognitive interviews and think-aloud protocols to understand thought processes
Wolcott, M. D., & Lobczowski, N. G. (2021) · Currents in Pharmacy Teaching and Learning · doi:10.1016/j.cptl.2020.09.005 · Cited by 256 · cited 1×
“This method has shown to be useful in usability test settings in order to understand people’s behavior with particular technologies [68,69], including a previous iteration of a study observing and listening to people search for health information online [70].”
Rains, S. A. (2007) · Journal of Health Communication · doi:10.1080/10810730701619992 · Cited by 236
Searching for Health Information: The Cancer Information Service Model
Freimuth, V. S., Stein, J. A., & Kean, T. J. (1989) · University of Pennsylvania Press eBooks · doi:10.9783/9781512816075 · Cited by 214 · cited 3×
“For example, the 6 elements of HISB outlined by Freimuth et al [25] in 1989 appeared in all of the 27 conversations.”
InfluentialIdentifying Credible Sources of Health Information in Social Media: Principles and Attributes
Kington, R. S., Arnesen, S., Chou, W.-Y. S., Curry, S. J., Lazer, D., & Villarruel, A. M. (2021) · NAM Perspectives · doi:10.31478/202107a · Cited by 210 · cited 1×
“…source credibility theory [35] has been applied to studies comparing doctors to online health sources [36], credibility judgments on social media [37], and user-generated content related to health [38], demonstrating how people combine considerations around trustworthiness and perceived…”
Consumer Use of “Dr Google”: A Survey on Health Information-Seeking Behaviors and Navigational Needs
Lee, K., Hoti, K., Hughes, J. D., & Emmerton, L. M. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4345 · Cited by 199
Health Information Technology Use Among Adults: United States, July-December 2022
Wang, X., & Cohen, R. A. (2023) · doi:10.15620/cdc:133700 · Cited by 152 · cited 1×
“According to the Centers for Disease Control and Prevention’s National Center for Health Statistics, 58.5% of people in the United States reported searching the internet for health information in the previous 12 months [1].”
InfluentialSeitz, L., Bekmeier‐Feuerhahn, S., & Gohil, K. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102848 · Cited by 142
Harris, P. R., Sillence, E., & Briggs, P. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1821 · Cited by 140
People Overtrust AI-Generated Medical Advice despite Low Accuracy
Shekar, S., Pataranutaporn, P., Sarabu, C. R., Cecchi, G., & Maes, P. (2025) · NEJM AI · doi:10.1056/aioa2300015 · Cited by 137
Ma, T. (., & Atkin, D. J. (2016) · Telematics and Informatics · doi:10.1016/j.tele.2016.09.009 · Cited by 135 · cited 1×
“…source credibility theory [35] has been applied to studies comparing doctors to online health sources [36], credibility judgments on social media [37], and user-generated content related to health [38], demonstrating how people combine considerations around trustworthiness and perceived expertise.”
Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking
Sharma, N., Liao, Q. V., & Xiao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642459 · Cited by 132
Cooke, L. (2010) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2010.2052859 · Cited by 128 · cited 1×
“This method has shown to be useful in usability test settings in order to understand people’s behavior with particular technologies [68,69], including a previous iteration of a study observing and listening to people search for health information online [70].”
Ayre, J., Mac, O., McCaffery, K., McKay, B. R., Liu, M., Shi, Y., Rezwan, A., & Dunn, A. G. (2024) · Journal of General Internal Medicine · doi:10.1007/s11606-023-08469-w · Cited by 124 · cited 1×
“While comprehensive research into the use of ChatGPT or other LLMs to seek out health information is not yet available, some scholars have argued that the technology has the potential to not only provide information in accessible formats but also to simplify complicated advice [62].”
Voice-based assessments of trustworthiness, competence, and warmth in blind and sighted adults
Oleszkiewicz, A., Pisanski, K., Lachowicz‐Tabaczek, K., & Sorokowska, A. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1146-y · Cited by 109 · cited 1×
“This finding highlights the need to critically examine how design choices in voice assistant technology influence how people use the technology and perceive the results [60].”
Lee, Y. J., Boden-Albala, B., Larson, E., Wilcox, A., & Bakken, S. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3499 · Cited by 104 · cited 1×
“A reliance on search engines is particularly the case for populations less able to access a primary care provider [8-15].”
Gender and Online Health Information Seeking: A Five Survey Meta-Analysis
Hallyburton, A., & Evarts, L. A. (2014) · Journal of Consumer Health On the Internet · doi:10.1080/15398285.2014.902268 · Cited by 102
Harrington, C. N., Garg, R., Woodward, A., & Williams, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501995 · Cited by 98
Why do people use ChatGPT? Exploring user motivations for generative conversational AI
Skjuve, M., Brandtzaeg, P. B., & Følstad, A. (2024) · First Monday · doi:10.5210/fm.v29i1.13541 · Cited by 95 · cited 1×
“Early research identified 6 reasons why users would turn to ChatGPT: productivity, novelty, creative work, learning and development, entertainment, and social interaction and support [61].”
Ayo-Ajibola, O., Davis, R. J., Lin, M. E., Riddell, J., & Kravitz, R. L. (2024) · Journal of Medical Internet Research · doi:10.2196/55138 · Cited by 95 · cited 3×
“These sampling decisions are supported by recent research that ChatGPT users were younger than nonusers (32.8 vs 39.1 years) with lower advanced degree attainment (BA or higher) [44].”
InfluentialChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results
Hristidis, V., Ruggiano, N., Brown, E. L., Ganta, S. R. R., & Stewart, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48966 · Cited by 90
Lee, K., Hoti, K., Hughes, J., & Emmerton, L. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7489 · Cited by 86 · cited 1×
“For over a decade, surveys have demonstrated that search engines are the most common platform for online information seeking [2], and this is supported in studies with a range of people, from everyday users [3,4] to patients with chronic health conditions [5-7].”
Perception of peer advice in online health communities: Access to lay expertise
Rueger, J., Dolfsma, W. A., & Aalbers, R. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113117 · Cited by 80 · cited 1×
“Research on online health communities shows that advice from similar peers, those who share similar interests or health conditions, is often valued over expert guidance, illustrating how homophily can outweigh expertise [39].”
Effects of Sponsorship, Web Site Design, and Google Ranking on the Credibility of Online Information
Westerwick, A. (2013) · Journal of Computer-Mediated Communication · doi:10.1111/jcc4.12006 · Cited by 77 · cited 1×
“Concurrently, the question of how people make decisions around the trustworthiness of a health source has also received significant attention [28-34].”
Sillence, E., Blythe, J. M., Briggs, P., & Moss, M. (2019) · Journal of Medical Internet Research · doi:10.2196/11125 · Cited by 72 · cited 2×
“More recent perspectives discuss how trust is iterative in online health contexts, but AI tools seem to be making this overlap between exploration and evaluation central to the search process [42].”
Trust in wikipedia: how users trust information from an unknown source
Lucassen, T., & Schraagen, J. M. (2010) · Proceedings of the 4th workshop on Information credibility · doi:10.1145/1772938.1772944 · Cited by 71
Consumer Use of St. John's Wort: A Survey on Effectiveness, Safety, and Tolerability
Beckman, S. E., Sommi, R. W., & Switzer, J. (2000) · Pharmacotherapy The Journal of Human Pharmacology and Drug Therapy · doi:10.1592/phco.20.6.568.35152 · Cited by 71 · cited 1×
“One study on consumer use of St John’s wort found that 74% of survey respondents admitted that they were taking the drug without seeking medical advice [71], suggesting some evidence for people seeking other forms of health information before making the decision to use the “drug.””
Mendel, T., Singh, N., Mann, D. M., Wiesenfeld, B., & Nov, O. (2025) · Journal of Medical Internet Research · doi:10.2196/64290 · Cited by 64 · cited 2×
“The survey also found that LLMs were perceived as less useful and less relevant than search engines but appeared more human than search and were seen as less biased [17].”
Palanica, A., Thommandram, A., Lee, A., Li, M., & Fossat, Y. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0133-x · Cited by 63
Hsu, W.-C., & Lee, M.-H. (2023) · Journal of Global Information Management · doi:10.4018/jgim.318661 · Cited by 62
SycEval: Evaluating LLM Sycophancy
Fanous, A., Goldberg, J., Agarwal, A. A., Lin, J., Zhou, A., Daneshjou, R., & Koyejo, S. (2025) · arXiv · doi:10.48550/arxiv.2502.08177 · Cited by 62 · cited 1×
“Recent studies suggest that concerns about LLMs and sycophancy are warranted [81,82].”
SycEval: Evaluating LLM Sycophancy
Fanous, A., Goldberg, J., Agarwal, A., Lin, J., Zhou, A., Xu, S., Bikia, V., Daneshjou, R., & Koyejo, S. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i1.36598 · Cited by 62
Pearson, G. (2020) · New Media & Society · doi:10.1177/1461444820910505 · Cited by 59 · cited 1×
“It has also been observed that web users struggle with “source blindness,” or the failure to process cues related to sources, in situations of “information context collapse”—when different types of content are placed in the same form and location [40,41].”
Shekar, S., Pataranutaporn, P., Sarabu, C. R., Cecchi, G., & Maes, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.15266 · Cited by 54 · cited 1×
“Much of this early research into the impact of LLMs has focused on “auditing” the quality of the results around health topics [19,20], comparing levels of trust between AI-powered health information and primary care providers [21-23] and investigating selective exposure via LLMs [24].”
The Influence of User Culture on Website Usability
Alexander, R., Thompson, N., McGill, T., & Murray, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102688 · Cited by 51 · cited 1×
“Studies have documented the challenges of using US-owned technology for people who speak languages other than English [89-91], but the adoption of LLMs raises further questions.”
Chin, H., Lima, G., Shin, M., Zhunis, A., Cha, C., Choi, J., & Cha, M. (2023) · Journal of Medical Internet Research · doi:10.2196/40922 · Cited by 49 · cited 1×
“…are supported by studies focused on non–AI-powered chatbots available during the COVID-19 pandemic, which found that user trust in chatbot answers was affected by the branding of the chatbot provider, disclosure, and empathy [55], as well as cultural sensitivity and the quality of language [56].”
Healthcare Voice AI Assistants: Factors Influencing Trust and Intention to Use
Zhan, X., Abdi, N., Seymour, W., & Such, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637339 · Cited by 46
Health Information Seeking Behaviors and Source Preferences between Chinese and U.S. Populations
Lu, L., Liu, J., & Yu, Y. (2020) · Journal of Health Communication · doi:10.1080/10810730.2020.1806414 · Cited by 45
Self-imposed filter bubbles: Selective attention and exposure in online search
Ekström, A. G., Niehorster, D. C., & Olsson, E. J. (2022) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2022.100226 · Cited by 45 · cited 1×
“…research on search engines suggests that differences in user queries are more likely to result in different search results rather than personalization [85], raising serious questions about the different responses that would be produced by different prompts for AI-and voice-assisted technologies.”
Brewer, R. N. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519642 · Cited by 43 · cited 1×
“…health information seeking reveals a complex trust dynamic: older adults may use voice assistants for confirming existing health knowledge but distrust the results, demonstrating a preference for more traditional information sources, such as direct communication with a medical professional [58].”
Inside the Mind of the Online Health Information Searcher using Think-Aloud Protocol
Macias, W., Lee, M., & Cunningham, N. (2017) · Health Communication · doi:10.1080/10410236.2017.1372040 · Cited by 40 · cited 1×
“This method has shown to be useful in usability test settings in order to understand people’s behavior with particular technologies [68,69], including a previous iteration of a study observing and listening to people search for health information online [70].”
Al Shboul, M. K. I., Alwreikat, A., & Alotaibi, F. A. (2024) · Science & Technology Libraries · doi:10.1080/0194262x.2023.2250835 · Cited by 38
Leslie‐Miller, C. J., Simon, S. L., Dean, K. M., Mokhallati, N., & Cushing, C. C. (2024) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsae075 · Cited by 38 · cited 1×
“Much of this early research into the impact of LLMs has focused on “auditing” the quality of the results around health topics [19,20], comparing levels of trust between AI-powered health information and primary care providers [21-23] and investigating selective exposure via LLMs [24].”
Layla’s Got You : Developing a tailored contraception chatbot for Black and Hispanic young women
Bonnevie, E., Lloyd, T. D., Rosenberg, S. D., Williams, K., Goldbarg, J., & Smyser, J. (2020) · Health Education Journal · doi:10.1177/0017896920981122 · Cited by 37 · cited 1×
“Another study focused on a chatbot designed specifically for Black and Hispanic female population emphasized the importance of language and tone for building trust [57].”
Snyder, E. C., Mendu, S., Sundar, S. S., & Abdullah, S. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103126 · Cited by 34 · cited 1×
“Another body of literature has explored trust in voice assistants in particular [51-54].”
Evaluating search engines and large language models for answering health questions
Fernández-Pichel, M., Pichel, J. C., & Losada, D. E. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01546-w · Cited by 33 · cited 3×
“However, more recent research shows that LLMs are slightly outperforming the search engines when answering health-related queries [19].”
Featured Snippets and their Influence on Users’ Credibility Judgements
Bink, M., Zimmerman, S., & Elsweiler, D. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505766 · Cited by 31 · cited 1×
“However, the convenience of short summaries has also prompted concerns about potential biases and misinformation [64-66].”
User Experience of COVID-19 Chatbots: Scoping Review
White, B. K., Martin, A., & White, J. A. (2022) · Journal of Medical Internet Research · doi:10.2196/35903 · Cited by 26 · cited 1×
“…are supported by studies focused on non–AI-powered chatbots available during the COVID-19 pandemic, which found that user trust in chatbot answers was affected by the branding of the chatbot provider, disclosure, and empathy [55], as well as cultural sensitivity and the quality of language [56].”
Factors influencing patients’ engagement with ChatGPT for accessing health-related information
Alanezi, F. (2024) · Critical Public Health · doi:10.1080/09581596.2024.2348164 · Cited by 26 · cited 1×
“…quantitative and qualitative literature attempting to understand the factors that are motivating the adoption of this new technology for HISB [18,45,46], highlighting that user-centric factors (perceived usefulness, convenience, trustworthiness, and AI experience and literacy) as well as…”
Mind + Machine: ChatGPT as a Basic Clinical Decisions Support Tool
Ayoub, M., Ballout, A. A., Zayek, R. A., & Ayoub, N. F. (2023) · Cureus · doi:10.7759/cureus.43690 · Cited by 25 · cited 2×
“A 2023 study that directly compared ChatGPT and Google revealed that ChatGPT struggled with specific and nuanced queries, particularly in specialized fields like health care [63].”
Comprehensive Model of Information Seeking: A Meta-Analysis
Yang, Q., Van Stee, S. K., & Rains, S. A. (2023) · Journal of Health Communication · doi:10.1080/10810730.2023.2214097 · Cited by 22 · cited 1×
“Existing models emphasize credibility, trustworthiness, and usefulness as primary drivers of HISB [75-77].”
Link, E., & Beckmann, S. (2025) · Communication Research Reports · doi:10.1080/08824096.2024.2427609 · Cited by 17 · cited 2×
“This is in line with other recent research on AI-HISB, which is starting to demonstrate that a person’s likelihood to use AI is not necessarily based on how much they trust the technology [16].”
Marcu, A., Black, G. B., & Whitaker, K. L. (2018) · Health Risk & Society · doi:10.1080/13698575.2018.1550742 · Cited by 16 · cited 1×
“For over a decade, surveys have demonstrated that search engines are the most common platform for online information seeking [2], and this is supported in studies with a range of people, from everyday users [3,4] to patients with chronic health conditions [5-7].”
Pycha, A., & Zellou, G. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1411414 · Cited by 16 · cited 1×
“Furthermore, we know that accents can affect credibility judgments with voice assistants [94,95], and our multilingual participants raised the challenges of making voice-assisted technologies understand their prompts.”
Tackling Language Modelling Bias in Support of Linguistic Diversity
Bella, G., Helm, P., Koch, G., & Giunchiglia, F. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658925 · Cited by 15 · cited 1×
“Models trained on English materials and then translated into other languages are vulnerable to mistakes and inherent biases [93].”
Trusting the Search: Unraveling Human Trust in Health Information from Google and ChatGPT
Sun, X., Ma, R., Zhao, X., Li, Z., Lindqvist, J., Ali, A. E., & Bosch, J. A. (2024) · arXiv · doi:10.48550/arxiv.2403.09987 · Cited by 14 · cited 3×
“…both quantitative and qualitative literature attempting to understand the factors that are motivating the adoption of this new technology for HISB [18,45,46], highlighting that user-centric factors (perceived usefulness, convenience, trustworthiness, and AI experience and literacy) as well as…”
A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and SolutionsPreprint
Vaugrante, L., Niepert, M., & Hagendorff, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.20303 · Cited by 14 · cited 1×
“Second, while replicating LLM results is essentially impossible [86], additional attempts to audit LLM health-related results across different users, particularly those who have very different worldviews, would be valuable.”
Understanding and Mitigating Bias in Online Health Search
Hashavit, A., Wang, H., Lin, R., Stern, T., & Kraus, S. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462930 · Cited by 13 · cited 1×
“However, the convenience of short summaries has also prompted concerns about potential biases and misinformation [64-66].”
Turner, B. A. (2013) · The statesman's yearbook online/Statesman's year-book · doi:10.1007/978-1-349-59643-0_155 · Cited by 12
TESTING THE ACCURACY OF MODERN LLMS IN ANSWERING GENERAL MEDICAL PROMPTS
Narula, S. S., Karkera, S., Challa, R., Virmani, S., Chilukuri, N., Elkas, M., Thammineni, N., Kamath, A., Jaiswal, P., & Krishnan, A. (2023) · International Journal of Social Science and Economic Research · doi:10.46609/ijsser.2023.v08i09.021 · Cited by 12 · cited 1×
“Much of this early research into the impact of LLMs has focused on “auditing” the quality of the results around health topics [19,20], comparing levels of trust between AI-powered health information and primary care providers [21-23] and investigating selective exposure via LLMs [24].”
ELEPHANT: Measuring and understanding social sycophancy in LLMs
Cheng, M., Yu, S., Lee, C., Khadpe, P., Ibrahim, L., & Jurafsky, D. (2025) · arXiv · doi:10.48550/arxiv.2505.13995 · Cited by 12
Amazeen, M. A., & Krishna, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3698102 · Cited by 11
The silicon service spectrum: warmth and competence explain people's preferences for AI assistants
Hernandez, I. A., & Chekili, A. (2024) · Frontiers in Social Psychology · doi:10.3389/frsps.2024.1396533 · Cited by 8 · cited 1×
“Another body of literature has explored trust in voice assistants in particular [51-54].”
Ugas, M., Calamia, M. A., Tan, J., Umakanthan, B., Hill, C., Tse, K., Cashell, A., Muraj, Z., Giuliani, M., & Papadakos, J. (2025) · Patient Education and Counseling · doi:10.1016/j.pec.2024.108560 · Cited by 8
Haddow, A. D., & Clarke, S. (2021) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.21-0216 · Cited by 3
Copestake, A., Duggan, L., Herbelot, A., Moeding, A., & von Redecker, E. (2024) · doi:10.33774/coe-2024-dx12p · Cited by 3
Research works printed but not linked: 3
- Hovland, CI, 1953, Communication and Persuasion; Psychological Studies of Opinion Change, xii
- Amazeen, MA, 2023, Int J Commun, 17, 23
- Oh, P, 2023, Research Handbook on Artificial Intelligence and Communication
Cited by
Trillo-Domínguez, M., Martín-Neira, J.-I., & Olvera‐Lobo, M. (2026) · Informatics · doi:10.3390/informatics13030041 · Cited by 4
“These hallucinations primarily manifest in the generation of fabricated references, in the absence of evidence or citations without a verifiable real origin [30,46–49].”
Tiao, P.-H., Chan, H., & Huang, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/82081 · Cited by 5
“The integration of large language model–based chatbots represents a further transformation in OHIS [10,126].”
Alon, L., & Levkovich, I. I. (2026) · Journal of Medical Internet Research · doi:10.2196/93944 · Cited by 5
“In 5 studies [6,7,13,25,35], evaluation of outputs felt more manageable when tools provided concrete verification and reuse affordances, such as visible sourcing, accessible transcripts, and options to save, revisit, or share content.”
InfluentialAn Eye Tracking Study: Are AI Overviews Changing Search Behavior?
Lawati, S. F. D. A., McKay, D., Sanderson, M., Thomas, P., & Trippas, J. R. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809530 · Cited by 5
“A recent study by Wardle et al. [41] re-searched how people meet their health-related information needs via search engines, including Google’s “AI Overviews” (Powered by Gemini), ChatGPT, and Alexa.”
Scherenberg, V. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1891658 · Cited by 5
“These answers may appear complete and personally relevant, even though users often cannot immediately judge how accurate, current, evidence-based, or appropriate they are for their own situation (1, 3).”
Metadata
- Authors
- Wardle, C., Urbani, S., & Wang, E.
- Affiliations
- Cornell University (US)
- Brown University (US)
- Corresponding author
- Wardle, Claire
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-10-07
- Volume
- 27
- Pages
- e79961
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC12541266
- OpenAlex
- W4414280203
- Cited by
- 43
- References
- 94
- Citations by year
- 2026: 43
- FWCI
- 53.84
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- health information seeking, AI Overviews, ChatGPT, voice assistants, Amazon Alexa, large language models, search behavior, trust in AI, credibility assessment, perceived usefulness, multilingual users, digital health literacy, platform selection, data provenance, voice recognition, cultural relevance, Google search results, user demographics
- MeSH terms
- Generative Artificial Intelligence, Adult, Aged, Aged, 80 and over, Artificial Intelligence, Female, Humans, Interviews as Topic, Male, Middle Aged, Internet, Qualitative Research, Young Adult, Information Seeking Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.