Head‐to‐Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition
Abstract
Objective: Chat Generative Pretrained Transformer (ChatGPT) is the newest iteration of OpenAI's generative artificial intelligence (AI) with the potential to influence many facets of life, including health care. This study sought to assess ChatGPT's capabilities as a source of medical knowledge, using Google Search as a comparison. Study Design: Cross‐sectional analysis. Setting: Online using ChatGPT, Google Seach, and Clinical Practice Guidelines (CPG). Methods: CPG Plain Language Summaries for 6 conditions were obtained. Questions relevant to specific conditions were developed and input into ChatGPT and Google Search. All questions were written from the patient perspective and sought (1) general medical knowledge or (2) medical recommendations, with varying levels of acuity (urgent or emergent vs routine clinical scenarios). Two blinded reviewers scored all passages and compared results from ChatGPT and Google Search, using the Patient Education Material Assessment Tool (PEMAT‐P) as the primary outcome. Additional customized questions were developed that assessed the medical content of the passages. Results: The overall average PEMAT‐P score for medical advice was 68.2% (standard deviation [SD]: 4.4) for ChatGPT and 89.4% (SD: 5.9) for Google Search ( p < .001). There was a statistically significant difference in the PEMAT‐P score by source ( p < .001) but not by urgency of the clinical situation ( p = .613). ChatGPT scored significantly higher than Google Search (87% vs 78%, p = .012) for patient education questions. Conclusion: ChatGPT fared better than Google Search when offering general medical knowledge, but it scored worse when providing medical recommendations. Health care providers should strive to understand the potential benefits and ramifications of generative AI to guide patients appropriately.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Patients may already refer to publicly accessible models like ChatGPT for medical questions in much the same way they use Google Search.
“Given that models like ChatGPT are already publicly accessible and widely used, patients may already refer to them for medical questions in much the same way they use Google Search, making concerns about their early adoption somewhat academic145.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [145]
The citation lists haven’t been fetched yet
References 12 · Cited by 111
- Authors
- Ayoub, N. F., Lee, Y., Grimm, D., & Divi, V.
- Affiliations
- Stanford Medicine (US)
- Corresponding author
- Ayoub, Noel F.
- Venue
- Otolaryngology–Head and Neck Surgery
- Publisher
- Wiley
- Published
- 2023-08-02
- Volume
- 170
- Issue
- 6
- Pages
- 1484-1491
- ISSN
- 0161-6439 (ISSN-L)
- 0194-5998
- 1097-6817
- OpenAlex
- W4385476863
- Cited by
- 111
- References
- 12
- Citations by year
- 2026: 27 · 2025: 34 · 2024: 48 · 2023: 2
- FWCI
- 4.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Otolaryngology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- ChatGPT, generative AI, Google search, medical knowledge acquisition, plain language summaries, patient education materials, PEMAT-P, medical advice, patient perspectives
- MeSH terms
- Artificial Intelligence, Cross-Sectional Studies, Humans, Internet, Search Engine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
