Pilot Testing of a Tool to Standardize the Assessment of the Quality of Health Information Generated by Artificial Intelligence-Based Models
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Background Artificial intelligence (AI)-based conversational models, such as Chat Generative Pre-trained Transformer (ChatGPT), Microsoft Bing, and Google Bard, have emerged as valuable sources of health information for lay individuals. However, the accuracy of the information provided by these AI models remains a significant concern. This pilot study aimed to test a new tool with key themes for inclusion as follows: Completeness of content, Lack of false information in the content, Evidence supporting the content, Appropriateness of the content, and Relevance, referred to as "CLEAR", designed to assess the quality of health information delivered by AI-based models. Methods Tool development involved a literature review on health information quality, followed by the initial establishment of the CLEAR tool, which comprised five items that aimed to assess the following: completeness, lack of false information, evidence support, appropriateness, and relevance. Each item was scored on a five-point Likert scale from excellent to poor. Content validity was checked by expert review. Pilot testing involved 32 healthcare professionals using the CLEAR tool to assess content on eight different health topics deliberately designed with varying qualities. The internal consistency was checked with Cronbach's alpha (α). Feedback from the pilot test resulted in language modifications to improve the clarity of the items. The final CLEAR tool was used to assess the quality of health information generated by four distinct AI models on five health topics. The AI models were ChatGPT 3.5, ChatGPT 4, Microsoft Bing, and Google Bard, and the content generated was scored by two independent raters with Cohen's kappa (κ) for inter-rater agreement. Results The final five CLEAR items were: (1) Is the content sufficient?; (2) Is the content accurate?; (3) Is the content evidence-based?; (4) Is the content clear, concise, and easy to understand?; and (5) Is the content free from irrelevant information? Pilot testing on the eight health topics revealed acceptable internal consistency with a Cronbach's α range of 0.669-0.981. The use of the final CLEAR tool yielded the following average scores: Microsoft Bing (mean=24.4±0.42), ChatGPT-4 (mean=23.6±0.96), Google Bard (mean=21.2±1.79), and ChatGPT-3.5 (mean=20.6±5.20). The inter-rater agreement revealed the following Cohen κ values: for ChatGPT-3.5 (κ=0.875, P<.001), ChatGPT-4 (κ=0.780, P<.001), Microsoft Bing (κ=0.348, P=.037), and Google Bard (κ=.749, P<.001). Conclusions The CLEAR tool is a brief yet helpful tool that can aid in standardizing testing of the quality of health information generated by AI-based models. Future studies are recommended to validate the utility of the CLEAR tool in the quality assessment of AI-generated health-related content using a larger sample across various complex health topics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212 · cited 1×
“Such evidence denotes the delivery of health information by AI models that is backed by the latest scientific advances and is free of bias, misinformation, or disinformation [1,33].”
BackgroundRevolutionizing healthcare: the role of artificial intelligence in clinical practice
Alowais, S. A., Alghamdi, S. S., Alsuhebany, N., Alqahtani, T., Alshaya, A. I., Almohareb, S. N., Aldairem, A., Alrashed, M., Bin Saleh, K., Badreldin, H. A., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04698-z · Cited by 3,146 · cited 1×
“Ensuring appropriateness in the AI model-generated health information also helps to enhance health literacy, which empowers lay individuals to make informed health decisions and understand the risks and benefits of various treatments and interventions [1,35].”
BackgroundSallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Self-citationA Review of the Role of Artificial Intelligence in Healthcare
Al Kuwaiti, A., Nazer, K., Al-Reedy, A., Al-Shehri, S., Al-Muhanna, A., Subbarayalu, A. V., Al Muhanna, D., & Al-Muhanna, F. A. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13060951 · Cited by 915
Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709
Drawbacks of artificial intelligence and their potential solutions in the healthcare sector.
khan, B., Fatima, H., Qureshi, A., Kumar, S., Hanan, A., Hussain, J., & Abdullah, S. (2023) · Biomedical Materials & Devices · doi:10.1007/s44174-023-00063-2 · Cited by 461 · cited 2×
“Consequently, it is important to assess the completeness of health information generated by AI-based tools and to identify the possible gaps in such information [1,24].”
BackgroundChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
Shahsavar, Y., & Choudhury, A. (2023) · JMIR Human Factors · doi:10.2196/47564 · Cited by 341 · cited 2×
“This issue poses health risks considering the current evidence showing a growing interest among lay individuals to use these AI-based tools for various health queries due to their perceived usefulness and ease of use [7-9].”
BackgroundSallam, M., Salim, N., Barakat, M., & Al-Tammemi, A. (2023) · Narra J · doi:10.52225/narra.v3i1.103 · Cited by 331 · cited 1×
“This issue poses health risks considering the current evidence showing a growing interest among lay individuals to use these AI-based tools for various health queries due to their perceived usefulness and ease of use [7-9].”
BackgroundSelf-citationMájovský, M., Černý, M., Kasal, M., Komarc, M., & Netuka, D. (2023) · Journal of Medical Internet Research · doi:10.2196/46924 · Cited by 301 · cited 2×
“Examples include incorrect self-diagnosis and treatment, delayed seeking of medical help, potential disease transmission, and undermining trust in healthcare professionals and health institutions [1,6,25].”
BackgroundChatGPT: these are not hallucinations – they’re fabrications and falsifications
Emsley, R. (2023) · Schizophrenia · doi:10.1038/s41537-023-00379-4 · Cited by 258
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234
Self-citationAli, K., Barhom, N., Tamimi, F., & Duggal, M. (2024) · European Journal of Dental Education · doi:10.1111/eje.12937 · Cited by 152
The impact of completeness and web use motivation on the credibility of e-health information
Dutta-Bergman, M. J. (2004) · Journal of Communication · doi:10.1111/j.1460-2466.2004.tb02627.x · Cited by 146
Al-Ashwal, F. Y., Zawiah, M., Gharaibeh, L., Abu-Farha, R., & Bitar, A. N. (2023) · Drug Healthcare and Patient Safety · doi:10.2147/dhps.s425858 · Cited by 125 · cited 1×
“Additionally, such an approach is recommended to be conducted in various health domains given the intricacies and peculiarities of each subject (e.g., maxillofacial surgery, dentistry, and pharmacy) [29-32].”
BackgroundPerformance of ChatGPT on the pharmacist licensing examination in Taiwan
Wang, Y.-M., Shen, H.-W., & Chen, T.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000942 · Cited by 109
González-Gonzalo, C., Thee, E. F., Klaver, C. C. W., Lee, A. Y., Schlingemann, R. O., Tufail, A., Verbraak, F., & Sánchez, C. I. (2022) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2021.101034 · Cited by 103
Farnood, A., Johnston, B., & Mair, F. S. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01243-6 · Cited by 100
Chat GPT and artificial intelligence in medical writing: concerns and ethical considerations
Doyal, A. S., Sender, D., Nanda, M., & Serrano, R. A. (2023) · Cureus · doi:10.7759/cureus.43292 · Cited by 65 · cited 2×
“Thus, ensuring the generation of correct, reliable, and credible medical information is of high importance and should be considered by AI-models’ developers, considering the current evidence showing the generation of inaccurate information by these AI-based models [26-28].”
BackgroundDhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.42972 · Cited by 59 · cited 1×
“These advances are exemplified by the emergence and widespread use of AI-based 1, 2 3, 4 5 conversational models that are characterized by ease of use and a high degree of perceived usefulness, such as Chat Generative Pre-trained Transformer (ChatGPT), Google Bard, and Microsoft Bing [3].”
BackgroundZhang, Y., Lee, E. W. J., & Teo, W.-P. (2023) · JMIR Aging · doi:10.2196/43709 · Cited by 26
Cited by
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424
AI-Driven Clinical Decision Support Systems: An Ongoing Pursuit of Potential
Elhaddad, M., & Hamam, S. (2024) · Cureus · doi:10.7759/cureus.57728 · Cited by 301
“These studies underscore the importance of criteria such as completeness, absence of false information, evidence support, appropriateness, and relevance in ensuring the effectiveness and reliability of AI-driven solutions [31].”
BackgroundBelow average ChatGPT performance in medical microbiology exam compared to university students
Sallam, M., & Al-Salahat, K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1333415 · Cited by 39
Self-citationKaiser, K. N., Hughes, A. J., Yang, A. D., Turk, A. A., Mohanty, S., Gonzalez, A. A., Patzer, R. E., Bilimoria, K. Y., & Ellis, R. J. (2024) · Journal of Surgical Oncology · doi:10.1002/jso.27821 · Cited by 29
Williams, B., & Erstad, B. L. (2025) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxaf075 · Cited by 13
Metadata
- Authors
- Sallam, M., Barakat, M., & Sallam, M.
- Venue
- Cureus
- Published
- 2023-11-24
- Volume
- 15
- Issue
- 11
- Pages
- e49373-e49373
- ISSN
- 2168-8184 (ISSN-L)
- OpenAlex
- W4389002232
- Cited by
- 63
- References
- 22
- FWCI
- 2.18
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.