An objective analysis of quality and readability of online information on COVID-19
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
High quality, readable health information is vital to mitigate the impact of the COVID-19 pandemic. The aim of this study was to assess the quality and readability of online COVID-19 information using 6 validated tools. This is a cross-sectional study. “COVID-19” was searched across the three most popular English language search engines. Quality was evaluated using the DISCERN score, Journal of the American Medical Association benchmark criteria and Health On the Net Foundation Code of Conduct. Readability was assessed using the Flesch Reading Ease Score, Flesch-Kincaid Grade Level and Gunning-Fog Index. 41 websites were suitable for analysis. 9.8% fulfilled all JAMA criteria. Only one website was HONCode certified. Mean DISCERN score was 47.8/80 (“fair”). This was highest in websites published by a professional society/medical journal/healthcare provider. Readability varied from an 8th to 12th grade level. The overall quality of online COVID-19 information was “fair”. Much of this information was above the recommended 5th to 6th grade level, impeding access for many.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“No scoring system has been widely accepted as the gold standard for evaluating the ease of readability.”
From Are paid tools worth the cost? A prospective cross-over study to find the right tool for plagiarism detection (2023) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Kincaid, J. P., Fishburne, Jr., Robert P., R., Richard L., C., & Brad, S. (1975) · doi:10.21236/ada006655 · Cited by 3,497 · cited 1×
“Higher scores indicate easier readability [25].”
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329 · cited 1×
“Websites are scored from 16 to 80, with a higher score indicating better-quality information [20].”
BackgroundConsumer health information seeking on the internet: the state of the art
Cline, R. J. W. (2001) · Health Education Research · doi:10.1093/her/16.6.671 · Cited by 1,708 · cited 1×
“There has been an increase in internet usage worldwide, as well as an increase in the depth and breadth of online content [3, 4].”
BackgroundHealth Information on the Internet: Accessibility, Quality, and Readability in English and Spanish
Berland, G. K., Elliott, M. N., Morales, L. S., Algazy, J. I., Kravitz, R. L., Broder, M. S., Kanouse, D. E., Muñoz, J. A., Puyol, J.-A., Lara, M., et al. (2001) · JAMA · doi:10.1001/jama.285.20.2612 · Cited by 1,208 · cited 1×
“While the availability and usage of online health information has expanded, the quality and readability of that online health information varies considerably [8].”
BackgroundSilberg, W. M. (1997) · JAMA · doi:10.1001/jama.277.15.1244 · Cited by 1,151
Readability Standards for Informed-Consent Forms as Compared with Actual Readability
Paasche-Orlow, M. K., Taylor, H. A., & Brancati, F. L. (2003) · New England Journal of Medicine · doi:10.1056/nejmsa021212 · Cited by 754 · cited 2×
“These readability scores were chosen as they are validated systems which have been shown to be consistent in terms of their results [29, 30], while also being easy for content makers to use.”
BackgroundMethodKincaid, J. P., Robert, P. F. L., Richard, L. R., & Brad, S. (1975) · Cited by 739
Wang, L.-W., Miller, M. J., Schmitt, M. R., & Wen, F. K. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.05.009 · Cited by 445 · cited 1×
“FRES and FKGL are the most widely used readability scoring systems, therefore allowing for broad comparability and generalizability of results [30, 31].”
BackgroundThe Health On the Net Code of Conduct for medical and health Websites
Boyer, C., Selby, M., Scherrer, J.-R., & Appel, R. D. (1998) · Computers in Biology and Medicine · doi:10.1016/s0010-4825(98)00037-7 · Cited by 427 · cited 1×
“HON is a non-profit organisation linked to the Economic and Social Council of the United Nations with the stated aim of enhancing the dissemination of quality health information globally [21].”
BackgroundEVALUATION OF LITERACY LEVEL OF PATIENT EDUCATION PAGES IN HEALTH-RELATED JOURNALS
Cotugna, N., Vickery, C. E., & Carpenter-Haefele, K. M. (2005) · Journal of Community Health · doi:10.1007/s10900-004-1959-x · Cited by 344 · cited 2×
“Overall this analysis showed that COVID-19 online information readability levels—as assessed by all three scoring systems—were greater than the recommended 5th to 6th grade level [9].”
BackgroundReadability Assessment of Internet-Based Consumer Health Information
Walsh, T. M., & Volsko, T. A. (2008) · Respiratory Care · doi:10.4187/respcare.08531310 · Cited by 328 · cited 1×
“Their results are interpreted according to the years of education typically required for that level of literacy or readability [24].”
BackgroundClarke, M. A., Moore, J. L., Steege, L. M., Koopman, R. J., Belden, J. L., Canfield, S. M., Meadows, S. E., Elliott, S. G., & Kim, M. S. (2016) · Health Informatics Journal · doi:10.1177/1460458215602939 · Cited by 312
Feeding Our Immune System: Impact on Metabolism
Wolowczuk, I., Verwaerde, C., Viltart, O., Delanoye, A., Delacre, M., Pot, B., & Grangette, C. (2008) · Clinical and Developmental Immunology · doi:10.1155/2008/639803 · Cited by 297 · cited 1×
“Normally distributed continuous data are presented as the mean and standard deviation [32]; non-normally distributed data are presented as the median and interquartile range [33].”
MethodReadability of online health information: A meta-narrative systematic review
Daraz, L., Morrow, A. S., Ponce, O. J., Farah, W., Katabi, A., Majzoub, A., Seisa, M. O., Benkhadra, R., Alsawas, M., Larry, P., et al. (2018) · American Journal of Medical Quality · doi:10.1177/1062860617751639 · Cited by 278
Weiss, B. D. (1998) · PubMed · Cited by 242
Surveys of Physicians and Electronic Health Information
Hesse, B. W., Moser, R. P., & Rutten, L. J. (2010) · New England Journal of Medicine · doi:10.1056/nejmc0909595 · Cited by 216
A statistical model for scientific readability
Si, L., & Callan, J. (2001) · Proceedings of the tenth international conference on Information and knowledge management · doi:10.1145/502585.502695 · Cited by 197 · cited 2×
“1 3 widely used, for example being the standard readability test employed by the US Department of Defense [26].”
Readability formulas: Useful or useless?
McClure, G. M. (1987) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.1987.6449109 · Cited by 166
Quality of health information on the Internet in pediatric neuro-oncology
Hargrave, D. R., Hargrave, U. A., & Bouffet, É. (2006) · Neuro-Oncology · doi:10.1215/15228517-2005-008 · Cited by 163
Szmuda, T., Özdemir, C., Ali, S., Singh, A., Syed, M. T., & Słoniewski, P. (2020) · Public Health · doi:10.1016/j.puhe.2020.05.041 · Cited by 163 · cited 1×
“The only published study assessing the readability of online COVID-19 information to date found the information to be at an 11 to 14-year-old reading level [15].”
BackgroundAdemiluyi, G., Rees, C. E., & Sheard, C. E. (2003) · Patient Education and Counseling · doi:10.1016/s0738-3991(02)00124-6 · Cited by 160
Readability of Online Patient Education Materials From the AAOS Web Site
Sabharwal, S., Badarudeen, S., & Unes Kunju, S. (2008) · Clinical Orthopaedics & Related Research · doi:10.1007/s11999-008-0193-8 · Cited by 157 · cited 1×
“This limitation has been identified in other similar studies [40].”
ResultJha, A., Lin, L., Short, S. M., Argentini, G., Gamhewage, G., & Savoia, E. (2018) · PLOS ONE · doi:10.1371/journal.pone.0205555 · Cited by 72
Cauda Equina Syndrome: Assessing the Readability and Quality of Patient Information on the Internet
OʼNeill, S. C., Baker, J. F., Fitzgerald, C., Fleming, C., Rowan, F., Byrne, D., & Synnott, K. (2014) · Spine · doi:10.1097/brs.0000000000000282 · Cited by 63 · cited 2×
“Website quality was assessed through three well established, validated tools: JAMA benchmark criteria, DISCERN criteria and Health On the Net (HON) Foundation Code of Conduct (HONcode) certification [10, 11, 18].”
BackgroundMethodMurray, K., Murray, T. E., O’Rourke, A., Low, C., & Veale, D. J. (2019) · Interactive Journal of Medical Research · doi:10.2196/12855 · Cited by 52
Self-citationReadability of the Most Commonly Accessed Arthroscopy‐Related Online Patient Education Materials
Akinleye, S. D., Krochak, R., Richardson, N., Garofolo, G., Culbertson, M. D., & Erez, O. (2017) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2017.09.043 · Cited by 32
Andrade, P. A., Hermsdorff, H. H. M., Leite, J. I. A., Shivappa, N., Hébert, J. R., Henriques, H. K. F., & Rosa, C. D. O. B. (2018) · Obesity Surgery · doi:10.1007/s11695-018-3530-3 · Cited by 24 · cited 1×
“Normally distributed continuous data are presented as the mean and standard deviation [32]; non-normally distributed data are presented as the median and interquartile range [33].”
MethodAn Evaluation of the Information Sources of Cancer Patients’ Relatives. A Prospective Survey
Yıldırım, S., Kazaz, S. N., Semiz, H. S., et al. (2018) · Journal of Cancer Education · doi:10.1007/s13187-018-1395-8 · Cited by 14
Oloidi, A. D., Nduaguba, S. O., & Obamiro, K. O. (2020) · Pan African Medical Journal · doi:10.11604/pamj.2020.35.70.18237 · Cited by 12
Murray, T. E., Mansoor, T., Bowden, D. J., O'Neill, D. C., & Lee, M. J. (2018) · Academic Radiology · doi:10.1016/j.acra.2017.11.007 · Cited by 11 · cited 1×
“Website quality was assessed through three well established, validated tools: JAMA benchmark criteria, DISCERN criteria and Health On the Net (HON) Foundation Code of Conduct (HONcode) certification [10, 11, 18].”
Method
Research works printed but not linked: 3
- Silberg WM. Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet. Jama. 1997;277(15). 9103351 · DOI · Paper page
- Janan D, Wray D. Readability: The limitations of an approach through formulae. in British Educational Research Association Annual Conference. 2012. University of Manchester: Education Line.
- Communicating risk in public health emergencies a WHO guideline for emergency risk communication (ERC) policy and practice 2017 Geneva World Health Organization 31059213
Cited by
People Overtrust AI-Generated Medical Advice despite Low Accuracy
Shekar, S., Pataranutaporn, P., Sarabu, C. R., Cecchi, G., & Maes, P. (2025) · NEJM AI · doi:10.1056/aioa2300015 · Cited by 138
The utility of ChatGPT in generating patient-facing and clinical responses for melanoma
Young, J. N., O’Hagan, R., Poplausky, D., Levoska, M. A., Gulati, N., Ungar, B., & Ungar, J. P. (2023) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2023.05.024 · Cited by 53
Shekar, S., Pataranutaporn, P., Sarabu, C. R., Cecchi, G., & Maes, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.15266 · Cited by 54
Readability Metrics in Patient Education: Where Do We Innovate?
Singh, S. P., Jamal, A., & Qureshi, F. (2024) · Clinics and Practice · doi:10.3390/clinpract14060183 · Cited by 35
Erkin, Y., Hancı, V., & Özduran, E. (2023) · Medicine · doi:10.1097/md.0000000000033529 · Cited by 28
“Wrigley et al[34] and Killip et al[35] reported median DISCERN scores with fair quality (43 and 43.”
Background
Metadata
- Authors
- Wrigley Kelly, N. E., Murray, K. E., McCarthy, C., & O’Shea, D. B.
- Affiliations
- University College Dublin (IE)
- Venue
- Health and Technology
- Published
- 2021-06-24
- Volume
- 11
- Issue
- 5
- Pages
- 1093-1099
- ISSN
- 2190-7188 (ISSN-L)
- 2190-7196
- PMC
- PMC8222704
- OpenAlex
- W3174777258
- Semantic Scholar
- ad8f1bd1f788da1d5a5538038c99fec66c34cf47
- Cited by
- 69
- References
- 31
- Citations by year
- 2026: 11 · 2025: 15 · 2024: 16 · 2023: 15 · 2022: 10 · 2021: 2
- FWCI
- 12.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Health and Technology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Lenus, The Irish Health Repository (Dr Steevens Hospital Library) (submittedVersion, cc-by)
- Funders
- University College Dublin
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Social Sciences › Health › Social Media in Health Education
- Social Sciences › Communication › Communication and COVID-19 Impact
- Keywords
- COVID-19 pandemic, online health information, information quality, readability, health websites, DISCERN, HONcode, JAMA benchmark criteria, Flesch-Kincaid Grade Level, Gunning Fog Index, health literacy, search engine results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.