Computerized versus hand-scored health literacy tools: a comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials
Open accessAbstract
Objective: The research compared and contrasted hand-scoring and computerized methods of evaluating the grade level of patient education materials that are distributed at an academic medical center in east Tennessee and sought to determine if these materials adhered to the American Medical Association’s (AMA’s) recommended reading level of sixth grade.Methods: Librarians at an academic medical center located in the heart of Appalachian Tennessee initiated the assessment of 150 of the most used printed patient education materials. Based on the Flesch-Kincaid (F-K) scoring rubric, 2 of the 150 documents were excluded from statistical comparisons due to the absence of text (images only). Researchers assessed the remaining 148 documents using the hand-scored Simple Measure of Gobbledygook (SMOG) method and the computerized F-K grade level method. For SMOG, 3 independent reviewers hand-scored each of the 150 documents. For F-K, documents were analyzed using Microsoft Word. Reading grade levels scores were entered into a database for statistical analysis. Inter-rater reliability was calculated using intra-class correlation coefficients (ICC). Paired t-tests were used to compare readability means.Results: Acceptable inter-rater reliability was found for SMOG (ICC=0.95). For the 148 documents assessed, SMOG produced a significantly higher mean reading grade level (M=9.6, SD=1.3) than F-K (M=6.5, SD=1.3; p<0.001). Additionally, when using the SMOG method of assessment, 147 of the 148 documents (99.3%) scored above the AMA’s recommended reading level of sixth grade.Conclusions: Computerized health literacy assessment tools, used by many national patient education material providers, might not be representative of the actual reading grade levels of patient education materials. This is problematic in regions like Appalachia because materials may not be comprehensible to the area’s low-literacy patients. Medical librarians have the potential to advance their role in patient education to better serve their patient populations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SMOG has shown acceptable interrater reliability for patient education materials.
“To assess the readability of a question, we used the Simple Measure of Gobbledygook (SMOG) as it has shown acceptable interrater reliability for patient education materials in the literature [21].”
From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Low Health Literacy and Health Outcomes: An Updated Systematic Review
Berkman, N. D., Sheridan, S. L., Donahue, K. E., Halpern, D. J., & Crotty, K. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-2-201107190-00005 · Cited by 5,505 · see citing works
National Center for Education Statistics
Carlson, C. L. (2008) · Encyclopedia of Educational Psychology · doi:10.4135/9781412963848.n192 · Cited by 4,546 · see citing works
SMOG Grading - A New Readability Formula.
McLaughlin, G. H. (1969) · The Journal of Reading · Cited by 2,067 · see citing works
The Principles of Readability.
DuBay, W. H. (2004) · Cited by 780 · see citing works
Assessing Readability of Patient Education Materials: Current Role in Orthopaedics
Badarudeen, S., & Sabharwal, S. (2010) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-010-1380-y · Cited by 663 · see citing works
National Action Plan to Improve Health Literacy
Thompson, T. L. (2014) · Encyclopedia of Health Communication · doi:10.4135/9781483346427.n360 · Cited by 572 · see citing works
Wang, L.-W., Miller, M. J., Schmitt, M. R., & Wen, F. K. (2012) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.05.009 · Cited by 449 · see citing works
Health Literacy in Primary Care Practice.
Hersh, L. R., Salzman, B., & Snyderman, D. (2015) · PubMed · Cited by 357 · see citing works
Readability Assessment of Internet-Based Consumer Health Information
Walsh, T. M., & Volsko, T. A. (2008) · Respiratory Care · doi:10.4187/respcare.08531310 · Cited by 328 · see citing works
The Readability of Pediatric Patient Education Materials on the World Wide Web
D’Alessandro, D. M., Kingsley, P., & Johnson-West, J. (2001) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpedi.155.7.807 · Cited by 286 · see citing works
Health Literacy and 30-Day Postdischarge Hospital Utilization
Mitchell, S. E., Sadikova, E., Jack, B. W., & Paasche‐Orlow, M. K. (2012) · Journal of Health Communication · doi:10.1080/10810730.2012.715233 · Cited by 273 · see citing works
Dee, C. R., & Stanley, E. E. (2005) · PubMed · Cited by 161 · see citing works
The Readability of AAOS Patient Education Materials: Evaluating the Progress Since 2008
Roberts, H. J., Zhang, D., & Dyer, G. S. M. (2016) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.15.00658 · Cited by 135 · see citing works
The readability of American Academy of Pediatrics patient education brochures
Freda, M. C. (2005) · Journal of Pediatric Health Care · doi:10.1016/j.pedhc.2005.01.013 · Cited by 120 · see citing works
Health Literacy Predicts Morbidity and Mortality in Rural Patients With Heart Failure
Moser, D. K., Robinson, S., Biddle, M., Pelter, M. M., Nesbitt, T. S., Southard, J., Cooper, L. S., & Dracup, K. (2015) · Journal of Cardiac Failure · doi:10.1016/j.cardfail.2015.04.004 · Cited by 112 · see citing works
American Academy of Family Physicians patient education materials: can patients read them?
Wallace, L. S., & Lennon, E. S. (2004) · PubMed · Cited by 90 · see citing works
Evaluation of Printed Health Education Materials for Use by Low‐Education Families
Ryan, L., Logsdon, M. C., McGill, S., et al. (2014) · Journal of Nursing Scholarship · doi:10.1111/jnu.12076 · Cited by 87 · see citing works
Schaffer, S. D., & Tian, L. (2004) · Clinical Nursing Research · doi:10.1177/1054773803259300 · Cited by 83 · see citing works
Neven, P., Markopoulos, C., Tanner, M., Marty, M., Kreienberg, R., Atkins, L., Franquet, A., Gnant, M., Neciosup, S., Tesarova, P., et al. (2014) · The Breast · doi:10.1016/j.breast.2014.02.009 · Cited by 53 · see citing works
The Evolving Role and Value of Libraries and Librarians in Health Care
Sollenberger, J. F., & Holloway, R. G. (2013) · JAMA · doi:10.1001/jama.2013.277050 · Cited by 53 · see citing works
Educational Attainment in Appalachia: Growing with the Nation, But Challenges Remain.
Shaw, T., DeYoung, A. J., & Rademacher, E. W. (2004) · Journal of Appalachian Studies · Cited by 47 · see citing works
Consumer health information: libraries as partners.
Eakin, D., Jackson, S. J., & Hannigan, G. G. (1980) · PubMed · Cited by 46 · see citing works
Rubenstein, E. L. (2012) · Library & Information History · doi:10.1179/1758348912z.00000000016 · Cited by 39 · see citing works
Is the Readability of Spine-Related Patient Education Material Improving?
Eltorai, A. E. M., Cheatham, M., Naqvi, S. S., et al. (2016) · Spine · doi:10.1097/brs.0000000000001446 · Cited by 35 · see citing works
Information prescriptions, 1930–2013: an international history and comprehensive review
McKnight, M. (2014) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.102.4.008 · Cited by 24 · see citing works
Nortriptyline toxicity secondary to interaction with bupropion sustained-release
Weintraub, D. A. (2001) · Depression and Anxiety · doi:10.1002/1520-6394(2001)13:1 · Cited by 16 · see citing works
Hospital Librarians: From Consumer Health to Patient Education and Beyond
Egeland, M. (2015) · Journal of Hospital Librarianship · doi:10.1080/15323269.2015.982944 · Cited by 10 · see citing works
Patient Education Materials From the Layperson's Perspective
Blanck, A. W., & Marshall, C. J. (2011) · Journal for Nurses in Staff Development · doi:10.1097/nnd.0b013e31820eeff6 · Cited by 9 · see citing works
Evaluating Patient Education Materials for Grade Level
Leonard, K. (2017) · Journal of Consumer Health on the Internet · doi:10.1080/15398285.2017.1280347 · Cited by 9 · see citing works
A Bibliography on Bibliotherapy and Hospital Libraries.
Schneck, J. M. (1945) · PubMed · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [12]Doak C Doak LG Root JH The literacy problem Teaching patients with low literacy skills Philadelphia, PA J.B. Lippincott 1996 1 8
- [18]McGraw HC SMOG conversion table 1 Office of Educational Research [class handout] Townson, MD Baltimore County Public Schools 2016
- [28]Elsevier Interactive patient education [Internet] Elsevier 2017 [cited 16 Oct 2017]. < https://www.elsevier.com/solutions/patient-engagement/interactive-patient-education>
Cited by
Sorted by citations
Meyer, A., Riese, J., & Streichert, T. (2024) · JMIR Medical Education · doi:10.2196/50965 · Cited by 110 · see citing works
Mac, O. A., Thayre, A., Tan, S., & Dodd, R. H. (2020) · Journal of Medical Internet Research · doi:10.2196/16701 · Cited by 35 · see citing works
Ugas, M., Calamia, M. A., Tan, J., Umakanthan, B., Hill, C., Tse, K., Cashell, A., Muraj, Z., Giuliani, M., & Papadakos, J. (2025) · Patient Education and Counseling · doi:10.1016/j.pec.2024.108560 · Cited by 8 · see citing works
- Authors
- Leonard Grabeel, K., Russomanno, J., Oelschlegel, S., Tester, E., & Heidel, R. E.
- Affiliations
- University of Tennessee Medical Center (US)
- Venue
- Journal of the Medical Library Association
- Published
- 2018-01-12
- Volume
- 106
- Issue
- 1
- Pages
- 38-45
- ISSN
- 1536-5050 (ISSN-L)
- 1558-9439
- PMC
- PMC5764592
- OpenAlex
- W2783929218
- Cited by
- 96
- References
- 30
- Citations by year
- 2026: 8 · 2025: 20 · 2024: 21 · 2023: 16 · 2022: 12 · 2021: 5 · 2020: 10 · 2019: 3 · 2018: 1
- FWCI
- 7.25
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Journal of the Medical Library Association JMLA (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Keywords
- health literacy, patient education materials, readability assessment, smog, Flesch-Kincaid Grade Level, inter-rater reliability, medical librarians, Appalachian population, low-literacy patients
- MeSH terms
- Adult, Computer-Assisted Instruction, Female, Humans, Male, Middle Aged, Patient Education as Topic, Reading, Teaching Materials, Tennessee, Comprehension, Health Literacy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
