A Multidimensional Tool Based on the eHealth Literacy Framework: Development and Initial Validity Testing of the eHealth Literacy Questionnaire (eHLQ)
Open accessPublished version of a preprint
Abstract
BACKGROUND: For people to be able to access, understand, and benefit from the increasing digitalization of health services, it is critical that services are provided in a way that meets the user's needs, resources, and competence. OBJECTIVE: The objective of the study was to develop a questionnaire that captures the 7-dimensional eHealth Literacy Framework (eHLF). METHODS: Draft items were created in parallel in English and Danish. The items were generated from 450 statements collected during the conceptual development of eHLF. In all, 57 items (7 to 9 items per scale) were generated and adjusted after cognitive testing. Items were tested in 475 people recruited from settings in which the scale was intended to be used (community and health care settings) and including people with a range of chronic conditions. Measurement properties were assessed using approaches from item response theory (IRT) and classical test theory (CTT) such as confirmatory factor analysis (CFA) and reliability using composite scale reliability (CSR); potential bias due to age and sex was evaluated using differential item functioning (DIF). RESULTS: CFA confirmed the presence of the 7 a priori dimensions of eHLF. Following item analysis, a 35-item 7-scale questionnaire was constructed, covering (1) using technology to process health information (5 items, CSR=.84), (2) understanding of health concepts and language (5 items, CSR=.75), (3) ability to actively engage with digital services (5 items, CSR=.86), (4) feel safe and in control (5 items, CSR=.87), (5) motivated to engage with digital services (5 items, CSR=.84), (6) access to digital services that work (6 items, CSR=.77), and (7) digital services that suit individual needs (4 items, CSR=.85). A 7-factor CFA model, using small-variance priors for cross-loadings and residual correlations, had a satisfactory fit (posterior productive P value: .27, 95% CI for the difference between the observed and replicated chi-square values: -63.7 to 133.8). The CFA showed that all items loaded strongly on their respective factors. The IRT analysis showed that no items were found to have disordered thresholds. For most scales, discriminant validity was acceptable; however, 2 pairs of dimensions were highly correlated; dimensions 1 and 5 (r=.95), and dimensions 6 and 7 (r=.96). All dimensions were retained because of strong content differentiation and potential causal relationships between these dimensions. There is no evidence of DIF. CONCLUSIONS: The eHealth Literacy Questionnaire (eHLQ) is a multidimensional tool based on a well-defined a priori eHLF framework with robust properties. It has satisfactory evidence of construct validity and reliable measurement across a broad range of concepts (using both CTT and IRT traditions) in various groups. It is designed to be used to understand and evaluate people's interaction with digital health services.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The eHealth Literacy Questionnaire (eHLQ) was found to have good psychometric abilities.
“The eHLQ was found to have good psychometric abilities [41].”
Social determinants of health may affect the adoption of digital health.
“Moreover, social determinants of health may affect the adoption of digital health, requiring the personalization of digital devices and navigation.57-59”
From Implementing a PROACTive Care Pathway to Empower and Support Survivors of Breast Cancer (2023) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Structural Equations with Latent Variables
Bollen, K. A. (1989) · Wiley series in probability and statistics · doi:10.1002/9781118619179 · Cited by 17,512 · see citing works
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956) · Cited by 9,342 · see citing works
Low Health Literacy and Health Outcomes: An Updated Systematic Review
Berkman, N. D., Sheridan, S. L., Donahue, K. E., Halpern, D. J., & Crotty, K. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-2-201107190-00005 · Cited by 5,505 · see citing works
eHEALS: The eHealth Literacy Scale
Norman, C. D., & Skinner, H. A. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.4.e27 · Cited by 2,680 · see citing works
eHealth Literacy: Essential Skills for Consumer Health in a Networked World.
Norman, C. D., & Skinner, H. A. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.2.e9 · Cited by 2,643 · see citing works
Handbook of Modern Item Response Theory
van der Linden, W. J., & Hambleton, R. K. (1997) · Springer eBooks · doi:10.1007/978-1-4757-2691-6 · Cited by 2,331 · see citing works
Health promotion glossary (1998)
Nutbeam, D. (1998) · Health Promotion International · doi:10.1093/heapro/13.4.349 · Cited by 2,229 · see citing works
Differential Item Functioning.
Gudgeon, A. C., Holland, P. W., & Wainer, H. (1994) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348590 · Cited by 2,103 · see citing works
Bayesian structural equation modeling: A more flexible representation of substantive theory
Muthén, B., & Asparouhov, T. (2012) · Psychological Methods · doi:10.1037/a0026802 · Cited by 1,562 · see citing works
A Generalized Partial Credit Model: Application of an EM Algorithm
Muraki, E. (1992) · Applied Psychological Measurement · doi:10.1177/014662169201600206 · Cited by 1,546 · see citing works
Osborne, R. H., Batterham, R. W., Elsworth, G. R., Hawkins, M., & Buchbinder, R. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-658 · Cited by 1,339 · see citing works
Self-citationDetecting Differential Item Functioning Using Logistic Regression Procedures
Swaminathan, H., & Rogers, H. J. (1990) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1990.tb00754.x · Cited by 1,078 · see citing works
An argument-based approach to validity.
Kane, M. T. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.112.3.527 · Cited by 995 · see citing works
Eremenco, S. L., Cella, D. F., & Arnold, B. J. (2005) · Evaluation & the Health Professions · doi:10.1177/0163278705275342 · Cited by 754 · see citing works
van der Vaart, R., & Drossaert, C. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6709 · Cited by 687 · see citing works
WHO Health Promotion Glossary: new terms
Smith, B. J., Tang, K. C., & Nutbeam, D. (2006) · Health Promotion International · doi:10.1093/heapro/dal033 · Cited by 598 · see citing works
Argument-Based Approach to Validity
Kane, M. T. (2022) · The SAGE Encyclopedia of Research Design · doi:10.4135/9781071812082.n28 · Cited by 557 · see citing works
Osborne, R. H., Elsworth, G. R., & Whitfield, K. (2007) · Patient Education and Counseling · doi:10.1016/j.pec.2006.12.002 · Cited by 556 · see citing works
Self-citationvan der Vaart, R., van Deursen, A. J., Drossaert, C. H., Taal, E., van Dijk, J. A., & van de Laar, M. A. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1840 · Cited by 404 · see citing works
Asparouhov, T., Muthén, B., & Morin, A. J. S. (2015) · Journal of Management · doi:10.1177/0149206315591075 · Cited by 340 · see citing works
Differential Item Functioning Analysis With Ordinal Logistic Regression Techniques
Crane, P. K., Gibbons, L. E., Jolley, L., & van Belle, G. (2006) · Medical Care · doi:10.1097/01.mlr.0000245183.28384.ed · Cited by 285 · see citing works
Batterham, R. W., Buchbinder, R., Beauchamp, A., Dodson, S., Elsworth, G. R., & Osborne, R. H. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-694 · Cited by 238 · see citing works
Self-citationA Framework for Characterizing eHealth Literacy Demands and Barriers
Chan, C., & Kaufman, D. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1750 · Cited by 189 · see citing works
Beauchamp, A., Batterham, R. W., Dodson, S., Astbury, B., Elsworth, G. R., McPhee, C., Jacobson, J., Buchbinder, R., & Osborne, R. H. (2017) · BMC Public Health · doi:10.1186/s12889-017-4147-5 · Cited by 182 · see citing works
Self-citationKayser, L., Kushniruk, A. W., Osborne, R. H., Nørgaard, O., & Turner, P. A. (2015) · JMIR Human Factors · doi:10.2196/humanfactors.3696 · Cited by 163 · see citing works
Self-citationBuchbinder, R., Batterham, R. W., Elsworth, G. R., Dionne, C. E., Irvin, E., & Osborne, R. H. (2011) · Arthritis Research & Therapy · doi:10.1186/ar3468 · Cited by 117 · see citing works
Self-citationRosas, S. R., & Ridings, J. W. (2016) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2016.08.016 · Cited by 105 · see citing works
Elsworth, G. R., Beauchamp, A., & Osborne, R. H. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1754-2 · Cited by 105 · see citing works
Self-citationCognitive Interviews to Test and Refine Questionnaires
García, A. A. (2011) · Public Health Nursing · doi:10.1111/j.1525-1446.2010.00938.x · Cited by 105 · see citing works
Measures of Health Literacy: Workshop Summary
Hernandez, L. M. (2009) · Cited by 101 · see citing works
der Zee, C. H. V. R.-V., Beurskens, A. J. H. M., Swinkels, R. A. H. M., Pool, J. J. M., Batterham, R. W., Osborne, R. H., & de Vet, H. C. W. (2015) · Quality of Life Research · doi:10.1007/s11136-015-1149-6 · Cited by 40 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [9]Gilstad H Toward a comprehensive model of eHealth literacy 2014 5 19 Proceedings of the 2nd European Workshop on Practical Aspects of Health Information May 19-20, 2014 Trondheim, Norway
- [10]Karnoe A Kayser L How is eHealth literacy measured and what do the measurements tell us? A systematic review Knowl Manag E-Learn Int J KMEL 2015 12 26 7 4 576 600
Cited by
Sorted by citations
Digital literacy as a new determinant of health: A scoping review
Arias López, M. D. P., Ong, B. A., Borrat Frigola, X., Fernández, A. L., Hicklent, R. S., Obeles, A. J. T., Rocimo, A. M., & Celi, L. A. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000279 · Cited by 422 · see citing works
Bittlingmayer, U. H., Dadaczynski, K., Sahrai, D., van den Broucke, S., & Okan, O. (2020) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-019-03087-6 · Cited by 103 · see citing works
Development and Validation of Digital Health Technology Literacy Assessment Questionnaire
Yoon, J., Lee, M., Ahn, J. S., Oh, D., Shin, S.-Y., Chang, Y. J., & Cho, J. (2022) · Journal of Medical Systems · doi:10.1007/s10916-022-01800-8 · Cited by 74 · see citing works
van de Hei, S. J., Poot, C. C., van den Berg, L. N., Meijer, E., van Boven, J. F. M., Flokstra-de Blok, B. M. J., Postma, M. J., Chavannes, N. H., & Kocks, J. W. H. (2022) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2022-001400 · Cited by 22 · see citing works
Implementing a PROACTive Care Pathway to Empower and Support Survivors of Breast Cancer
Franzoi, M. A., Degousée, L., Martin, E., Semedo, P. M., Aupomerol, M., Soldato, D., Di Meglio, A., Chiodi, C., Barbier, A., Chaouachi, H., et al. (2023) · JCO Oncology Practice · doi:10.1200/op.23.00016 · Cited by 18 · see citing works
Hoogerwerf, E., Greeven, A., Goekoop, R., & Spinhoven, P. (2024) · Trials · doi:10.1186/s13063-023-07780-5 · Cited by 1 · see citing works
“The eHealth Literacy Questionnaire (eHLQ) aims to measure eHealth literacy based on the 7-dimensional eHealth Literacy Framework [41].”
Method
- Authors
- Kayser, L., Karnoe, A., Furstrand, D., Batterham, R., Christensen, K. B., Elsworth, G., & Osborne, R. H.
- Affiliations
- University of Copenhagen (DK)
- Thammasat University (TH)
- Deakin University (AU)
- Corresponding author
- Kayser, Lars
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-02-12
- Volume
- 20
- Issue
- 2
- Pages
- e36
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC5826975
- OpenAlex
- W2786841744
- Cited by
- 340
- References
- 32
- Citations by year
- 2026: 56 · 2025: 63 · 2024: 62 · 2023: 40 · 2022: 47 · 2021: 30 · 2020: 14 · 2019: 18 · 2018: 8 · 2013: 1
- FWCI
- 26.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- Own your potential (DEAKIN) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Swinburne figshare (Swinburne University of Technology) (submittedVersion, other-oa)
- Funders
- Scleroseforeningen
- Kræftens Bekæmpelse
- Innovationsfonden
- Multiple Sclerosis Society
- Deakin University (APP1059122)
- Gentofte Hospital
- TrygFonden
- Danish Foundation TrygFonden
- Medical Research Council
- National Health and Medical Research Council (1059122)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Social Sciences › Demography › Technology Use by Older Adults
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- digital health literacy, questionnaire development, scale validation, digital health services, chronic disease, differential item functioning, item response theory, cognitive assessment, medical data processing, access to digital services, cross-cultural adaptation, Danish version
- MeSH terms
- Female, Humans, Male, Surveys and Questionnaires, Reproducibility of Results, Telemedicine, Health Literacy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
