Online usability and patients with long-term conditions: A mixed-methods approach
Abstract
Background To improve the information position of health care consumers and to facilitate decision-making behavior in health the Dutch ministry of Health commissioned the National Institute for Public Health and the Environment to develop, host and manage a public national health and care portal (www.kiesbeter.nl) on the Internet. The portal is used by over 4 million visitors in 2010. Among them, an increasing amount of patients that use the portal for information and decision making on medical issues, healthy living, health care providers and other topics.Objective First objective is to examine what usability aspects of the portal kiesBeter.nl matter for chronic patients and their informal carers with regard to information seeking, self-management, decision making, on line health information and other variables. Second objective is to make evidence-based practical recommendations for usability improvement.Methods An innovative combination of techniques (semi-structured interviews; eHealth Literacy scale; scenario-based study using think-aloud protocol and screen capture software; focus group) is used to study usability and on line information seeking behavior in a non random judgment sample of three groups of patients (N=21) with long-term medical conditions (arthritis, asthma and diabetes).Results The search strategy mostly used (65%) by the relatively well-educated subjects is 'orienteering'. Users with long-term conditions and their carers expect tailored support from a national health portal, to help them navigate, search and find the detailed information they need. They encounter serious problems with these usability issues some of which are disease-specific. Patients indicate a need for personalized information. They report low impact on self-management and decision making. Overall judgment of usability is rated 7 on a Likert type 0-10 scale. Based on the outcomes recommendations could be formulated. These have led to major adaptations to improve usability.Limitations A non-representative composition of a small non random judgment sample does not permit generalization to other populations and cognitive bias cannot be quantified. However if mixed methods are applied valid conclusions can be drawn with regard to usability issues.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multiple authors in the included studies stated that ideally, multiple methods should be combined to paint a full picture of a prototype's usability.
“Second, authors stated that ideally, multiple methods should be combined to paint a full picture of a prototype’s usability”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [24]
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483 · see citing works
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035 · see citing works
eHEALS: The eHealth Literacy Scale
Norman, C. D., & Skinner, H. A. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.4.e27 · Cited by 2,680 · see citing works
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · see citing works
Talking about tactile experiences
Obrist, M., Seah, S. A., & Subramanian, S. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466220 · Cited by 1,627 · see citing works
Eye Tracking in Human-Computer Interaction and Usability Research
Jacob, R. J. K., & Karn, K. S. (2003) · Elsevier eBooks · doi:10.1016/b978-044451020-4/50031-1 · Cited by 1,575 · see citing works
Information seeking in electronic environments
(1996) · Choice Reviews Online · doi:10.5860/choice.33-3366 · Cited by 1,487 · see citing works
Assessing dimensions of perceived visual aesthetics of web sites
Lavie, T., & Tractinsky, N. (2003) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2003.09.002 · Cited by 1,349 · see citing works
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · see citing works
Self-citationCurrent practice in measuring usability: Challenges to usability studies and research
Hornbæk, K. (2006) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.06.002 · Cited by 1,123 · see citing works
Thinking aloud: reconciling theory and practice
Boren, T., & Ramey, J. A. (2000) · IEEE Transactions on Professional Communication · doi:10.1109/47.867942 · Cited by 849 · see citing works
Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(92)90039-n · Cited by 797 · see citing works
Interactive Health Communication Applications for people with chronic disease
Murray, E., Burns, J., See Tai, S., Lai, R., & Nazareth, I. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004274.pub4 · Cited by 722 · see citing works
Jaspers, M. W. M. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.10.002 · Cited by 649 · see citing works
Nielsen, J., & Loranger, H. (2006) · Cited by 648 · see citing works
The mind's eye : cognitive and applied aspects of eye movement research
Hyönä, J., Radach, R. R., & Deubel, H. (2003) · Cited by 588 · see citing works
Computer-tailored health interventions delivered over the web: Review and analysis of key components
Lustria, M. L. A., Cortese, J., Noar, S. M., & Glueckauf, R. L. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2008.08.023 · Cited by 566 · see citing works
The perfect search engine is not enough
Teevan, J. B., Alvarado, C., Ackerman, M. S., & Karger, D. R. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985745 · Cited by 547 · see citing works
van Uden‐Kraan, C. F., Drossaert, C. H. C., Taal, E., Shaw, B. R., Seydel, E. R., & van de Laar, M. A. F. J. (2008) · Qualitative Health Research · doi:10.1177/1049732307313429 · Cited by 465 · see citing works
Self-citationTheories of information behavior
Fisher, K. E. (2006) · Choice Reviews Online · doi:10.5860/choice.43-4366 · Cited by 459 · see citing works
Research-based web design & usability guidelines
Shneiderman, B. (2006) · Cited by 379 · see citing works
Information Seeking in Electronic Environments
Schamber, L., & Marchionini, G. (1996) · Journal of Education for Library and Information Science · doi:10.2307/40324289 · Cited by 309 · see citing works
Investigating evaluation frameworks for health information systems
Yusof, M. M., Papazafeiropoulou, A., Paul, R. J., & Stergioulas, L. K. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.08.004 · Cited by 304 · see citing works
Epstein, R. M., & Peters, E. M. (2009) · JAMA · doi:10.1001/jama.2009.984 · Cited by 303 · see citing works
Usability in the real world: assessing medical information technologies in patients’ homes
Kaufman, D. R., Patel, V. L., Hilliman, C. A., Morin, P. C., Pevzner, J., Weinstock, R. S., Goland, R., Shea, S., & Starren, J. B. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00056-x · Cited by 254 · see citing works
Internet Skills Performance Tests: Are People Ready for eHealth?
van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1581 · Cited by 163 · see citing works
Kelders, S. M., Van Gemert-Pijnen, J. E. W. C., Werkman, A., Nijland, N., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1624 · Cited by 156 · see citing works
Self-citationNijland, N., van Gemert-Pijnen, J., Boer, H., Steehouder, M. F., & Seydel, E. R. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.957 · Cited by 150 · see citing works
Self-citationeHealth Research from the User’s Perspective
Hesse, B. W., & Shneiderman, B. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.01.019 · Cited by 142 · see citing works
Chiu, T., & Eysenbach, G. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-73 · Cited by 142 · see citing works
Thinking About Thinking Aloud: A Comparison of TwoVerbal Protocols for Usability Testing
Krahmer (2004) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2004.828205 · Cited by 141 · see citing works
Theories of Information Behavior
(2006) · Journal of Documentation · doi:10.1108/00220410610666538 · Cited by 132 · see citing works
Elling, S., Lentz, L. R., & de Jong, M. D. T. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74444-3_25 · Cited by 95 · see citing works
Mediated health: sociotechnical practices of providing and using online health information
Mager, A. (2009) · New Media & Society · doi:10.1177/1461444809341700 · Cited by 70 · see citing works
Alpay, L., Verhoef, J., Xie, B., Te’eni, D., & Zwetsloot-Schonk, J. H. M. (2009) · Biomedical Informatics Insights · doi:10.4137/bii.s2223 · Cited by 69 · see citing works
Физика, А. (2025) · The MIT Press eBooks · doi:10.7551/mitpress/15082.003.0013 · Cited by 57 · see citing works
Critical Issues in eHealth Research
Atienza, A. A., Hesse, B. W., Baker, T. B., Abrams, D. B., Rimer, B. K., Croyle, R. T., & Volckmann, L. N. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.02.013 · Cited by 54 · see citing works
(2013) · Routledge eBooks · doi:10.4324/9780203810675 · Cited by 40 · see citing works
Verhoeven, F. (2009) · doi:10.3990/1.9789036528689 · Cited by 17 · see citing works
The Enhanced Restricted Focus Viewer
Tarasewich, P., Pomplun, M., Fillion, S., et al. (2005) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1901_4 · Cited by 15 · see citing works
Patient-Centered E-Health Design
Mauro, A. (2011) · IGI Global eBooks · doi:10.4018/978-1-60566-988-5.ch029 · Cited by 4 · see citing works
Geboortejaar kiesBeter.nl in feiten en cijfers
Gravestein, X., van der, H. E., Jmj, J., et al. (2006) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Web 2.0 chronic disease self-management for older adults: a systematic review
Stellefson, M., Chaney, B., Barry, A. E., Chavarria, E., Tennant, B., Walsh-Childers, K., Sriram, P. S., & Zagora, J. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2439 · Cited by 315 · see citing works
Methods for Human-Centered eHealth Development: Narrative Scoping Review
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2022) · Journal of Medical Internet Research · doi:10.2196/31858 · Cited by 67 · see citing works
Methods for Human-Centered eHealth Development: Narrative Scoping Review (Preprint)
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2021) · doi:10.2196/preprints.31858 · Cited by 67 · see citing works
Papadakos, J., Trang, A., Cyr, A. B., Abdelmutti, N., Giuliani, M. E., Snow, M., McCurdie, T., Pulandiran, M., Urowitz, S., & Wiljer, D. (2017) · JMIR Cancer · doi:10.2196/cancer.6933 · Cited by 12 · see citing works
- Authors
- Ossebaard, H. C., Seydel, E. R., & van Gemert-Pijnen, L.
- Affiliations
- National Institute for Public Health and the Environment (NL)
- University of Twente (NL)
- Corresponding author
- Ossebaard, Hans C.
- ORCID
- Ossebaard, Hans C.
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2012-06
- Volume
- 81
- Issue
- 6
- Pages
- 374-387
- ISSN
- 1386-5056 (ISSN-L)
- OpenAlex
- W2111288239
- Cited by
- 43
- References
- 43
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2022: 5 · 2021: 3 · 2020: 3 · 2019: 2 · 2018: 4 · 2017: 3 · 2016: 4 · 2015: 5 · 2014: 3 · 2013: 4 · 2012: 4
- FWCI
- 9.08
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Medical Informatics (publishedVersion)
- PubMed (publishedVersion)
- University of Twente Research Information (submittedVersion)
- Rivm Repository (Netherlands National Institute for Public Health and the Environment) (submittedVersion)
- Funders
- Rijksinstituut voor Volksgezondheid en Milieu
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- chronic disease, chronic disease patients, informal caregivers, online health information seeking, self-management, health decision-making, digital health literacy, personalized health information, arthritis, asthma, diabetes mellitus, usability improvement recommendations
- MeSH terms
- Adult, Aged, Chronic Disease, Educational Status, Female, Health Education, Humans, Male, Middle Aged, Netherlands, Patient Education as Topic, Prevalence, Internet, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review