Cognitive and usability engineering methods for the evaluation of clinical information systems.
Open accessAbstract
Increasingly healthcare policy and decision makers are demanding evidence to justify investments in health information systems. This demand requires an adequate evaluation of these systems. A wide variety of approaches and methodologies have been applied in assessing the impact of information systems in health care, ranging from controlled clinical trials to use of questionnaires and interviews with users. In this paper we describe methodological approaches which we have applied and refined for the past 10 years for the evaluation of health information systems. The approaches are strongly rooted in theories and methods from cognitive science and the emerging field of usability engineering. The focus is on assessing human computer interaction and in particular, the usability of computer systems in both laboratory and naturalistic settings. The methods described can be a part of the formative evaluation of systems during their iterative development, and can also complement more traditional assessment methods used in summative system evaluation of completed systems. The paper provides a review of the general area of systems evaluation with the motivation and rationale for methodological approaches underlying usability engineering and cognitive task analysis as applied to health information systems. This is followed by a detailed description of the methods we have applied in a variety of settings in conducting usability testing and usability inspection of systems such as computer-based patient records. Emerging trends in the evaluation of complex information systems are discussed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is often difficult simply to generalise from results obtained using RCTs when studying complex interventions such as eHealth technologies.
“It is therefore often difficult simply to generalise from results obtained using RCTs when studying complex interventions such as eHealth technologies [16],[17],[20].”
From Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation (2009) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1993) · Cited by 9,879 · see citing works
Norman, D. A. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,390 · see citing works
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,533 · see citing works
Book Review: Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,361 · see citing works
Nielsen, J. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260531 · Cited by 2,759 · see citing works
Nielsen, J. (1995) · Conference companion on Human factors in computing systems - CHI '95 · doi:10.1145/223355.223730 · Cited by 2,620 · see citing works
Klein, G. A. (2008) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x288385 · Cited by 2,266 · see citing works
(1986) · North-Holland eBooks · Cited by 1,883 · see citing works
User Centered System Design: New Perspectives on Human-Computer Interaction
Anderson, N. S., Norman, D. A., & Draper, S. W. (1988) · The American Journal of Psychology · doi:10.2307/1422802 · Cited by 1,851 · see citing works
Interaction Design: Beyond Human-Computer Interaction
Rogers, Y., Sharp, H., & Preece, J. J. (2002) · Open Research Online - ORO (The Open University) · Cited by 1,843 · see citing works
(1991) · Cited by 1,767 · see citing works
Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests
Rubin, J. N. (1994) · Cited by 1,750 · see citing works
Categorization of action slips
Norman, D. A. (1981) · Psychological Review · doi:10.1037/0033-295x.88.1.1 · Cited by 1,736 · see citing works
Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work
Vicente, K. J. (1999) · Cited by 1,686 · see citing works
Nielsen, J. (1999) · University of Maribor digital library (University of Maribor) · Cited by 1,620 · see citing works
(1987) · Andalas University Repository (Andalas University) · Cited by 1,584 · see citing works
(2003) · CRC Press eBooks · doi:10.1201/9780367804787 · Cited by 1,227 · see citing works
(2017) · CRC Press eBooks · doi:10.4324/9781315155401 · Cited by 1,199 · see citing works
(1995) · Choice Reviews Online · doi:10.5860/choice.32-2769 · Cited by 1,000 · see citing works
(2000) · Psychology Press eBooks · doi:10.4324/9781410605795 · Cited by 924 · see citing works
Preece, J. J., Rogers, Y., & Sharp, H. (2002) · Cited by 887 · see citing works
Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(92)90039-n · Cited by 797 · see citing works
The cognitive walkthrough method: a practitioner's guide
Wharton, C., Rieman, J., Lewis, C., & Polson, P. G. (1994) · John Wiley & Sons, Inc. eBooks · Cited by 713 · see citing works
(1993) · Lecture notes in computer science · doi:10.1007/3-540-57433-6 · Cited by 698 · see citing works
Rapid Development: Taming Wild Software Schedules
McConnell, S. (1996) · CERN Document Server (European Organization for Nuclear Research) · Cited by 611 · see citing works
Using usability heuristics to evaluate patient safety of medical devices
Zhang, J., Johnson, T. R., Patel, V. L., Paige, D. L., & Kubose, T. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00060-1 · Cited by 591 · see citing works
Self-citationEvaluation Methods in Medical Informatics
Friedman, C. P., & Wyatt, J. C. (1997) · Computers and medicine · doi:10.1007/978-1-4757-2685-5 · Cited by 450 · see citing works
Intelligent tutoring systems : lessons learned
Psotka, J., Massey, L., & Mutter, S. A. (1988) · Medical Entomology and Zoology · Cited by 447 · see citing works
Goldoff, A. C. (2007) · Routledge eBooks · doi:10.1081/e-epap2-263 · Cited by 405 · see citing works
Emerging paradigms of cognition in medical decision-making
Patel, V. L., Kaufman, D. R., & Arocha, J. F. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(02)00009-6 · Cited by 338 · see citing works
Self-citationHuman Computer Interaction In The New Millennium
Carroll, J. M. (2000) · Cited by 278 · see citing works
Hunter, L. (2003) · Document Design · doi:10.1075/dd.4.1.02hun · Cited by 262 · see citing works
Human-computer interaction in the new millennium
Weber-Wulff, D. (2002) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/638574.638592 · Cited by 255 · see citing works
Usability in the real world: assessing medical information technologies in patients’ homes
Kaufman, D. R., Patel, V. L., Hilliman, C. A., Morin, P. C., Pevzner, J., Weinstock, R. S., Goland, R., Shea, S., & Starren, J. B. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00056-x · Cited by 254 · see citing works
Self-citationKushniruk, A. W. (2002) · Computers in Biology and Medicine · doi:10.1016/s0010-4825(02)00011-2 · Cited by 241 · see citing works
Self-citationPatel, V. L., Kushniruk, A. W., Yang, S., & Yale, J. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070569 · Cited by 232 · see citing works
Self-citationEvaluating Health Care Information Systems: Methods and Applications
Anderson, J. G., Aydin, C. E., & Jay, S. J. (1993) · Cited by 180 · see citing works
Designing the user interface (videotape)
Shneiderman, B. (1987) · University of Maryland at College Park eBooks · Cited by 179 · see citing works
Kushniruk, A. W., Patel, V. L., & Cimino, J. J. (1997) · Journal of the American Medical Informatics Association · Cited by 173 · see citing works
Self-citationA Primer on Aspects of Cognition for Medical Informatics
Patel, V. L., Arocha, J. F., & Kaufman, D. R. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080324 · Cited by 170 · see citing works
Self-citationUsability testing in 2000 and beyond
Wichansky, A. M. (2000) · Ergonomics · doi:10.1080/001401300409170 · Cited by 159 · see citing works
A Comprehension-based Model of Web Navigation and Its Application to Web Usability Analysis
Kitajima, M., Blackmon, M. H., & Polson, P. G. (2000) · Springer eBooks · doi:10.1007/978-1-4471-0515-2_24 · Cited by 153 · see citing works
A framework for analyzing the cognitive complexity of computer-assisted clinical ordering
Horsky, J., Kaufman, D. R., Oppenheim, M. I., & Patel, V. L. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00062-5 · Cited by 137 · see citing works
Self-citationThe pluralistic usability walkthrough: coordinated empathies
Bias, R. G. (1994) · John Wiley & Sons, Inc. eBooks · Cited by 124 · see citing works
Designing human-centered distributed information systems
Zhang, J., Patel, V. L., Johnson, K. A., & Smith, J. W. (2002) · IEEE Intelligent Systems · doi:10.1109/mis.2002.1039831 · Cited by 121 · see citing works
Self-citationKushniruk, A. W. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1021 · Cited by 110 · see citing works
Self-citationKushniruk, A. W., Kaufman, D. R., Patel, V. L., Lévesque, Y., & Lottin, P. (1996) · PubMed · Cited by 98 · see citing works
Self-citationStudying the Human--Computer--Terminology Interface
Cimino, J. J., Patel, V. L., & Kushniruk, A. W. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080163 · Cited by 85 · see citing works
Self-citationMeans, B., & Gott, S. P. (1988) · Cited by 77 · see citing works
Mackenzie, C. F., Jefferies, N. J., Hunter, W. A., et al. (1996) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872096778827297 · Cited by 76 · see citing works
PalmCIS: A Wireless Handheld Application for Satisfying Clinician Information Needs
Chen, E. S., Mendonça, E. A., McKnight, L. K., Stetson, P. D., Lei, J., & Cimino, J. J. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1387 · Cited by 71 · see citing works
Kushniruk, A. W., & Patel, V. L. (1998) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(98)00106-3 · Cited by 70 · see citing works
Self-citationLang, M. K. (2002) · it - Information Technology · doi:10.1524/itit.2002.44.1.003 · Cited by 63 · see citing works
Cognitive evaluation: How to assess the usability of information technology in healthcare
Beuscart-Zéphir, M.-C., Brender, J., Beuscart, R., & Ménager-Depriester, I. (1997) · Computer Methods and Programs in Biomedicine · doi:10.1016/s0169-2607(97)00030-8 · Cited by 58 · see citing works
Kushniruk, A. W., Patel, C., Patel, V. L., & Cimino, J. J. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(00)00133-7 · Cited by 53 · see citing works
Self-citationUsability Inspection Methods: Executive Summary
Mack, R. L., & Nielsen, J. (1995) · Elsevier eBooks · doi:10.1016/b978-0-08-051574-8.50020-0 · Cited by 49 · see citing works
Elkin, P. L., Mohr, D. N., Tuttle, M. S., et al. (1997) · PubMed · Cited by 48 · see citing works
Cognitive evaluation of the user interface and vocabulary of an outpatient information system.
Kushniruk, A. W., Patel, V. L., Cimino, J. J., et al. (1996) · PubMed · Cited by 46 · see citing works
Self-citationKushniruk, A. W., & Patel, V. L. (1995) · PubMed · Cited by 35 · see citing works
Self-citationIterative usability testing: ensuring a usable clinical workstation.
Coble, J., Karat, J., Orland, M. J., et al. (1997) · PubMed · Cited by 28 · see citing works
Web Site Usability Handbook with Cdrom
Pearrow, M. (2000) · Charles River Media, Inc. eBooks · Cited by 23 · see citing works
Evaluation and Technology Assessment
Friedman, C. P., Owens, D. K., & Wyatt, J. C. (2001) · Computers in health care · doi:10.1007/978-0-387-21721-5_8 · Cited by 18 · see citing works
Kushniruk, A. W., Kan, M.-Y., McKeown, K. R., et al. (2002) · PubMed · Cited by 17 · see citing works
Self-citationBeuscart-Zéphir, M.-C., Watbled, L., Carpentier, A., Degroisse, M., & Alao, O. (2002) · PubMed Central · Cited by 17 · see citing works
Fox, J. (1993) · Methods of Information in Medicine · doi:10.1055/s-0038-1634949 · Cited by 12 · see citing works
The Wireless Web Usability Handbook
Pearrow, M. (2002) · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Wiklund ME. Usability in practice. New York: Academic Press; 1994.
- Ericsson K, Simon H. Protocol analysis: verbal reports as data. Cambridge, MA: Academic Press; 1993.
- Wharton C, Rieman J, Lewis C, Polson P. The cognitive walk- through method: a practitionerÕs guide. In: Nielsen J, Mack RL, editors. Usability inspection methods. New York: Wiley; 1994. p. 105-39.
- Weiss S. Handheld usability. New York: Wiley; 2002.
Cited by
Sorted by citations
Patient Portals and Patient Engagement: A State of the Science Review
Irizarry, T., DeVito Dabbs, A., & Curran, C. R. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4255 · Cited by 732 · see citing works
The think aloud method: a guide to user interface design
JASPERS, M., STEEN, T., BOS, C., & GEENEN, M. (2004) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.08.003 · Cited by 544 · see citing works
Review of health information technology usability study methodologies
Yen, P.-Y., & Bakken, S. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2010-000020 · Cited by 390 · see citing works
Harte, R., Glynn, L., Rodríguez-Molinero, A., Baker, P. M., Scharf, T., Quinlan, L. R., & ÓLaighin, G. (2017) · JMIR Human Factors · doi:10.2196/humanfactors.5443 · Cited by 399 · see citing works
Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307 · see citing works
TURF: Toward a unified framework of EHR usability
Zhang, J., & Walji, M. F. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.08.005 · Cited by 301 · see citing works
Georgsson, M., & Staggers, N. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv099 · Cited by 200 · see citing works
Comprehensive Analysis of a Medication Dosing Error Related to CPOE: Table 1
Horsky, J., Kuperman, G. J., & Patel, V. L. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1740 · Cited by 197 · see citing works
Self-citationPrimary Care Physicians’ Use of an Electronic Medical Record System: A Cognitive Task Analysis
Shachak, A., Hadas-Dayagi, M., Ziv, A., & Reis, S. (2009) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0892-6 · Cited by 188 · see citing works
Evaluating artificial intelligence in medicine: phases of clinical research
Park, Y., Jackson, G. P., Foreman, M. A., Gruen, D., Hu, J., & Das, A. K. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa033 · Cited by 150 · see citing works
Rundo, L., Pirrone, R., Vitabile, S., Sala, E., & Gambino, O. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103479 · Cited by 113 · see citing works
Applying heuristic evaluation to improve the usability of a telemedicine system
Tang, Z., Johnson, T. R., Tindall, R. D., & Zhang, J. (2006) · Telemedicine and e-Health · doi:10.1089/tmj.2006.12.24 · Cited by 102 · see citing works
Heuristic evaluation of paper-based Web pages: a simplified inspection usability methodology
Allen, M., Currie, L. M., Bakken, S., Patel, V. L., & Cimino, J. J. (2006) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.10.004 · Cited by 95 · see citing works
Self-citationKorpershoek, Y. J. G., Hermsen, S., Schoonhoven, L., Schuurmans, M. J., & Trappenburg, J. C. A. (2020) · Journal of Medical Internet Research · doi:10.2196/15449 · Cited by 78 · see citing works
physician information needs and electronic health records (EHRs): time to reengineer the clinic note
Koopman, R. J., Steege, L. M. B., Moore, J. L., Clarke, M. A., Canfield, S. M., Kim, M. S., & Belden, J. L. (2015) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2015.03.140244 · Cited by 90 · see citing works
Optimizing the language and format of guidelines to improve guideline uptake
Gupta, S., Rai, N., Bhattacharrya, O., Cheng, A. Y. Y., Connelly, K. A., Boulet, L.-P., Kaplan, A., Brouwers, M. C., & Kastner, M. (2016) · Canadian Medical Association Journal · doi:10.1503/cmaj.151102 · Cited by 82 · see citing works
Moradian, S., Krzyzanowska, M. K., Maguire, R., Morita, P. P., Kukreti, V., Avery, J., Liu, G., Cafazzo, J., & Howell, D. (2018) · JMIR Cancer · doi:10.2196/10932 · Cited by 56 · see citing works
Patel, T., Ivo, J., Faisal, S., McDougall, A., Carducci, J., Pritchard, S., & Chang, F. (2020) · Journal of Medical Internet Research · doi:10.2196/18073 · Cited by 49 · see citing works
Dimeff, L. A., Jobes, D. A., Chalker, S. A., Piehl, B. M., Duvivier, L. L., Lok, B. C., Zalake, M. S., Chung, J., & Koerner, K. (2020) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2018.05.005 · Cited by 49 · see citing works
A Review of Mobile Terminal-Based Applications for Self-Management of Patients with Diabetes
Tatara, N., Årsand, E., Nilsen, H., & Hartvigsen, G. (2009) · 2009 International Conference on eHealth, Telemedicine, and Social Medicine · doi:10.1109/etelemed.2009.14 · Cited by 42 · see citing works
A novel multimodal tool for telemonitoring patients with COPD
Sánchez-Morillo, D., Crespo, M., León, A., & Crespo Foix, L. F. (2015) · Informatics for Health and Social Care · doi:10.3109/17538157.2013.872114 · Cited by 25 · see citing works
Pflug, B., Kumarapeli, P., van Vlymen, J., Ammenwerth, E., & de Lusignan, S. (2010) · Informatics for Health and Social Care · doi:10.3109/17538150903358693 · Cited by 8 · see citing works
Co-Designing the MOSAIC mHealth App With Breast Cancer Survivors: User-Centered Design Approach
Nuseibeh, B. Z., Johns, S. A., Shih, P. C., Lewis, G. F., Gowan, T. M., & Jordan, E. J. (2024) · JMIR Formative Research · doi:10.2196/59426 · Cited by 5 · see citing works
Zhu, Y., Sui, D., Wang, Z., Hu, X., Gu, L., Qi, Y., Wu, T., Wang, L., Wei, Y., Tang, W., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791458 · Cited by 3 · see citing works
“Qualitatively, the superiority of the AI copilot stems from its ability to mitigate the information overload inherent in high-dimensional tabular datasets [34].”
Ingeman, K., Frostholm, L., Wellnitz, K. B., Wright, K., Frydendal, D. H., Onghena, P., & Rask, C. U. (2023) · JMIR Research Protocols · doi:10.2196/46927 · Cited by 3 · see citing works
“The method involved participants verbalizing their thoughts while using the web-based program [18,19].”
- Authors
- Kushniruk, A. W., & Patel, V. L.
- Affiliations
- York University (CA)
- Columbia University (US)
- Corresponding author
- Kushniruk, Andre W.
- ORCID
- Kushniruk, Andre W.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2004-02
- Volume
- 37
- Issue
- 1
- Pages
- 56-76
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2160271672
- Cited by
- 760
- References
- 68
- Citations by year
- 2026: 25 · 2025: 25 · 2024: 42 · 2023: 44 · 2022: 40 · 2021: 50 · 2020: 44 · 2019: 23 · 2018: 46 · 2017: 41 · 2016: 37 · 2015: 49 · 2014: 44 · 2013: 37 · 2012: 34
- FWCI
- 50.17
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Keywords
- health information systems, usability evaluation, usability engineering, cognitive task analysis, clinical information systems, usability testing, usability inspection, formative assessment, summative assessment, computer-based patient records, human-computer interaction, naturalistic assessment, laboratory evaluation
- MeSH terms
- Biomedical Engineering, Cognition, Medical Informatics, Medical Informatics Applications, Psychomotor Performance, Quality Assurance, Health Care, Software Validation, Task Performance and Analysis, Technology Assessment, Biomedical, User-Computer Interface
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
