A framework for analyzing the cognitive complexity of computer-assisted clinical ordering
Open accessAbstract
Computer-assisted provider order entry is a technology that is designed to expedite medical ordering and to reduce the frequency of preventable errors. This paper presents a multifaceted cognitive methodology for the characterization of cognitive demands of a medical information system. Our investigation was informed by the distributed resources (DR) model, a novel approach designed to describe the dimensions of user interfaces that introduce unnecessary cognitive complexity. This method evaluates the relative distribution of external (system) and internal (user) representations embodied in system interaction. We conducted an expert walkthrough evaluation of a commercial order entry system, followed by a simulated clinical ordering task performed by seven clinicians. The DR model was employed to explain variation in user performance and to characterize the relationship of resource distribution and ordering errors. The analysis revealed that the configuration of resources in this ordering application placed unnecessarily heavy cognitive demands on the user, especially on those who lacked a robust conceptual model of the system. The resources model also provided some insight into clinicians' interactive strategies and patterns of associated errors. Implications for user training and interface design based on the principles of human-computer interaction in the medical domain are discussed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Only five studies have examined overall safety and effectiveness.
“Although some measures of error showed improvement, overall safety and effectiveness have been addressed in only five studies.[21][22][23][24][25]”
From EHR Safety: The Way Forward to Safe and Effective Systems (2008) · cites this paper as [24]
The fourth study, in the same hospital, found no significant mortality change.
“The fourth study, conducted in the same hospital, showed no significant change in mortality.24”
From EHR Safety: The Way Forward to Safe and Effective Systems (2008) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Institute of Medicine (2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,174 · see citing works
Hutchins, E. (1995) · doi:10.7551/mitpress/1881.001.0001 · Cited by 10,801 · see citing works
To Err is Human: Building a Safer Health System
Hendee, W. R. (2001) · Journal of Vascular and Interventional Radiology · doi:10.1016/s1051-0443(01)70072-3 · Cited by 10,122 · see citing works
Wall, A. (2000) · British Journal of Healthcare Management · doi:10.12968/bjhc.2000.6.9.19311 · Cited by 5,909 · see citing works
Plans and Situated Actions: The Problem of Human-Machine Communication
Kantowitz, B. H., & Suchman, L. A. (1990) · The American Journal of Psychology · doi:10.2307/1423221 · Cited by 5,306 · see citing works
Carayannis, E. G. (2013) · Springer eBooks · doi:10.1007/978-1-4614-3858-8_100431 · Cited by 4,818 · see citing works
The Psychology of Human-Computer Interaction
Card, S. K., Moran, T. P., & Newell, A. (2018) · CRC Press eBooks · doi:10.1201/9780203736166 · Cited by 4,103 · see citing works
The psychology of human-computer interaction
Rogers, Y. (1984) · Applied Ergonomics · doi:10.1016/0003-6870(84)90205-9 · Cited by 1,970 · see citing works
Medication Errors and Adverse Drug Events in Pediatric Inpatients
Kaushal, R. (2001) · JAMA · doi:10.1001/jama.285.16.2114 · Cited by 1,917 · see citing works
User Centered System Design: New Perspectives on Human-Computer Interaction
Anderson, N. S., Norman, D. A., & Draper, S. W. (1988) · The American Journal of Psychology · doi:10.2307/1422802 · Cited by 1,851 · see citing works
Plans and situated actions: The problem of human-machine communication
Agre, P. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90078-e · Cited by 1,684 · see citing works
Hunt, D. L., Haynes, R. B., Hanna, S., & Smith, K. (1998) · JAMA · doi:10.1001/jama.280.15.1339 · Cited by 1,684 · see citing works
Things That Make Us Smart: Defending Human Attributes in the Age of the Machine
Robinson, R., & Norman, D. (1994) · Design Issues · doi:10.2307/1511661 · Cited by 1,528 · see citing works
The Impact of Computerized Physician Order Entry on Medication Error Prevention
Bates, D. W., Teich, J. M., Lee, J., Seger, D., Kuperman, G. J., Ma'Luf, N., Boyle, D., & Leape, L. (1999) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1999.00660313 · Cited by 1,260 · see citing works
Encyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,168 · see citing works
A Computer-Assisted Management Program for Antibiotics and Other Antiinfective Agents
Evans, R. S., Pestotnik, S. L., Classen, D. C., Clemmer, T. P., Weaver, L. K., Orme, J. F., Lloyd, J. F., & Burke, J. P. (1998) · New England Journal of Medicine · doi:10.1056/nejm199801223380406 · Cited by 1,060 · see citing works
Reducing the Frequency of Errors in Medicine Using Information Technology
Bates, D. W., Cohen, M., Leape, L. L., Overhage, J. M., Shabot, M. M., & Sheridan, T. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080299 · Cited by 959 · see citing works
(2000) · Psychology Press eBooks · doi:10.4324/9781410605795 · Cited by 924 · see citing works
The Nature of External Representations in Problem Solving
Zhang, J. (1997) · Cognitive Science · doi:10.1207/s15516709cog2102_3 · Cited by 897 · see citing works
Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(92)90039-n · Cited by 797 · see citing works
Views of Practicing Physicians and the Public on Medical Errors
Blendon, R. J., DesRoches, C. M., Brodie, M., Benson, J. M., Rosen, A. B., Schneider, E. C., Altman, D. E., Zapert, K., Herrmann, M. J., & Steffenson, A. E. (2002) · New England Journal of Medicine · doi:10.1056/nejmsa022151 · Cited by 770 · see citing works
Johnston, M. E., Langton, K. B., Haynes, R. B., & Mathieu, A. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-2-199401150-00007 · Cited by 713 · see citing works
A COMPUTER-ASSISTED MANAGEMENT PROGRAM FOR ANTIBIOTICS AND OTHER ANTIINFECTIVE AGENTS
Vans, R. S. C. E., Estotnik, S. T. L. P., Lassen, D. A. C. C., Lemmer, T., Eaver, L. I. K. W., Rme, J. A. F. O., Loyd, J. A. F. L., & Urke, J. O. P. B. (1998) · Cited by 692 · see citing works
Effects of Computerized Physician Order Entry on Prescribing Practices
Teich, J. M., Merchia, P., Schmiz, J. L., Kuperman, G. J., Spurr, C. D., & Bates, D. W. (2000) · Archives of Internal Medicine · doi:10.1001/archinte.160.18.2741 · Cited by 475 · see citing works
Guided Medication Dosing for Inpatients With Renal Insufficiency
Chertow, G. M. (2001) · JAMA · doi:10.1001/jama.286.22.2839 · Cited by 447 · see citing works
Linking Expertise and Naturalistic Decision Making
Salas, E., & Klein, G. A. (2001) · Psychology Press eBooks · doi:10.4324/9781410604200 · Cited by 386 · see citing works
A Randomized Trial of "Corollary Orders" to Prevent Errors of Omission
Overhage, J. M., Tierney, W. M., Zhou, X., & McDonald, C. J. (1997) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1997.0040364 · Cited by 375 · see citing works
Emerging paradigms of cognition in medical decision-making
Patel, V. L., Kaufman, D. R., & Arocha, J. F. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(02)00009-6 · Cited by 338 · see citing works
Self-citationCockton, G., DRAPER, S. W., & Weir, G. R. S. (1994) · Cited by 277 · see citing works
Computer-based Physician Order Entry: The State of the Art
Sittig, D. F., & Stead, W. W. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236142 · Cited by 273 · see citing works
Implementation of Physician Order Entry: User Satisfaction and Self-Reported usage Patterns
Lee, F., Teich, J. M., Spurr, C. D., & Bates, D. W. (1996) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1996.96342648 · Cited by 261 · see citing works
Analyzing Human-Computer Interaction as Distributed Cognition: The Resources Model
Wright, P., Fields, R. E., & Harrison, M. D. (2000) · Human-Computer Interaction · doi:10.1207/s15327051hci1501_01 · Cited by 237 · see citing works
Overhage, J. M., Perkins, S., Tierney, W. M., & McDonald, C. J. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080361 · Cited by 232 · see citing works
Patel, V. L., Kushniruk, A. W., Yang, S., & Yale, J. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070569 · Cited by 232 · see citing works
Self-citationSPECIAL SECTION: Adapting to New Technology in the Operating Room
Cook, R. I., & Woods, D. D. (1996) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872096778827224 · Cited by 227 · see citing works
Reducing Vancomycin Use Utilizing a Computer Guideline: Results of a Randomized Controlled Trial
Shojania, K. G., Yokoe, D. S., Platt, R. G., Fiskio, J. M., Ma'Luf, N., & Bates, D. W. (1998) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1998.0050554 · Cited by 211 · see citing works
Cognitive Engineering: Human Problem Solving with Tools
Woods, D., & Roth, E. (1988) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872088803000404 · Cited by 198 · see citing works
A Primer on Aspects of Cognition for Medical Informatics
Patel, V. L., Arocha, J. F., & Kaufman, D. R. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080324 · Cited by 170 · see citing works
Self-citationPhysician Satisfaction with Two Order Entry Systems
Murff, H. J., & Kannry, J. L. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080499 · Cited by 152 · see citing works
Applying Human Factors to the Design of Medical Equipment: Patient-Controlled Analgesia
Lin, L., Isla, R., Doniz, K., Harkness, H., Vicente, K. J., & Doyle, D. J. (1998) · Journal of Clinical Monitoring and Computing · doi:10.1023/a:1009928203196 · Cited by 141 · see citing works
Representing Clinical Guidelines in GLIF: Individual and Collaborative Expertise
Patel, V. L., Allen, V. G., Arocha, J. F., & Shortliffe, E. H. (1998) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1998.0050467 · Cited by 119 · see citing works
Self-citationKushniruk, A. W. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1021 · Cited by 110 · see citing works
Comparison of Time Spent Writing Orders on Paper with Computerized Physician Order Entry
Shu, K. M., Boyle, D., Spurr, C. D., Horsky, J., Heiman, H. L., Paula, O., John, L., & Bates, D. W. (2001) · Studies in health technology and informatics · doi:10.3233/978-1-60750-928-8-1207 · Cited by 108 · see citing works
Self-citationKushniruk, A. W., Kaufman, D. R., Patel, V. L., Lévesque, Y., & Lottin, P. (1996) · PubMed · Cited by 98 · see citing works
Self-citationPatel, V. L., Arocha, J. F., Diermeier, M., Greenes, R. A., & Shortliffe, E. H. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1002 · Cited by 88 · see citing works
Self-citationCognitive psychological studies of representation and use of clinical practice guidelines
Patel, V. L., Arocha, J. F., Diermeier, M., How, J., & Mottur-Pilson, C. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00165-4 · Cited by 87 · see citing works
Self-citationThe Strategic Use of Complex Computer Systems
Bhavnani, S. K., & John, B. E. (2000) · Human–Computer Interaction · doi:10.1207/s15327051hci1523_3 · Cited by 82 · see citing works
Physician order entry in U.S. hospitals.
Ash, J. S., Gorman, P., & Hersh, W. (1998) · PubMed Central · Cited by 75 · see citing works
Electronic prescribing in ambulatory practice: promises, pitfalls, and potential solutions.
Papshev, D., & Peterson, A. (2001) · PubMed · Cited by 66 · see citing works
Display-based competence: towards user models for menu-driven interfaces
Howes, A., & Payne, S. J. (1990) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(05)80067-7 · Cited by 57 · see citing works
Financial analysis projects clear returns from electronic medical records.
Schmitt, K. F., & Wofford, D. A. (2002) · PubMed · Cited by 56 · see citing works
A diffusion of innovations model of physician order entry.
Ash, J. S., Lyman, J. A., Carpenter, J., & Fournier, L. (2001) · PubMed Central · Cited by 54 · see citing works
(2002) · PubMed Central · Cited by 53 · see citing works
Burke, J. P., & Pestotnik, S. L. (1999) · Journal of Chemotherapy · doi:10.1179/joc.1999.11.6.530 · Cited by 53 · see citing works
Sparks of Innovation in Human-Computer Interaction
Shneiderman, B. (1993) · Greenwood Publishing Group Inc. eBooks · Cited by 51 · see citing works
Health Informatics: Linking Investment to Value
Stead, W. W., & Lorenzi, N. M. (1999) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1999.0060341 · Cited by 49 · see citing works
Increasing the success of physician order entry through human factors engineering.
Chan, W. (2002) · PubMed · Cited by 46 · see citing works
Sparks of innovation in human computer interaction
Dillon, A. P. (1994) · Library & Information Science Research · doi:10.1016/0740-8188(94)90036-1 · Cited by 42 · see citing works
Distributed Representation as a Principle for the Analysis of Cockpit Information Displays
Zhang, J. (1997) · International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0702_1 · Cited by 32 · see citing works
State-of-the-Art Review of Cognitive Task Analysis Techniques
Schraagen, S. F. C. J. M. (2000) · Psychology Press eBooks · doi:10.4324/9781410605795-40 · Cited by 29 · see citing works
The need for a national infrastructure to improve the rational use of therapeutics
Califf, R. M. (2002) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.699 · Cited by 21 · see citing works
Prescription drugs in nursing homes: managing costs and quality in a complex environment.
Mendelson, D., Ramchand, R. N., Abramson, R. G., et al. (2002) · PubMed · Cited by 20 · see citing works
Kaufman, D. R., Kushniruk, A. W., Yale, J., et al. (1999) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(99)00028-3 · Cited by 20 · see citing works
Self-citationMandating shorter work hours and enhancing patient safety: a new challenge for resident education.
Rohrich, R. J., Persing, J. A., & Phillips, L. (2003) · PubMed · doi:10.1097/01.prs.0000039706.18132.96 · Cited by 15 · see citing works
Darby, M. (1998) · Academic Medicine · doi:10.1097/00001888-199808000-00009 · Cited by 12 · see citing works
A Computerized Patient Information System in a Psychiatric Hospital
Modai, I., Ritsner, M. S., Silver, H., et al. (2002) · Psychiatric Services · doi:10.1176/appi.ps.53.4.476 · Cited by 9 · see citing works
Kuperman, G. J., Tom, C., Joanne, T., Teich, J. M., & Churchill, W. W. (1998) · Studies in health technology and informatics · doi:10.3233/978-1-60750-896-0-467 · Cited by 9 · see citing works
Mandating Shorter Work Hours and Enhancing Patient Safety: A New Challenge for Resident Education
Rohrich, R. J., Persing, J. A., & Phillips, L. G. (2003) · Plastic & Reconstructive Surgery · doi:10.1097/00006534-200301000-00071 · Cited by 9 · see citing works
Expertise in Laparoscopic Surgery: Anticipation and Affordances
Salas, E., & Klein, G. A. (2001) · Psychology Press eBooks · doi:10.4324/9781410604200-27 · Cited by 8 · see citing works
Balancing cost containment, quality care and IS investments.
Drazen, E. L. (1992) · PubMed · Cited by 2 · see citing works
Be all that you can be. Information technology and quality care go hand in hand.
Sandrick, K. M. (2000) · PubMed · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Cognitive and usability engineering methods for the evaluation of clinical information systems.
Kushniruk, A. W., & Patel, V. L. (2004) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2004.01.003 · Cited by 760 · see citing works
Self-citationJaspers, M. W. M. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.10.002 · Cited by 649 · see citing works
Laxmisan, A., Hakimzada, F., Sayan, O. R., Green, R. A., Zhang, J., & Patel, V. L. (2007) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.09.019 · Cited by 449 · see citing works
Self-citationComprehensive Analysis of a Medication Dosing Error Related to CPOE: Table 1
Horsky, J., Kuperman, G. J., & Patel, V. L. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1740 · Cited by 197 · see citing works
Self-citationEHR Safety: The Way Forward to Safe and Effective Systems
Walker, J. M., Carayon, P., Leveson, N., Paulus, R. A., Tooker, J., Chin, H., Bothe, A., & Stewart, W. F. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2618 · Cited by 113 · see citing works
- Authors
- Horsky, J., Kaufman, D. R., Oppenheim, M. I., & Patel, V. L.
- Affiliations
- Columbia University (US)
- Northwell Health (US)
- Corresponding author
- Horsky, Jan
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2003-02
- Volume
- 36
- Issue
- 1-2
- Pages
- 4-22
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2107373141
- Cited by
- 137
- References
- 71
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 8 · 2023: 3 · 2022: 2 · 2021: 4 · 2020: 5 · 2019: 4 · 2018: 3 · 2017: 5 · 2016: 8 · 2015: 4 · 2014: 9 · 2013: 9 · 2012: 7
- FWCI
- 15.47
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. National Library of Medicine (LM07079, LM-07079-09, T15LM007079)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- cognitive complexity, health information systems, user interface design, cognitive demand, conceptual model, user training
- MeSH terms
- Cognition, Database Management Systems, Decision Making, Computer-Assisted, Decision Support Techniques, Humans, Patient Admission, Software Validation, Statistics as Topic, Task Performance and Analysis, User-Computer Interface, Databases, Factual, Information Storage and Retrieval, Medical Records Systems, Computerized, Medical Errors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
