Toward a better understanding of task demands, workload, and performance during physician-computer interactions.
Open accessThe DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index.
Abstract
Objective To assess the relationship between (1) task demands and workload, (2) task demands and performance, and (3) workload and performance, all during physician-computer interactions in a simulated environment. Methods Two experiments were performed in 2 different electronic medical record (EMR) environments: WebCIS ( n = 12) and Epic ( n = 17). Each participant was instructed to complete a set of prespecified tasks on 3 routine clinical EMR-based scenarios: urinary tract infection (UTI), pneumonia (PN), and heart failure (HF). Task demands were quantified using behavioral responses (click and time analysis). At the end of each scenario, subjective workload was measured using the NASA-Task-Load Index (NASA-TLX). Physiological workload was measured using pupillary dilation and electroencephalography (EEG) data collected throughout the scenarios. Performance was quantified based on the maximum severity of omission errors. Results Data analysis indicated that the PN and HF scenarios were significantly more demanding than the UTI scenario for participants using WebCIS ( P < .01), and that the PN scenario was significantly more demanding than the UTI and HF scenarios for participants using Epic ( P < .01). In both experiments, the regression analysis indicated a significant relationship only between task demands and performance ( P < .01). Discussion Results suggest that task demands as experienced by participants are related to participants' performance. Future work may support the notion that task demands could be used as a quality metric that is likely representative of performance, and perhaps patient outcomes. Conclusion The present study is a reasonable next step in a systematic assessment of how task demands and workload are related to performance in EMR-evolving environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The NASA–Task Load Index (NASA-TLX) is a widely applied and valid tool used to measure workload,29,30,31,32,33,34 including the following 6 dimensions:”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
Chaudhry, B., Wang, J., Wu, S., Maglione, M., Mojica, W., Roth, E., Morton, S. C., & Shekelle, P. G. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-10-200605160-00125 · Cited by 3,159
Independent Component Analysis of Electroencephalographic Data
Makeig, S., Bell, A. J., Jung, T., & B, T. J. S. A. (1995) · neural information processing systems · Cited by 2,476
Bates, D. W. (1998) · JAMA · doi:10.1001/jama.280.15.1311 · Cited by 2,107
Task-evoked pupillary responses, processing load, and the structure of processing resources.
Beatty, J. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.91.2.276 · Cited by 1,959
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789
Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work
Vicente, K. J. (1999) · Cited by 1,686
Gevins, A. (1997) · Cerebral Cortex · doi:10.1093/cercor/7.4.374 · Cited by 1,623
Cycle-octave and related transforms in seismic signal analysis
Goupillaud, P., Grossmann, A., & Morlet, J. (1984) · Geoexploration · doi:10.1016/0016-7142(84)90025-5 · Cited by 1,504
Use of Electronic Health Records in U.S. Hospitals
Jha, A. K., DesRoches, C. M., Campbell, E. G., Donelan, K., Rao, S. R., Ferris, T. G., Shields, A., Rosenbaum, S., & Blumenthal, D. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0900592 · Cited by 1,473 · cited 1×
“8,9 18 These studies are concerning and suggest that thoughtful systems design is required for effective electronic health records system implementation, incorporating only key safety-and quality-enhancing features while minimizing additional workload placed on providers.”
BackgroundVicente, K. J. (1999) · CRC Press eBooks · doi:10.1201/b12457 · Cited by 1,389
Han, Y. Y., Carcillo, J. A., Venkataraman, S. T., Clark, R. S. B., Watson, R. S., Nguyen, T. C., Bayir, H., & Orr, R. A. (2005) · Pediatrics · doi:10.1542/peds.2005-1287 · Cited by 1,149
EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks.
Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., Olmstead, R. E., Tremoulet, P. D., & Craven, P. L. (2007) · PubMed · Cited by 1,050
Types of Unintended Consequences Related to Computerized Provider Order Entry
Campbell, E. M., Sittig, D. F., Ash, J. S., Guappone, K. P., & Dykstra, R. H. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2042 · Cited by 823
Hospital mortality in relation to staff workload: a 4-year study in an adult intensive-care unit
Tarnow-Mordi, W., Hau, C., Warden, A., & Shearer, A. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02478-8 · Cited by 735
Monitoring Working Memory Load during Computer-Based Tasks with EEG Pattern Recognition Methods
Gevins, A., Smith, M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R., & Rush, G. (1998) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872098779480578 · Cited by 535
Berka, C., Levendowski, D. J., Cvetinovic, M. M., Petrovic, M. M., Davis, G., Lumicao, M. N., Zivkovic, V. T., Popovic, M. V., & Olmstead, R. (2004) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1702_3 · Cited by 388
Karsh, B.-T., Holden, R. J., Alper, S. J., & Or, C. K. L. (2006) · Quality in Health Care · doi:10.1136/qshc.2005.015974 · Cited by 372
Health information technology: fallacies and sober realities
Karsh, B.-T., Weinger, M. B., Abbott, P. A., & Wears, R. L. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005637 · Cited by 365
Information Technology Comes to Medicine
Blumenthal, D., & Glaser, J. P. (2007) · New England Journal of Medicine · doi:10.1056/nejmhpr066212 · Cited by 354
Yurko, Y. Y., Scerbo, M. W., Prabhu, A. S., Acker, C. E., & Stefanidis, D. (2010) · Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0b013e3181e3f329 · Cited by 328 · cited 1×
“Considering all NASATLX dimensions as dependent variables simultaneously, the MANOVA revealed significant differences in both experiments (WebCIS: mental (46)> effort (40)> temporal (39)> frustration (36)> performance (26), with physical dimension not applicable [not selected during pairwise comparisons]; Epic: mental…”
MethodFoundations of Augmented Cognition
(2005) · CRC Press eBooks · doi:10.1201/9781482289701 · Cited by 315
Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions
(2009) · National Academies Press eBooks · doi:10.17226/12572 · Cited by 282
Categorizing the unintended sociotechnical consequences of computerized provider order entry
Ash, J. S., Sittig, D. F., Dykstra, R. H., Guappone, K., Carpenter, J. D., & Seshadri, V. (2007) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.05.017 · Cited by 267 · cited 2×
“01, with NASA-TLX scores UTI [30 (12)], PN [46 (11)], and HF [39 (9)] (Figure 2a); but not in the Epic experiment, with NASA-TLX scores UTI [41 (18)], PN [48 (17)], and HF [43 (20)] (Figure 2b).”
BackgroundDexter, P. R., Perkins, S. M., Maharry, K. S., Jones, K., & McDonald, C. J. (2004) · JAMA · doi:10.1001/jama.292.19.2366 · Cited by 237
Leape, L. L. (2009) · Clinica Chimica Acta · doi:10.1016/j.cca.2009.03.020 · Cited by 220
Smith, M. E., Gevins, A., Brown, H., Karnik, A., & Du, R. (2001) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872001775898287 · Cited by 214
Electronic Health Records’ Limited Successes Suggest More Targeted Uses
DesRoches, C. M., Campbell, E. G., Vogeli, C., Zheng, J., Rao, S. R., Shields, A. E., Donelan, K., Rosenbaum, S., Bristol, S. J., & Jha, A. K. (2010) · Health Affairs · doi:10.1377/hlthaff.2009.1086 · Cited by 212
Ash, J. S., Sittig, D. F., Dykstra, R., Campbell, E., & Guappone, K. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.07.015 · Cited by 207
Pupil Size, Information Overload, and Performance Differences
Peavler, W. S. (1974) · Psychophysiology · doi:10.1111/j.1469-8986.1974.tb01114.x · Cited by 202
Assessing Mental Workload During Laparoscopic Surgery
Carswell, C. M., Clarke, D., & Seales, W. B. (2005) · Surgical Innovation · doi:10.1177/155335060501200112 · Cited by 196
Weigl, M., Müller, A., Vincent, C., Angerer, P., & Sevdalis, N. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2011-000188 · Cited by 190
Assessment of workload using NASA Task Load Index in perianesthesia nursing.
Young, G., Zavelina, L., & Hooper, V. (2008) · Journal of PeriAnesthesia Nursing · doi:10.1016/j.jopan.2008.01.008 · Cited by 169
Pupillary responses in a pitch-discrimination task
Hahnemann, D., & Beatty, J. (1967) · Perception & Psychophysics · doi:10.3758/bf03210302 · Cited by 161
Effect of Hospitalist Workload on the Quality and Efficiency of Care
Elliott, D. J., Young, R. S., Brice, J., Aguiar, R., & Kolm, P. (2014) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.300 · Cited by 161
Russ, A. L., Zillich, A. J., Melton, B. L., Russell, S. A., Chen, S., Spina, J. R., Weiner, M., Johnson, E. G., Daggy, J. K., McManus, M. S., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002045 · Cited by 136
Holden, R. J. (2011) · Cognition, Technology & Work · doi:10.1007/s10111-010-0141-8 · Cited by 133
van den Hombergh, P., Künzi, B., Elwyn, G., van Doremalen, J., Akkermans, R., Grol, R., & Wensing, M. (2009) · BMC Health Services Research · doi:10.1186/1472-6963-9-118 · Cited by 124
Implementation Of Computerized Physician Order Entry In Seven Countries
Aarts, J., & Koppel, R. (2009) · Health Affairs · doi:10.1377/hlthaff.28.2.404 · Cited by 114
Morrison, I., & Smith, R. (2000) · BMJ · doi:10.1136/bmj.321.7276.1541 · Cited by 102
The significance of a usability evaluation of an emerging laboratory order entry system
Peute, L., & Jaspers, M. (2006) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.06.003 · Cited by 101
Cognitive Workload Assessment of Air Traffic Controllers Using Optical Brain Imaging Sensors
Ayaz, H., Willemi, B., Bunce, S. C., Shewokis, P. A., İzzetoğlu, K., Hah, S., Deshmukh, A., & Onaral, B. (2010) · Advances in human factors and ergonomics series · doi:10.1201/ebk1439835012-c3 · Cited by 97
Measuring the Workload of the Anesthesiologist
Gaba, D. M., & Lee, T. (1990) · Anesthesia & Analgesia · doi:10.1213/00000539-199010000-00006 · Cited by 94
Quantitative Assessment of Workload and Stressors in Clinical Radiation Oncology
Mazur, L. M., Mosaly, P. R., Jackson, M., Chang, S. X., Burkhardt, K. D., Adams, R. D., Jones, E. L., Hoyle, L., Xu, J., Rockwell, J., et al. (2012) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2012.01.063 · Cited by 91
Self-citationPhysician productivity and the ambulatory EHR in a large academic multi-specialty physician group
Cheriff, A. D., Kapur, A. G., Qiu, M., & Cole, C. L. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.04.006 · Cited by 85
Evaluation of an EEG workload model in an Aegis simulation environment
Berka, C., Levendowski, D. J., Ramsey, C. K., Davis, G., Lumicao, M. N., Stanney, K. M., Reeves, L. M., Regli, S. H., Tremoulet, P. D., & Stibler, K. M. (2005) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.598555 · Cited by 81
Carayon, P., Smith, P., Hundt, A. S., Kuruchittham, V., & Li, Q. (2009) · Behaviour & Information Technology · doi:10.1080/01449290701628178 · Cited by 78
Relating physician’s workload with errors during radiation therapy planning
Mazur, L. M., Mosaly, P. R., Hoyle, L., Jones, E. L., Chera, B. S., & Marks, L. B. (2013) · Practical Radiation Oncology · doi:10.1016/j.prro.2013.05.010 · Cited by 75
Self-citationSlagle, J. M., & Weinger, M. B. (2009) · Anesthesiology · doi:10.1097/aln.0b013e318194b1fc · Cited by 62
Measurement of Mental Workload in Clinical Medicine: A Review Study
Byrne, A. J. (2011) · Anesthesiology and Pain Medicine · doi:10.5812/kowsar.22287523.2045 · Cited by 59
Wachter, S. B., Johnson, K., Albert, R., Syroid, N., Drews, F., & Westenskow, D. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2123 · Cited by 57
Effects of task difficulty and time‐on‐task on mental workload
Haga, S., Shinoda, H., & Kokubun, M. (2002) · Japanese Psychological Research · doi:10.1111/1468-5884.00016 · Cited by 50
Estimating a Mental Workload Redline in a Simulated Air-to-Ground Combat Mission
Colle, H. A., & Reid, G. B. (2005) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1504_1 · Cited by 45
Doctors Are More Dangerous Than Gun Owners: A Rejoinder to Error Counting
Dekker, S. W. A. (2007) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872007x312423 · Cited by 41
Electronic health record functions differ between best and worst hospitals.
Elnahal, S. M., Joynt, K. E., Bristol, S. J., & Jha, A. K. (2011) · PubMed · Cited by 37
Supporting Real-Time Cognitive State Classification on a Mobile Individual
Dorneich, M. C., Whitlow, S. D., Mathan, S. A., Ververs, P. M., Erdogmus, D., Adami, A., Pavel, M., & Lan, T. (2007) · Journal of Cognitive Engineering and Decision Making · doi:10.1518/155534307x255618 · Cited by 36
Effects of CPOE on Provider Cognitive Workload: A Randomized Crossover Trial
Avansino, J., & Leu, M. G. (2012) · Pediatrics · doi:10.1542/peds.2011-3408 · Cited by 36
Berka, C., Levendowski, D. J., Westbrook, P. R., Davis, G., Lumicao, M. N., Olmstead, R. E., Popovic, M. V., Zivkovic, V. T., & Ramsey, C. K. (2005) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.597503 · Cited by 31
Kaushal, R., Shojania, K. G., & Bates, D. W. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.12.1409 · Cited by 30
Simulation in Health Care: Setting Realistic Expectations
Henriksen, K., & Patterson, M. (2007) · Journal of Patient Safety · doi:10.1097/pts.0b013e31813e5e39 · Cited by 27
Evaluation and prediction of on‐line maintenance workload in nuclear power plants
Liang, G., Lin, J., Hwang, S., Huang, F.-H., Yenn, T., & Hsu, C. (2008) · Human Factors and Ergonomics in Manufacturing & Service Industries · doi:10.1002/hfm.20136 · Cited by 24
Mosaly, P. R., Mazur, L. M., Jones, E. L., Hoyle, L., Zagar, T. M., Chera, B. S., & Marks, L. B. (2013) · Practical Radiation Oncology · doi:10.1016/j.prro.2013.02.007 · Cited by 18
Self-citationReducing Provider Cognitive Workload in CPOE Use: Optimizing Order Sets
Yiye, Z., Rema, P., & Levin, J. E. (2013) · Studies in health technology and informatics · doi:10.3233/978-1-61499-289-9-734 · Cited by 18
Perna, G. (2012) · PubMed · Cited by 15
Cluster tool module communication based on a high-level fieldbus
Lee, J.-H., Lee, T., & Park, J. (2003) · International Journal of Computer Integrated Manufacturing · doi:10.1080/09511920310000 · Cited by 8
Byun, S. N., & Choi, S. N. (2002) · Journal of Korean Institute of Industrial Engineers · Cited by 7
Harrison, M. I., & Koppel, R. (2010) · IGI Global eBooks · doi:10.4018/978-1-60566-030-1.ch003 · Cited by 7
Data security101: avoiding the list.
Perna, G. (2013) · PubMed · Cited by 2
(2000)
Research works printed but not linked: 3
- Warden, 2011, Health IT and Patient Safety: Builder Safer Systems for Better Care
- Hart, NASA-TLX Load Index (NASA-TLX); 20 years later, 50th Annual Proceedings of the Human Factors and Ergonomics Society Meeting, Santa Monica, CA, 2006, 904
- Tremoulet, 2006, Augmented cognition for tactical tomahawk weapons control system operators, 313
Cited by
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019) · arXiv · doi:10.48550/arxiv.1905.05134 · Cited by 318
The Validity of Physiological Measures to Identify Differences in Intrinsic Cognitive Load
Ayres, P., Lee, J. Y., Paas, F., & van Merriënboer, J. J. G. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.702538 · Cited by 222
Mazur, L. M., Mosaly, P. R., Moore, C., & Marks, L. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1709 · Cited by 127
Self-citationAssociation of electronic health record use with physician fatigue and efficiency
Khairat, S., Coleman, C., Ottmar, P., Jayachander, D. I., Bice, T., & Carson, S. S. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.7385 · Cited by 118
Health Information Technology in Healthcare Quality and Patient Safety: Literature Review
Feldman, S. S., Buchalter, S. E., & Hayes, L. (2018) · JMIR Medical Informatics · doi:10.2196/10264 · Cited by 98
Metadata
- Authors
- Mazur, L. M., Mosaly, P. R., Moore, C., Comitz, E., Yu, F., Falchook, A. D., Eblan, M. J., Hoyle, L. M., Tracton, G., Chera, B. S., & Marks, L. B.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-03-28
- Volume
- 23
- Issue
- 6
- Pages
- 1113-1120
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7784278
- OpenAlex
- W2334858943
- Cited by
- 45
- References
- 71
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 2 · 2022: 7 · 2021: 5 · 2020: 9 · 2019: 7 · 2018: 4 · 2017: 3 · 2016: 2
- FWCI
- 9.08
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- HighWire Press Open Archive (submittedVersion)
- UNC Libraries
- Funders
- Johns Hopkins University
- Agency for Healthcare Research and Quality (R18HS023458, R21 HS024062)
- Sidney Kimmel Comprehensive Cancer Center
- Sidney Kimmel Center for Prostate and Urologic Cancers
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- task demands, workload, physician performance, electronic health records, NASA-TLX, pupil dilation, EEG, omission errors, clinical decision support, simulated clinical scenarios, epic, quality metrics
- MeSH terms
- Efficiency, Electroencephalography, Humans, Physicians, Task Performance and Analysis, User-Computer Interface, Workload, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.