Toward a better understanding of task demands, workload, and performance during physician-computer interactions.
Open accessThe DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index.
Abstract
Objective To assess the relationship between (1) task demands and workload, (2) task demands and performance, and (3) workload and performance, all during physician-computer interactions in a simulated environment. Methods Two experiments were performed in 2 different electronic medical record (EMR) environments: WebCIS ( n = 12) and Epic ( n = 17). Each participant was instructed to complete a set of prespecified tasks on 3 routine clinical EMR-based scenarios: urinary tract infection (UTI), pneumonia (PN), and heart failure (HF). Task demands were quantified using behavioral responses (click and time analysis). At the end of each scenario, subjective workload was measured using the NASA-Task-Load Index (NASA-TLX). Physiological workload was measured using pupillary dilation and electroencephalography (EEG) data collected throughout the scenarios. Performance was quantified based on the maximum severity of omission errors. Results Data analysis indicated that the PN and HF scenarios were significantly more demanding than the UTI scenario for participants using WebCIS ( P < .01), and that the PN scenario was significantly more demanding than the UTI and HF scenarios for participants using Epic ( P < .01). In both experiments, the regression analysis indicated a significant relationship only between task demands and performance ( P < .01). Discussion Results suggest that task demands as experienced by participants are related to participants' performance. Future work may support the notion that task demands could be used as a quality metric that is likely representative of performance, and perhaps patient outcomes. Conclusion The present study is a reasonable next step in a systematic assessment of how task demands and workload are related to performance in EMR-evolving environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The NASA–Task Load Index (NASA-TLX) is a widely applied and valid tool used to measure workload,29,30,31,32,33,34 including the following 6 dimensions:”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
Chaudhry, B., Wang, J., Wu, S., Maglione, M., Mojica, W., Roth, E., Morton, S. C., & Shekelle, P. G. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-10-200605160-00125 · Cited by 3,159
Independent Component Analysis of Electroencephalographic Data
Makeig, S., Bell, A. J., Jung, T., & B, T. J. S. A. (1995) · neural information processing systems · Cited by 2,476
Bates, D. W. (1998) · JAMA · doi:10.1001/jama.280.15.1311 · Cited by 2,107 · cited 1×
“1,6,11,13,16,33,69 Performance was quantified using the following set of metrics: Errors: Not ordering predetermined tests and/or medications specified in clinical scenarios as shown in Table 1 was considered an omission error (see Table 1 for severity scores as related to studied tasks).”
MethodTask-evoked pupillary responses, processing load, and the structure of processing resources.
Beatty, J. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.91.2.276 · Cited by 1,959
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789
Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work
Vicente, K. J. (1999) · Cited by 1,686
Gevins, A. (1997) · Cerebral Cortex · doi:10.1093/cercor/7.4.374 · Cited by 1,623
Cycle-octave and related transforms in seismic signal analysis
Goupillaud, P., Grossmann, A., & Morlet, J. (1984) · Geoexploration · doi:10.1016/0016-7142(84)90025-5 · Cited by 1,504
Research works printed but not linked: 4
- 41. Lee KK Kerns K Bones R et al. . Development and validation of the controller acceptance rating scale (CARS): Results of empirical research. 4th USA/Europe Air Traffic Management Research and Development Seminar (ATM-2001) . Santa Fe, NM, 2001 . [Google Scholar]
- 42. Hoffman E Pene N Rognin L et al. . Introducing a new spacing instruction, impact of spacing tolerance on flight crew activity. Proceedings of the 47th Annual Meeting of the Human Factors and Ergonomics Society . Santa Monica, CA, 2003 . [Google Scholar]
- 50. Warden GL Bagian JP . Health IT and Patient Safety: Builder Safer Systems for Better Care . Washington, DC: : Institute of Medicine, National Academies Press; ; 2011. . [PubMed] [Google Scholar]
- 53. Hart SG . NASA-TLX Load Index (NASA-TLX); 20 years later . 50th Annual Proceedings of the Human Factors and Ergonomics Society Meeting, Santa Monica, CA ; 2006 : 904 – 908 . [Google Scholar]
Cited by
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019) · arXiv · doi:10.48550/arxiv.1905.05134 · Cited by 318
“A few user studies have evaluated prototypical Electronic Health Recording systems (Mazur et al., 2016) or associated visualization prototypes (Ghassemi et al., 2018a) for efficacy of facilitating diverse clinical workflows (Nolan et al., 2017).”
BackgroundThe Validity of Physiological Measures to Identify Differences in Intrinsic Cognitive Load
Ayres, P., Lee, J. Y., Paas, F., & van Merriënboer, J. J. G. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.702538 · Cited by 222
Mazur, L. M., Mosaly, P. R., Moore, C., & Marks, L. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1709 · Cited by 127
Self-citationAssociation of electronic health record use with physician fatigue and efficiency
Khairat, S., Coleman, C., Ottmar, P., Jayachander, D. I., Bice, T., & Carson, S. S. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.7385 · Cited by 118
Health Information Technology in Healthcare Quality and Patient Safety: Literature Review
Feldman, S. S., Buchalter, S. E., & Hayes, L. (2018) · JMIR Medical Informatics · doi:10.2196/10264 · Cited by 99
“The greatest areas of use were around alerts [30,31,44,56,58], clinical decision support [39,44,47,56], implementation [10,32,37,38,56], interface design [26,34,42,45,56,59], and customized health IT solutions [29,30,32,34,46-50,56,58,59].”
A systematic review of automatic text summarization for biomedical literature and EHRs
Wang, M., Wang, M., Yu, F., Yang, Y., Walker, J., & Mostafa, J. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab143 · Cited by 63
Self-citationA mixed-methods evaluation framework for electronic health records usability studies
Khairat, S., Coleman, C., Newlin, T., Rand, V., Ottmar, P., Bice, T., & Carson, S. S. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103175 · Cited by 50
“Prior EHR usability work has also employed eye-tracking technology [23] as well as simulation-based testing [24, 25], but our study will provide the largest dataset with regard to physicians in critical care settings where the stakes are quite high.”
MethodObjective measures of workload in healthcare: a narrative review
Fishbein, D., Nambiar, S., McKenzie, K., Mayorga, M. E., Miller, K., Tran, K., Schubel, L., Agor, J., Kim, T., & Capan, M. (2019) · International Journal of Health Care Quality Assurance · doi:10.1108/ijhcqa-12-2018-0288 · Cited by 55
Tscholl, D. W., Rössler, J., Said, S., Kaserer, A., Spahn, D. R., & Nöthiger, C. B. (2020) · Sensors · doi:10.3390/s20072112 · Cited by 51
Pelletier, J. H., Watson, K. L., Michel, J., McGregor, R. S., & Rush, S. Z. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf068 · Cited by 44
“33,34,38,39,69,70 Burnout was assessed using the burnout index of the Mini-Z survey.”
Garot, O., Rössler, J., Pfarr, J., Ganter, M. T., Spahn, D. R., Nöthiger, C. B., & Tscholl, D. W. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1032-4 · Cited by 44
ClinicalVis: Supporting Clinical Task-Focused Design Evaluation
Ghassemi, M., Pushkarna, M., Wexler, J., Johnson, J., & Varghese, P. (2018) · arXiv · doi:10.48550/arxiv.1810.05798 · Cited by 26
“and cognitive workload in 17 clinical participants performing a set of clinical tasks on simulated patients in three clinical diagnosis scenarios: urinary tract infection, pneumonia and heart failure [43]. The authors’ results suggest that task demands as experienced by HCPs (e.g., needing more clicks, requiring more…”
BackgroundExamining Mental Workload Relating to Digital Health Technologies in Health Care: Systematic Review
Kremer, L., Lipprandt, M., Röhrig, R., & Breil, B. (2022) · Journal of Medical Internet Research · doi:10.2196/40946 · Cited by 36
“A total of 18 studies applied subjective methods [34-36, 38, 40-42 40-42”
InfluentialMosaly, P. R., Mazur, L. M., Yu, F., Guo, H., Derek, M., Laidlaw, D. H., Moore, C., Marks, L. B., & Mostafa, J. (2017) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2017.1365459 · Cited by 32
“…indicating that higher task demands lead to increase in mental effort that increases errors in various domains (Da Silva, 2014; Stetina, Groves, & Pafford, 2005) including physician – EHR interactions (Ariza, Kalra, & Potts, 2015; Mazur et al., 2016; Mosaly et al., 2016; Zhang & Walji, 2011).”
BackgroundInfluentialSelf-citationPhysician activity during outpatient visits and subjective workload
Calvitti, A., Hochheiser, H., Ashfaq, S., Bell, K., Chen, Y., El-Kareh, R. E., Gabuzda, M. T., Liu, L., Mortensen, S., Pandey, B. B., et al. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.03.011 · Cited by 29
The use of analytic hierarchy process for measuring the complexity of medical diagnosis
Ben‐Assuli, O., Kumar, N., Arazy, O., & Shabtai, I. (2019) · Health Informatics Journal · doi:10.1177/1460458218824708 · Cited by 26
“Typically, clinical measures have tended to emphasize the number of chronic conditions, medications and even the use of medical resources/costs as a proxy for complexity (29, 30).”
BackgroundDesign for stress, fatigue, and workload management
Nuamah, J. K., & Mehta, R. K. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-816427-3.00011-7 · Cited by 24
Tran, B. D., Rosenbaum, K., & Zheng, K. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa345 · Cited by 13
Why Are Explainable AI Methods for Prostate Lesion Detection Rated Poorly by Radiologists?
Gulum, M. A., Trombley, C. M., Özen, M., Esen, E., Akşamoğlu, M., & Kantardzic, M. M. (2024) · Applied Sciences · doi:10.3390/app14114654 · Cited by 16
“Furthermore, a study assessed the relationship between task demands and workload during physician–computer interaction in a simulated environment [17], while another proposed a visualization system for electronic health records and evaluated its accuracy and confidence in treatment decisions using…”
BackgroundToy, S., Huh, D. D., Materi, J. D., Nanavati, J., & Schwengel, D. A. (2022) · Medical Education Online · doi:10.1080/10872981.2021.2016357 · Cited by 12
“[49] Mazur LM, Mosaly PR, Moore C, et al.”
BackgroundMazur, L. M., Marks, L. B., McLeod, R., Karwowski, W., Mosaly, P. R., Tracton, G. S., Adams, R., Hoyle, L., Das, S. K., & Chera, B. S. (2018) · Advances in Radiation Oncology · doi:10.1016/j.adro.2018.01.002 · Cited by 12
Self-citationModelling users' trust in online health rumours: an experiment-based study in China
Deng, S., Fu, S., Yong, L., & Li, H. (2021) · Information Research · doi:10.47989/irpaper890 · Cited by 10
“Our choice is consistent with prior literature on health information, for which the number of experiment subjects is 7-29 (Bolgva et al., 2017; Jean et al., 2015; Mazur et al., 2016; Wu and Li, 2016).”
MethodRadaviciute, I., Hunn, C. A., Lunkiewicz, J., Milovanovic, P. D., Willms, J. F., Nöthiger, C. B., Keller, E., Tscholl, D. W., & Gasciauskaite, G. (2024) · Scientific Reports · doi:10.1038/s41598-024-72338-7 · Cited by 9
Orlando, E. M., Nenna, F., Zanardi, D., Buodo, G., Mingardi, M., Sarlo, M., & Gamberini, L. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103516 · Cited by 14
Developing and assessing electronic checklists for safety mindfulness, workload, and performance
Tracton, G. S., Mazur, L. M., Mosaly, P. R., Marks, L. B., & Das, S. K. (2018) · Practical Radiation Oncology · doi:10.1016/j.prro.2018.05.001 · Cited by 9
Self-citationKremer, L., Gehrmann, J., Rainer, R., & Breil, B. (2023) · Applied Ergonomics · doi:10.1016/j.apergo.2023.104150 · Cited by 11
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00013-3 · Cited by 13
Nambiar, S., Mayorga, M. E., & Liu, Y. (2022) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2022.2100522 · Cited by 7
“clinician performance and staffing decisions) in healthcare systems (Mazur et al., 2016; Upenieks et al., 2007).”
BackgroundClassification of EEG Features for Prediction of Working Memory Load
Abrantes, A., Comitz, E., Mosaly, P. R., & Mazur, L. M. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-41947-3_12 · Cited by 8
Self-citationGermano, D., Sciaraffa, N., Ronca, V., Giorgi, A., Trulli, G., Borghini, G., Flumeri, G. D., Babiloni, F., & Aricò, P. (2023) · Applied Sciences · doi:10.3390/app132312800 · Cited by 9
“It is worth noting to highlight that the phenomenon of covariate shift persists regardless of the amount of new testing data, as depicted in Figures 9 and 13.”
BackgroundMosaly, P. R., Adams, R., Tracton, G. S., Dooley, J., Adapa, K., Nuamah, J. K., Marks, L. B., & Mazur, L. M. (2020) · Practical Radiation Oncology · doi:10.1016/j.prro.2020.07.002 · Cited by 7
Self-citationJu, S., Kim, Y. J., Ausin, M. S., Mayorga, M. E., & Chi, M. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671407 · Cited by 8
Nuamah, J. K., Mosaly, P. R., Adams, R., Adapa, K., Chera, B. S., Marks, L. B., & Mazur, L. M. (2020) · Advances in Radiation Oncology · doi:10.1016/j.adro.2020.09.012 · Cited by 3
“Prior research suggests that the performance of health care providers, including radiation oncology professionals, often depends on their level of workload (WL) and situation awareness (SA).(1-5) Radiation therapy technologists (RTTs) are front-line professionals who have”
BackgroundSelf-citationCOG Tool: An Automated Cognitive Measurement of Workload for Mobile Event Logging
Saleh, A. M., & Enaizan, O. (2022) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v16i20.33363 · Cited by 5
Zamudio, J., Catchpole, K., Kanji, F. F., Anger, J. T., & Cohen, T. (2022) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181322661326 · Cited by 5
“Additionally, pupillary responses (Mazur et al., 2016) and eye tracking (Wagner et al., 2021) have the potential to measure workload and attention during tasks that require the use of technology to assess the feasibility and usability of those tools.”
BackgroundMarks, L. B. (2016) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2015.11.027 · Cited by 3
Self-citationMazur, L. M., Adams, R., Mosaly, P. R., Nuamah, J. K., Adapa, K., & Marks, L. B. (2020) · Practical Radiation Oncology · doi:10.1016/j.prro.2020.08.005 · Cited by 2
Self-citationAre Post-Hoc Explanation Methods for Prostate Lesion Detection Effective for Radiology End Use?
Gulum, M. A., Trombley, C. M., Özen, M., & Kantardzic, M. M. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00191 · Cited by 3
“[16] access the relationship be-tween task demands and workload during physician-computer interaction in a simulated study.”
BackgroundAlghamdi, S. M. (2024) · Journal of the Pakistan Medical Association · doi:10.47391/jpma.11216 · Cited by 1
Lyu, M., Cheng, P., Li, X., Balian, P., Bian, J., & Wu, Y. (2024) · Europe PMC (PubMed Central) · doi:10.48550/arxiv.2403.13089 · Cited by 13
“However, the growing burden of clinical documentation has been a significant challenge in healthcare, contributing to physician burnout, reducing the efficiency of care delivery, and potentially compromising patient safety 2–4 .”
BackgroundLi, Y., Guo, X., Hsu, C. W., Liu, X., & Vogel, D. (2019) · JMIR Medical Informatics · doi:10.2196/11663
Castellano, N., Mosaly, P. R., & Mazur, L. M. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-20473-0_3
Self-citationTolentino, D. A. (2020) · UA Campus Repository (The University of Arizona) · Cited by 1
Kremer, L., Lipprandt, M., Röhrig, R., & Breil, B. (2022) · doi:10.2196/preprints.40946
Garot, O., Rössler, J., Pfarr, J., Ganter, M. T., Spahn, D. R., Nöthiger, C. B., & Tscholl, D. W. (2020) · Figshare · doi:10.6084/m9.figshare.12033102
Metadata
- Authors
- Mazur, L. M., Mosaly, P. R., Moore, C., Comitz, E., Yu, F., Falchook, A. D., Eblan, M. J., Hoyle, L. M., Tracton, G., Chera, B. S., & Marks, L. B.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-03-28
- Volume
- 23
- Issue
- 6
- Pages
- 1113-1120
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7784278
- OpenAlex
- W2334858943
- Cited by
- 45
- References
- 71
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 2 · 2022: 7 · 2021: 5 · 2020: 9 · 2019: 7 · 2018: 4 · 2017: 3 · 2016: 2
- FWCI
- 9.08
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- HighWire Press Open Archive (submittedVersion)
- UNC Libraries
- Funders
- Johns Hopkins University
- Agency for Healthcare Research and Quality (R18HS023458, R21 HS024062)
- Sidney Kimmel Comprehensive Cancer Center
- Sidney Kimmel Center for Prostate and Urologic Cancers
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- task demands, workload, physician performance, electronic health records, NASA-TLX, pupil dilation, EEG, omission errors, clinical decision support, simulated clinical scenarios, epic, quality metrics
- MeSH terms
- Efficiency, Electroencephalography, Humans, Physicians, Task Performance and Analysis, User-Computer Interface, Workload, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.