CoChat
Sign Up
Cited by 174Healthcare Technology and Patient MonitoringHuman-Automation Interaction and SafetyElectronic Health Records Systems

Validation of the Raw National Aeronautics and Space Administration Task Load Index (NASA-TLX) Questionnaire to Assess Perceived Workload in Patient Monitoring Tasks: Pooled Analysis Study Using Mixed Models

Open access
Said, Sadiq, Gozdzik, Malgorzata, Roche, Tadzio Raoul, Braun, Julia, Rössler, Julian, Kaserer, Alexander, Spahn, Donat R, Nöthiger, Christoph B +1 more·2020·Journal of Medical Internet Research
DOI 10.2196/19472PMC PMC7506540

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Patient monitoring is indispensable in any operating room to follow the patient’s current health state based on measured physiological parameters. Reducing workload helps to free cognitive resources and thus influences human performance, which ultimately improves the quality of care. Among the many methods available to assess perceived workload, the National Aeronautics and Space Administration Task Load Index (NASA-TLX) provides the most widely accepted tool. However, only few studies have investigated the validity of the NASA-TLX in the health care sector. Objective: This study aimed to validate a modified version of the raw NASA-TLX in patient monitoring tasks by investigating its correspondence with expected lower and higher workload situations and its robustness against nonworkload-related covariates. This defines criterion validity. Methods: In this pooled analysis, we evaluated raw NASA-TLX scores collected after performing patient monitoring tasks in four different investigator-initiated, computer-based, prospective, multicenter studies. All of them were conducted in three hospitals with a high standard of care in central Europe. In these already published studies, we compared conventional patient monitoring with two newly developed situation awareness–oriented monitoring technologies called Visual Patient and Visual Clot. The participants were resident and staff anesthesia and intensive care physicians, and nurse anesthetists with completed specialization qualification. We analyzed the raw NASA-TLX scores by fitting mixed linear regression models and univariate models with different covariates. Results: We assessed a total of 1160 raw NASA-TLX questionnaires after performing specific patient monitoring tasks. Good test performance and higher self-rated diagnostic confidence correlated significantly with lower raw NASA-TLX scores and the subscores (all P<.001). Staff physicians rated significantly lower workload scores than residents (P=.001), whereas nurse anesthetists did not show any difference in the same comparison (P=.83). Standardized distraction resulted in higher rated total raw NASA-TLX scores (P<.001) and subscores. There was no gender difference regarding perceived workload (P=.26). The new visualization technologies Visual Patient and Visual Clot resulted in significantly lower total raw NASA-TLX scores and all subscores, including high self-rated performance, when compared with conventional monitoring (all P<.001). Conclusions: This study validated a modified raw NASA-TLX questionnaire for patient monitoring tasks. The scores obtained correctly represented the assumed influences of the examined covariates on the perceived workload. We reported high criterion validity. The NASA-TLX questionnaire appears to be a reliable tool for measuring subjective workload. Further research should focus on its applicability in a clinical setting.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/19472

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • No sex-related differences have been reported for perception of fatigue or SPL in people with PD.

    “Sex was not included as a covariate as no sex-related differences have been reported for perception of fatigue (Said et al., 2020) or SPL in people with PD (Fox & Ramig, 1997; Levitt et al., 2015).”

    From Perception of Physical Demand, Mental Demand, and Performance: A Comparison of Two Voice Interventions for Parkinson's Disease (2022) · cites this paper as Said et al., 2020

    The Methods (page unavailable) say the authors found no evidence of sex differences in perceived workload, and the Results (page unavailable) report no gender difference in NASA-TLX subscores. That supports the sex-difference aspect of the citation, but Said et al. measured workload during patient-monitoring tasks, not perceived fatigue. The SPL finding is attributed to other citations in the sentence and is not assessed here. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Raw NASA-TLX scores are the most commonly reported in the literature.

    “This study used the raw NASA-TLX scores, which are most commonly reported in the literature (Said et al., 2020).”

    From Perception of Physical Demand, Mental Demand, and Performance: A Comparison of Two Voice Interventions for Parkinson's Disease (2022) · cites this paper as Said et al., 2020

    The Introduction (page unavailable) calls NASA-TLX a widely accepted workload tool and describes raw and weighted scoring, but does not say raw scores are the most commonly reported in the literature. Searches for reporting frequency and comparisons with weighted scores found no passage establishing that claim. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Said, S., Gozdzik, M., Roche, T. R., Braun, J., Rössler, J., Kaserer, A., Spahn, D. R., Nöthiger, C. B., & Tscholl, D. W.
Affiliations
  • University Hospital Zurich (CH)
  • University of Zurich (CH)
Corresponding author
Said, Sadiq
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2020-09-07
Volume
22
Issue
9
Pages
e19472
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/19472
PMC
PMC7506540
OpenAlex
W3048666924
Cited by
173
References
58
Citations by year
2026: 47 · 2025: 40 · 2024: 28 · 2023: 28 · 2022: 23 · 2021: 6 · 2020: 1
FWCI
13.25
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • Universität Zürich
  • Universitätsspital Zürich
Topics
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
NASA-TLX, perceived workload, patient monitoring, criterion validity, operating room, situational awareness, visualization techniques, diagnostic confidence, distraction, anesthesiology residents, nurse anesthetists, intensive care physicians, mixed models, pooled analysis, gender differences, cognitive resources, quality of care, human performance
MeSH terms
Female, Humans, Male, Monitoring, Physiologic, Prospective Studies, Surveys and Questionnaires, Switzerland, Task Performance and Analysis, Workload

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review