CoChat
Sign Up
Cited by 94Visual and Cognitive Learning ProcessesInnovations in Medical EducationClinical Reasoning and Diagnostic Skills

Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool

Open access
Szulewski, Adam, Gegenfurtner, Andreas, Howes, Daniel W., Sivilotti, Marco L. A., van Merriënboer, Jeroen J. G.·2017·Advances in Health Sciences Education
DOI 10.1007/s10459-016-9725-2
Open this paper in CoChat — free accountOpen PDF

Abstract

In general, researchers attempt to quantify cognitive load using physiologic and psychometric measures. Although the construct measured by both of these metrics is thought to represent overall cognitive load, there is a paucity of studies that compares these techniques to one another. The authors compared data obtained from one physiologic tool (pupillometry) to one psychometric tool (Paas scale) to explore whether they actually measured the construct of cognitive load as purported. Thirty-two participants with a range of resuscitation medicine experience and expertise completed resuscitation-medicine based multiple-choice-questions as well as arithmetic questions. Cognitive load, as measured by both tools, was found to be higher for the more difficult questions as well as for questions that were answered incorrectly (p < 0.001). The group with the least medical experience had higher cognitive load than both the intermediate and experienced groups when answering domain-specific questions (p = 0.023 and p = 0.003 respectively for the physiologic tool; p = 0.006 and p < 0.001 respectively for the psychometric tool). There was a strong positive correlation (Spearman's ρ = 0.827, p < 0.001 for arithmetic questions; Spearman's ρ = 0.606, p < 0.001 for medical questions) between the two cognitive load measurement tools. These findings support the validity argument that both physiologic and psychometric metrics measure the construct of cognitive load.

AI snapshot

Written from the paper’s full text.

TL;DR

This study compared a physiologic cognitive-load measure (pupillometry-derived pupillary change index, PCI) with a psychometric measure (Paas mental-effort scale) in 32 participants with varying resuscitation-medicine experience. The two measures correlated strongly and showed expected patterns by question difficulty, correctness, and training level, supporting their validity as surrogate markers of overall cognitive load.

Key findings

  • PCI and Paas scale correlated strongly overall: arithmetic questions Pearson’s r = 0.675 (p < 0.001), Spearman’s q = 0.827 (p < 0.001); medical questions Pearson’s r = 0.542 (p < 0.001), Spearman’s q = 0.606 (p < 0.001). Correlations persisted within novice, intermediate, and experienced subgroups (Table 1).
  • Difficult questions produced higher cognitive load than easy questions: PCI 123.40 vs. 45.14 and Paas 5.40 vs. 3.62 (both p < 0.001). Incorrectly answered questions produced higher load than correct answers: PCI 152.51 vs. 66.04 and Paas 5.89 vs. 4.22 (both p < 0.001).
  • Novices showed higher PCI and Paas values than experienced participants for arithmetic questions (PCI p = 0.005; Paas p = 0.040) and medical questions (PCI p = 0.003; Paas p < 0.001); novices also had higher medical-question values than intermediates (PCI p = 0.023; Paas p = 0.006), with no intermediate–experienced difference. On medical questions, novices performed worse than both other groups (p < 0.001), and intermediates outperformed experienced participants (p = 0.044).
  • No significant PCI or Paas difference was found between arithmetic and medical questions (PCI p = 0.42; Paas p = 0.43). Peak pupil size differed between novices and experienced participants only for medical questions (p = 0.002), not arithmetic questions.

Methods

  • Convenience sample of 32 participants recruited September–November 2014: 13 novices (first two years of medical school), 9 intermediate (senior residents/early attending physicians), and 10 experienced attending physicians with >10 years of clinical experience. Mean age was 34.1 years (SD = 10.8).
  • Participants answered the same 12 resuscitation-medicine multiple-choice questions and 4 arithmetic questions, classified a priori as difficult or easy. Pupil size was recorded with Tobii Glasses at 30 Hz as percentage of baseline; PCI was the area under the pupil-size curve above baseline from question start to verbal answer. After each question, participants rated mental effort on the 9-point Paas scale.
  • Analyses used Pearson and Spearman correlations, chi-square, t tests/Mann–Whitney U, ANOVA/Kruskal–Wallis, and post hoc Tukey tests in SPSS 21; significance was p < 0.05, with correlation effect sizes designated weak (0.10–0.29), moderate (0.30–0.49), and strong (≥0.50).

Limitations

  • The authors state they could not control for participant age between groups; known-groups comparisons are non-specific and potentially confounded; they measured total cognitive load without distinguishing intrinsic, extraneous, and germane load; selection bias may have arisen because 2 experienced and 1 intermediate participant declined; the low proportion of female participants could skew results; and they cannot absolutely confirm the construct is overall cognitive load rather than a related concept. They also note the study does not address content and consequences as validity sources and used a controlled MCQ environment rather than true clinical practice.
  • The paper states that PCI determination was labor-intensive (about 15 min per PCI), so analysis focused on the first half of arithmetic and medical questions; 10 of 232 possible PCI data points were missing due to poor pupillary output quality.
  • The text makes evident additional practical limitations: pupillometry was expensive and not practical for routine real-world use, psychometric scales are prone to participant manipulation and provide only a single measurement, and the authors found pupillometric data analysis difficult to automate because of changing baseline pupil sizes between questions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10459-016-9725-2

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 38 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    There was no significant difference between the intermediate group and the novice group in PCI for arithmetic questions (p = 0.128).

    “In addition, there was no significant difference between the intermediate group and the novice group (p = 0.128).”
    Quote in the paperNot a substantial claim
  • Findingown data

    The correlation between PCI and Paas scale was consistent within subgroups of experience level.

    “Further analysis revealed that this correlation was consistent within subgroups of experience level.”
    Quote in the paperNot a substantial claim
  • Findingown data

    There was no significant difference between the intermediate group and the novice group in Paas scale for arithmetic questions (p = 0.365).

    “In addition, there was no significant difference between the intermediate group and the novice group (p = 0.365).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Similar to the PCI findings, the Paas scale results for novices were higher than for experienced participants when answering arithmetic questions (p = 0.040).

    “Similar to the PCI findings, the Paas scale results for novices were higher than for experienced participants when answering arithmetic questions (p = 0.040).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Overall, the PCI (a physiologic measure of cognitive load) correlated well with the Paas scale (a psychometric measure of cognitive load).

    “Overall, the PCI (a physiologic measure of cognitive load) correlated well with the Paas scale (a psychometric measure of cognitive load).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 50
  • May combine two claims. 26
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    All 232 possible data points for the Paas scale were collected, with no missing values.

    “All 232 possible data points for the Paas scale were collected, with no missing values.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of 232 possible data points, 10 were missing in the PCI data set because of poor pupillary size output quality in the raw data.

    “Of 232 possible data points, 10 were missing in the PCI data set because of poor pupillary size output quality in the raw data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    PCI values were not normally distributed and were skewed toward smaller PCIs.

    “Values were not normally distributed and were skewed toward smaller PCIs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There was no significant difference between peak pupil size in the novice and intermediate groups (p = 0.38) or the intermediate and experienced groups (p = 0.10) for medical questions.

    “There was no significant difference between peak pupil size in the novice and intermediate groups (p = 0.38) or the intermediate and experienced groups (p = 0.10).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There was no significant difference in performance on the arithmetic questions between the novice group and the intermediate group, the novice group and the expert group and the intermediate group and the expert group (p = 0.858, p = 0.410, and p = 0.555 respectively).

    “There was no significant difference in performance on the arithmetic questions between the novice group and the intermediate group, the novice group and the expert group and the intermediate group and the expert group (p = 0.858, p = 0.410, and p = 0.555 respectively).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Analysis of peak pupil size data for the arithmetic questions revealed no significant differences between any of the subgroups (novice and intermediate p = 0.15; novice and experienced p = 0.08; intermediate and experienced p = 0.97).

    “Analysis of peak pupil size data for the arithmetic questions revealed no significant differences between any of the subgroups (novice and intermediate p = 0.15; novice and experienced p = 0.08; intermediate and experienced p = 0.97).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Paas scale data were normally distributed, with no outliers.

    “The data were normally distributed, with no outliers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There were a limited number of outliers, all on the high PCI side.

    “There were a limited number of outliers, all on the high PCI side.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A relatively small number of outliers were identified in the PCI analysis.

    “No outliers were identified in the Paas responses and a relatively small number of outliers were identified in the PCI analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The intermediate group had a significantly increased score over the experienced group on medical questions.

    “The intermediate group's significantly increased score over the experienced group is likely related to this group's relative proximity to their specialty examinations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    No outliers were identified in the Paas responses.

    “No outliers were identified in the Paas responses and a relatively small number of outliers were identified in the PCI analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The analysis captured all Paas responses and missed less than 5% of PCI values due to poor data quality.

    “The analysis captured all Paas responses and missed less than 5 % of PCI values (due to poor data quality).”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    There was no significant difference in peak pupil size between groups when answering arithmetic questions, and the difference seen in medical questions was not present.

    “Of note, there was no significant difference between the peak pupil size between the groups when answering the arithmetic questions.”
    • Also stated in: § Discussion · plain
      “This difference was not present when answering arithmetic questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Comparing the measurement of a construct thought to represent cognitive load on medical professionals with both tools reveals a strong positive correlation between the two techniques as well as expected patterns based on question type, difficulty, correctness of answers and training status.

    “Comparing the measurement of a construct thought to represent cognitive load on medical professionals (both with a pupillometry-based physiologic tool as well as a psychometric survey) reveals a strong positive correlation between the two techniques as well as expected patterns based on question type, difficulty, correctness of answers and training status.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    If the tools are measuring the same construct, then there should be a correlation between their scores.

    “This source of evidence for validity is based on the idea that if the tools are measuring the same construct, then there should be a correlation between their scores (Cook and Beckman 2006).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Cook and Beckman 2006 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A particular task may yield high intrinsic cognitive load for a novice task performer but a much lower intrinsic load for an expert task performer.

    “As a result, a particular task may yield high intrinsic cognitive load for a novice task performer but a much lower intrinsic load for an expert task performer.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    A recent systematic review on cognitive load measures concluded that the quality of evidence for cognitive load measurement is low and that multiple quantification techniques should be used together in future studies to address this issue.

    “A recent systematic review on cognitive load measures concluded that the quality of evidence for cognitive load measurement is low and that multiple quantification techniques should be used together in future studies to address this issue (Naismith and Cavalcanti 2015).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Naismith and Cavalcanti 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Deliberately practicing these cognitive strategies in the context of years of experience allows certain individuals to develop expertise in a domain.

    “Deliberately practicing these cognitive strategies (as well as others) in the context of years of experience allows certain individuals to develop expertise in a domain (Ericsson et al. 2007;Norman 2005).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Ericsson et al. 2007

      Not checked in the free run.

    • cites Norman 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The science of cognitive load quantification has been traditionally separated into physiologic measurements and psychometric measurements of this construct.

    “The science of cognitive load quantification has been traditionally separated into physiologic measurements and psychometric measurements of this construct (Paas et al. 2003).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Paas et al. 2003 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Psychometric scales that measure subjective cognitive load are widely used in the literature.

    “In addition to physiologic measures of cognitive load, psychometric scales that measure subjective cognitive load are widely used in the literature.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Numerous studies in various fields have found pupillometry to be useful to measure cognitive load.

    “Numerous studies in various fields have also found pupillometry to be useful to measure cognitive load (Beatty 1982;Hess 1965;Hess and Polt 1964;Kahneman and Beatty 1966;Klingner et al. 2008Klingner et al. , 2011;;Paas et al. 2003;Szulewski et al. 2014).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Beatty 1982 source

      Not checked in the free run.

    • cites Hess 1965 source

      Not checked in the free run.

    • cites Hess and Polt 1964 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Physician cognitive load is an intrinsic characteristic of work in acute-care medical settings and is known to affect performance.

    “Physician cognitive load is an intrinsic characteristic of work in acute-care medical settings and is known to affect performance (Perry et al. 2013).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Perry et al. 2013

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Pupillometry is thought to provide an estimate of the intensity of a participant's cognitive load at a given instant in time.

    “As such, pupillometry is thought to provide an estimate of the intensity of a participant's cognitive load at a given instant in time (Laeng et al. 2012).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Laeng et al. 2012 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The nature of work in the emergency department, where physicians are frequently interrupted, treat multiple patients simultaneously and must regularly prioritize decision-making, places considerable demands on their cognitive resources and thus increases the likelihood of making errors.

    “The nature of work in the emergency department, where physicians are frequently interrupted, treat multiple patients simultaneously and must regularly prioritize decision-making, places considerable demands on their cognitive resources and thus increases the likelihood of making errors (Laxmisan et al. 2007).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Laxmisan et al. 2007 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Individuals are thought to be able to extend domain-specific long-term working memory with experience in a given field despite the traditional supposition that working memory itself is static.

    “In addition to the efficiency afforded by schema creation, individuals are thought to be able to extend domain-specific long-term working memory with experience in a given field despite the traditional supposition that working memory itself is static (Ericsson and Kintsch 1995).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ericsson and Kintsch 1995 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    These same skills allow clinicians to efficiently recognize when a new clinical scenario might not fit a previously identified mental model, thus altering subsequent management decisions.

    “These same skills allow clinicians to efficiently recognize when a new clinical scenario might not fit a previously identified mental model, thus altering subsequent management decisions (Schubert et al. 2013).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Schubert et al. 2013 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Cognitive load is thought to be comprised of three basic elements: intrinsic cognitive load, extraneous cognitive load and germane cognitive load.

    “From a theoretical perspective, cognitive load is thought to be comprised of three basic elements: intrinsic cognitive load, extraneous cognitive load and germane cognitive load (Young et al. 2014).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Young et al. 2014 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Richer mental models are created which allow experienced clinicians to more readily recognize when a new clinical scenario may fit with a previously identified pattern.

    “Richer mental models are created which allow experienced clinicians to more readily recognize when a new clinical scenario may fit with a previously identified pattern (Gegenfurtner and Seppa ¨nen 2013).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Gegenfurtner and Seppa ¨nen 2013 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Mental processing is limited by the capacity of working memory.

    “According to CLT, mental processing is limited by the capacity of working memory (De Jong 2010;Sweller et al. 1998).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites De Jong 2010 source

      Not checked in the free run.

    • cites Sweller et al. 1998 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    With the development of domain-specific expertise, those with more experience are thought to be able to chunk related concepts together in elaborated schemas, thus maximizing the efficiency of their working memory.

    “With the development of domain-specific expertise, those with more experience are thought to be able to chunk related concepts together in elaborated schemas, thus maximizing the efficiency of their working memory (Gegenfurtner et al. 2011;Sweller et al. 1998).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Gegenfurtner et al. 2011 source

      Not checked in the free run.

    • cites Sweller et al. 1998 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The nine-point mental effort scale developed by Paas has been shown to be a reliable and valid measure of overall cognitive load.

    “This scale has been widely used in the literature and has been shown to be a reliable and valid measure of overall cognitive load (Ayres 2006;Paas et al. 2003).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Ayres 2006 source

      Not checked in the free run.

    • cites Paas et al. 2003 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Correlational analyses were performed using parametric statistics (Pearson correlation) as well as non-parametric statistics (Spearman's q) as not all data were normally distributed.

    “Correlational analyses were performed using parametric statistics (Pearson correlation) as well as non-parametric statistics (Spearman's q) as not all data were normally distributed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    To reduce artefactual (e.g. blinking) and missing data, the 30 Hz raw data was smoothed by replacing values that were blank or deviated by more than 10 % absolute from the previous value with the rolling average of the previous 1/6 of a second.

    “To reduce artefactual (e.g.blinking) and missing data, the 30 Hz raw data was smoothed by replacing values that were blank or deviated by more than 10 % absolute from the previous value with the rolling average of the previous 1/6 of a second.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each participant encountered the same questions in the same order.

    “Each participant encountered the same questions in the same order.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Hess (1965) showed that pupil size increased when arithmetic problems were presented to participants, peaked when the answer was given and then dropped off again.

    “In an early experiment, Hess (1965) showed that pupil size increased when arithmetic problems were presented to participants, peaked when the answer was given and then dropped off again.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hess 1965 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Previous studies utilizing pupillometry as a physiologic measure of cognitive load have concluded that measuring cognitive load accurately requires an analysis of both the magnitude of the change in pupil size as well as the duration of time between question presentation and verbalization of a response.

    “Previous studies utilizing pupillometry as a physiologic measure of cognitive load have concluded that measuring cognitive load accurately requires an analysis of both the magnitude of the change in pupil size as well as the duration of time between question presentation and verbalization of a response (i.e. the time that a participant is thinking about an answer) (Szulewski et al. 2015).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Szulewski et al. 2015 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    If the actions (pupillometry data) fit with the thought process of participants, this would provide evidence validity based on 'response process'.

    “If the actions (pupillometry data in this case) fit with the thought process of participants, this would provide evidence validity based on ''response process'' (Cook and Beckman 2006).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Cook and Beckman 2006 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This new technology has also been successfully utilized in cognitive load measurement experiments in medical testing where questions posed to participants were shown to affect pupil size in predictable ways based on question and participant characteristics.

    “This new technology has also been successfully utilized in cognitive load measurement experiments in medical testing where questions posed to participants were shown to affect pupil size in predictable ways based on question and participant characteristics (Szulewski et al. 2015).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Szulewski et al. 2015 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Ambient light and screen brightness were standardized.

    “Ambient light and screen brightness were standardized and participants were asked to continue looking at the screen throughout the experiment and to verbalize their responses to the questions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The baseline value for pupil size was recalibrated for each question by averaging the raw data just before each question was presented to account for possible baseline drift or residual cognitive load from a previous question.

    “In order to account for possible baseline drift or residual cognitive load from a previous question, the baseline value for pupil size was recalibrated for each question by averaging the raw data just before each question was presented (corresponding to the time that each participant was focusing on a black circle presented between each question).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Szulewski et al. (2014) were able to replicate these findings using arithmetic questions with newer mobile eye-tracking technology.

    “Szulewski et al. (2014) were able to replicate these findings using arithmetic questions with newer mobile eye-tracking technology.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Szulewski et al. 2014 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Beatty (1982) found that digit and linguistic tasks of increasing complexity caused pupillary size to increase to greater degrees.

    “Beatty (1982) found that digit and linguistic tasks of increasing complexity caused pupillary size to increase to greater degrees.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Beatty 1982 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In a group of high school students solving algebra problems, Ayres (2006) showed that the Paas scale provided a cognitive load rating that was reliable and correlated highly with errors, as expected.

    “For example, in a group of high school students solving algebra problems, Ayres (2006) showed that the Paas scale provided a cognitive load rating that was reliable and correlated highly with errors, as expected.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ayres 2006 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other groups of researchers have also found consistent results using the mental effort scale developed by Paas.

    “Other groups of researchers have also found consistent results using the mental effort scale developed by Paas found in ''Appendix 1'' (Ayres 2006;Tuovinen and Paas 2004).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Ayres 2006 source

      Not checked in the free run.

    • cites Tuovinen and Paas 2004 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Audio data were recorded synchronously and analyzed later by a research assistant blinded to the pupillometry data.

    “Participants verbalized their responses to the questions and surveys; these audio data were recorded synchronously and analyzed later by a research assistant blinded to the pupillometry data.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Questions were classified a priori into 'difficult' and 'easy' questions based on their origin (medical student handbook vs. specialty board examination preparation material) as well as the authors' judgement.

    “Questions were classified a priori into ''difficult'' and ''easy'' questions based on their origin (medical student handbook vs. specialty board examination preparation material) as well as the authors' judgement in an effort to provide evidence from a ''relations to other variables'' source.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A research assistant who was not involved in data analysis or experimental design conducted the experiment with each participant.

    “A research assistant who was not involved in data analysis or experimental design conducted the experiment with each participant.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Because the Paas scale asks participants to rate their level of investment of cognitive resources and pupillometry is supposed to quantify the investment of cognitive resources, a correlation between the two tools would provide some 'response process' evidence.

    “Because the Paas scale asks participants to rate their level of investment of cognitive resources and pupillometry is supposed to quantify the investment of cognitive resources, it was theorized that a correlation between the two tools would provide some of this ''response process'' evidence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The low proportion of female participants has the potential to skew the results.

    “In addition, the low proportion of female participants in this study has the potential to skew the results; however, we do not know of any literature that supports a difference in pupillary responses based on sex.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Two potential participants from the experienced group and one from the intermediate group declined to participate, which could lead to selection bias.

    “The fact that two potential participants from the experienced group and one potential participant from the intermediate group declined to participate in the study could lead to selection bias.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We did not distinguish between types of cognitive load and instead focused on total measurable cognitive load.

    “In addition, we did not distinguish between the types of cognitive load (intrinsic vs. extraneous vs. germane) and instead chose to focus on total measurable cognitive load.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Researchers and educators can have increased confidence using either measure depending on the context and purpose of their study, as they both appear to measure the same construct.

    “Researchers and educators can have increased confidence using either measure depending on the context and the purpose of their study, as they both appear to measure the same construct.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Paas survey asked participants to rate investment of cognitive resources, which is what the pupillometry metrics were designed to measure.

    “In this study, though participants were not asked to specifically describe their thought processes (as may have been done with a think-aloud protocol), the Paas survey asked participants to rate investment of cognitive resources, which is what the pupillometry metrics were designed to measure.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Based on the results of this study, we have been able to provide further evidence of construct validity that the PCI and the psychometric Paas scale are indeed reasonable surrogate markers of cognitive load.

    “Based on the results of this study, we have been able to provide further evidence of construct validity that the PCI and the psychometric Paas scale are indeed reasonable surrogate markers of cognitive load.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    This study provides some evidence of internal structure, response process and relations to other variables, but does not address content and consequences as sources for validity evidence.

    “From the perspective of Cook's review of Messick's framework of validity, although we provide some evidence of internal structure, response process and relations to other variables, this study does not address content and consequences as sources for validity evidence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The intermediate group's increased score over the experienced group is likely related to their relative proximity to specialty examinations.

    “The intermediate group's significantly increased score over the experienced group is likely related to this group's relative proximity to their specialty examinations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The analysis of training-relevant and training-irrelevant questions strengthens the argument, but the possibility of confounding remains.

    “The analysis of training-relevant (medical) and training-irrelevant (arithmetic) questions strengthens this argument, but the possibility of confounding remains.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The arithmetic questions are training-irrelevant and the medical questions are training-relevant for the participants.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    We used known-groups comparisons in part of our analysis of validity from a 'relations to other variables' source.

    “Secondly, we used known-groups comparisons in part of our analysis of validity from a ''relations to other variables'' source.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    We believe the age effect to be small given that pupillary size and psychometric responses varied in the same direction throughout this study.

    “Although we recognize that there might be an effect, we believe it to be small given that pupillary size and psychometric responses varied in the same direction throughout this study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The inability to control for age raises questions about possible confounding effects of age on both pupillometry analysis and questionnaire responses.

    “This is unavoidable given the inherent nature of experience, but it does raise questions about the possible confounding effects of age on both pupillometry analysis as well as questionnaire responses.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The currently accepted understanding of validity places an emphasis on construct validity as the 'whole' of the validity argument.

    “The currently accepted understanding of validity places an emphasis on construct validity as the ''whole'' of the validity argument (Downing 2003).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Downing 2003 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Our work focused on cognitive load measurement in a tightly controlled (not true-to-clinical-life) MCQ environment.

    “Our work focused on cognitive load measurement in a tightly controlled (not true-toclinical-life) MCQ environment.”
    Quote in the paperSubstantial claim
62 of 62

  • The paper relies on this without support.

    There is a paucity of studies comparing physiologic and psychometric tools head-to-head.

    “Moreover, there is a paucity of studies that compares physiologic and psychometric tools head-to-head.”

  • The paper relies on this without support.

    There is no accepted gold-standard for cognitive load measurement.

    “Though both physiologic and psychometric measures attempt to quantify cognitive load, there is no accepted gold-standard for cognitive load measurement.”

  • The paper relies on this without support.

    The classification of questions as difficult or easy based on their origin and the authors' judgement is valid.

    “Questions were classified a priori into ''difficult'' and ''easy'' questions based on their origin (medical student handbook vs. specialty board examination preparation material) as well as the authors' judgement in an effort to provide evidence from a ''relations to other variables'' source.”

  • The paper relies on this without support.

    The arithmetic questions are training-irrelevant and the medical questions are training-relevant for the participants.

    “The analysis of training-relevant (medical) and training-irrelevant (arithmetic) questions strengthens this argument”
    “between question types (domain-specific medical questions vs. arithmetic questions)”

  • The paper relies on this without support.

    The experience groups (novice, intermediate, experienced) accurately reflect different levels of resuscitation medicine expertise.

    “participants were divided into groups based on resuscitation medicine experience, not arithmetic experience.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Comparisons with physiological measurement techniques have shown that the subjective rating scale is just as valid and reliable and easier to use as objective techniques.

    “even when comparisons with physiological measurement techniques have shown that the subjective rating scale is just as valid and reliable and easier to use as objective techniques (e.g. Szulewski et al. 2018).”

    From Cognitive Architecture and Instructional Design: 20 Years Later (2019) · cites this paper as Szulewski et al. 2018

    The Conclusion reports a strong correlation between the Paas subjective scale and pupillometry and supports the validity of either as a surrogate for cognitive load. The Discussion says the scale is easier to use, particularly for a general assessment in a test-taking setting. The paper does not establish that the two techniques are equally reliable or equally valid across settings. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Not every subgroup correlation meets the paper's threshold for strong

    Minor

    The paper defines correlations from 0.30 to 0.49 as moderate and those at or above 0.50 as strong. The novice-group Pearson correlation for medical questions is 0.422, so the claim that strong correlations persisted across experience subgroups is not true for every reported correlation.

1 of 1

Cites

Flagged first, then by citations

1 of the 42 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Szulewski, A., Gegenfurtner, A., Howes, D. W., Sivilotti, M. L. A., & van Merriënboer, J. J. G.
Affiliations
  • Queen's University (CA)
  • Deggendorf Institute of Technology (DE)
  • Kingston General Hospital (CA)
Corresponding author
  • Szulewski, Adam
  • Gegenfurtner, Andreas
  • Howes, Daniel W.
  • Sivilotti, Marco L. A.
ORCID
Venue
Advances in Health Sciences Education
Publisher
Springer Science and Business Media LLC
Published
2016-10-27
Volume
22
Issue
4
Pages
951-968
ISSN
  • 1382-4996 (ISSN-L)
  • 1573-1677
DOI
10.1007/s10459-016-9725-2
OpenAlex
W2545952627
Cited by
94
References
42
Citations by year
2026: 6 · 2025: 8 · 2024: 13 · 2023: 11 · 2022: 8 · 2021: 14 · 2020: 9 · 2019: 13 · 2018: 5 · 2017: 6 · 2016: 1
FWCI
11.86
Type
journal article
Language
en
Open access
Licence
www.springer.com/tdm (unspecified)
Locations
  • Advances in Health Sciences Education (publishedVersion)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, other-oa)
  • OPUS (Augsburg University) (submittedVersion)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
cognitive load, pupillometry, resuscitation medicine, multiple-choice questions, medical expertise, construct validity
MeSH terms
Adult, Cognition, Female, Humans, Male, Physicians, Psychological Theory, Psychometrics, Resuscitation, Reproducibility of Results, Eye Movement Measurements

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review