Executive functioning rating scales: Ecologically valid or construct invalid?

“A parsimonious explanation for this modest association could be that executive function tests and ratings are assessed using different methods completed in different settings across different time periods (Podsakoff et al., 2003).”

“…validity; rating scales; performance tests; academic achievement Executive functions are higher-order neurocognitive processes associated with regulating thoughts and behaviors by maintaining problem sets to attain future goals (Miyake et al. 2000; van der Ven et al., 2013; Wiebe et al., 2011).”

“…counterbalanced tests developed specifically to assess executive functioning abilities as defined in the cognitive sciences (e.g., Baddeley, 2007; Miyake & Friedman, 2012); (b) multiple standardized, gold standard executive function rating scales completed by multiple informants (parents and…”

“…including schizophrenia (Mesholam-Gately et al., 2009), depression (Joormann & Gotlib, 2008; Snyder, 2013), bipolar disorders (Bora et al., 2009), conduct disorder (Morgan & Lilienfeld, 2000) and attention-deficit/hyperactivity disorder (ADHD; Kasper et al., 2012; Willcutt et al., 2005).”

“Partial-credit unit scoring (i.e., stimuli correct per trial) was used to index overall working memory performance at each set size 3–6 was used as recommended (Conway et al., 2005).”

“…of (a) multiple counterbalanced tests developed specifically to assess executive functioning abilities as defined in the cognitive sciences (e.g., Baddeley, 2007; Miyake & Friedman, 2012); (b) multiple standardized, gold standard executive function rating scales completed by multiple informants…”

“…between executive function ratings and performance tests, and used the test of dependent correlations as implemented in the R package cocor (Diedenhofen & Musch, 2015) to test whether the component EF ratings/tests correlation was similar to the academic functioning ratings/performance…”

“Neurocorrelates of working memory include the mid-lateral prefrontal cortex as well as interconnected neural networks (Nee & Jonides, 2013; Wager & Smith, 2003).”

“A ratio of 80:20 go:no-go stimuli was selected to maximize prepotency (Kane & Engle 2003; Unsworth & Engle 2007).”

“, 2009), depression (Joormann & Gotlib, 2008; Snyder, 2013), bipolar disorders (Bora et al.”

“Several studies have demonstrated the contributions of executive functions to a wide range of academic achievement outcomes (Bull et al., 2008; Hitch et al., 2001; Miller & Hinshaw, 2010).”

“, 2013), executive function ratings have been hypothesized to instead reflect mental constructs involving the success of goal pursuit (Toplak et al., 2013) and/or non-mental constructs such as externalizing behavior problems (Spiegel et al.”

“In addition, empirical evidence indicates that executive functions play a critical role in many important functional outcomes beginning as early as young childhood (Alloway & Alloway, 2010; Bierman et al., 2008; Gathercole et al., 2004; Moffitt et al., 2011; O’Shaughnessy et al., 2003).”

“A ratio of 80:20 go:no-go stimuli was selected to maximize prepotency (Kane & Engle 2003; Unsworth & Engle 2007).”

“Among the diverse models of executive functions, factor analytic and theoretical work provides the most empirical support for models that include two primary executive function domains in early and middle childhood: working memory and inhibitory control (Karr et al., 2018; Lerner & Lonigan, 2014; St. Clair-Thompson &…”

“Global Intellectual Functioning (IQ) and Socioeconomic Status (SES) All children were administered the Verbal Comprehension Index of a Wechsler (2011, 2014) scale (WISC-IV, WISC-V).”

“Deficits in executive functions are theorized to be etiologically important for a broad range of psychopathologies, including schizophrenia (Mesholam-Gately et al., 2009), depression (Joormann & Gotlib, 2008; Snyder, 2013), bipolar disorders (Bora et al.”

“Inhibitory control is supported by diverse brain regions including the right thalamic, right superior temporal and left inferior occipital gyri, bilateral frontal, and mid-brain structures (Cortese et al., 2012).”

“…including schizophrenia (Mesholam-Gately et al., 2009), depression (Joormann & Gotlib, 2008; Snyder, 2013), bipolar disorders (Bora et al., 2009), conduct disorder (Morgan & Lilienfeld, 2000) and attention-deficit/hyperactivity disorder (ADHD; Kasper et al., 2012; Willcutt et al., 2005).”

“…of psychopathologies, including schizophrenia (Mesholam-Gately et al., 2009), depression (Joormann & Gotlib, 2008; Snyder, 2013), bipolar disorders (Bora et al., 2009), conduct disorder (Morgan & Lilienfeld, 2000) and attention-deficit/hyperactivity disorder (ADHD; Kasper et al., 2012; Willcutt et…”

“Given the well-established associations between deficits in executive functions and adverse behavioral, academic, and occupational outcomes (Best et al., 2009; Kofler et al., 2016), accurate assessment of executive functions in childhood is critical for early detection and intervention efforts (Diamond, 2012).”

“…with academic skills involving written language (Alloway et al., 2005), oral language (McInnes et al., 2003), reading and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009), reading comprehension (Kieffer et al., 2013;…”

“, 2018), science (Gathercole et al., 2004; St. Clair-Thompson & Gathercole, 2006), and following instructions (Jaroslawska et al.”

“Correlations of this magnitude among individual executive function tests highlight the task impurity problem, such that the majority of variance in a single executive function test is not attributable specifically to the executive function the test is designed to measure (Snyder et al., 2015).”

“Executive functions are higher-order neurocognitive processes associated with regulating thoughts and behaviors by maintaining problem sets to attain future goals (Miyake et al. 2000; van der Ven et al., 2013; Wiebe et al., 2011).”

“, 2016), accurate assessment of executive functions in childhood is critical for early detection and intervention efforts (Diamond, 2012).”

“In addition, empirical evidence indicates that executive functions play a critical role in many important functional outcomes beginning as early as young childhood (Alloway & Alloway, 2010; Bierman et al., 2008; Gathercole et al., 2004; Moffitt et al., 2011; O’Shaughnessy et al., 2003).”

“…emotion recognition, reading skills, math performance, social skills, organizational skills, parent-child relationship quality, and activities of daily living (Carlson & Wang, 2007; Karalunas et al., 2017; Kofler et al., 2010, 2017; Raiker et al., 2012; Rapport et al., 2009; Wells et al., 2018).”

“…children’s lives, association with social and occupational functioning both cross-sectionally and longitudinally (Kuriyan et al., 2013; Walker, & Nabuzoka, 2007; Wentzel, 1991), and strong theoretical and empirical links with executive functioning (e.g., Ahmed et al., 2018; Samuels et al., 2016 ).”

“…theorized to be etiologically important for a broad range of psychopathologies, including schizophrenia (Mesholam-Gately et al., 2009), depression (Joormann & Gotlib, 2008; Snyder, 2013), bipolar disorders (Bora et al., 2009), conduct disorder (Morgan & Lilienfeld, 2000) and…”

“SSD was selected based on conclusions from recent meta-analytic reviews that it is the most direct measure of inhibitory control in stop-signal tasks that utilize dynamic SSDs, given that SSDs change systematically according to inhibitory success or failure (Alderson et al., 2007; Lijffijt et al., 2005).”

“…skills involving written language (Alloway et al., 2005), oral language (McInnes et al., 2003), reading and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009), reading comprehension (Kieffer et al., 2013; Kofler, Wells et al.,…”

“…vs. ratings appears inconsistent with several studies finding that executive function ratings predicted functional outcomes better than executive function performance tests (Barkley & Murphy, 2010; Gioia et al., 2000; Gross et al., 2015; Kamradt et al., 2014; Mahone & Hoffman, 2007; Toplak, 2008).”

“…achievement – a key functional outcome domain with strong theoretical links to executive function as reviewed below; and (d) a large and diverse sample of children oversampled for ADHD and other forms of child psychopathology associated with executive dysfunction (e.g., Kasper et al., 2012).”

“Academic functioning was selected as the predictive validity outcome given its centrality in children’s lives, association with social and occupational functioning both cross-sectionally and longitudinally (Kuriyan et al., 2013; Walker, & Nabuzoka, 2007; Wentzel, 1991), and strong theoretical and empirical links with…”

“, 2015), social relations (Clark et al., 2002), organizational skills (Kofler et al.”

“For example, executive functions have been associated with academic skills involving written language (Alloway et al., 2005), oral language (McInnes et al., 2003), reading and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009),…”

“In addition, their convergent validity has been supported across multiple studies showing strong convergence with other questionnaire-based executive function assessments completed by the same informant at the same time point (Buchanan, 2016; Gross et al., 2015; McAuley et al., 2010; Muris et al., 2008).”

“…two putative methods for assessing the same construct in child, adolescent, and adult populations (Biederman et al., 2008; Conklin et al., 2008; Mahone et al., 2002), and extended previous findings by demonstrating that this relation remained small when controlling for mono-measure and…”

“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript discord between executive function rating scales and performance-based tests (Conklin et al., 2008; Toplak et al., 2013; Vriezen & Pigott, 2002).”

“Reliability and validity evidence includes high internal consistency (α = .81–.97) and 1–3-week test-retest reliability (.76–.90; Kofler et al., 2019; Sarver et al., 2015), and expected relations with criterion working memory complex span (r = .69) and updating tasks (r = .61) (Wells et al., 2018).”

“For example, executive functions have been associated with academic skills involving written language (Alloway et al., 2005), oral language (McInnes et al., 2003), reading and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009),…”

“Several studies have demonstrated the contributions of executive functions to a wide range of academic achievement outcomes (Bull et al., 2008; Hitch et al., 2001; Miller & Hinshaw, 2010).”

“…and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009), reading comprehension (Kieffer et al., 2013; Kofler, Wells et al., 2018), science (Gathercole et al., 2004; St. Clair-Thompson & Gathercole, 2006), and following…”

“Go/no-go inhibitory control—The go/no-go is a response inhibition task in which a motor response must be executed or inhibited based on a stimulus cue (Bezdjian et al. 2009).”

“…emotion recognition, reading skills, math performance, social skills, organizational skills, parent-child relationship quality, and activities of daily living (Carlson & Wang, 2007; Karalunas et al., 2017; Kofler et al., 2010, 2017; Raiker et al., 2012; Rapport et al., 2009; Wells et al., 2018).”

“, 2003), reading and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009), reading comprehension (Kieffer et al.”

“, typically 5–10 minutes per test), whereas executive function ratings are completed by teachers/parents based on children’s behavior in the classroom/at home over a longer duration (i.e., most rating forms ask informants to consider the last several weeks or months; Gioia et al., 1996; Reynolds & Kamphaus, 2004, 2015…”

“, 2003), reading and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009), reading comprehension (Kieffer et al.”

“…language (Alloway et al., 2005), oral language (McInnes et al., 2003), reading and math attainment (Raghubar et al., 2010; Sesma et al., 2009; Swanson & Kim, 2007; Thorell, 2007; Wåhlstedt et al., 2009), reading comprehension (Kieffer et al., 2013; Kofler, Wells et al., 2018), science…”

“Several studies have demonstrated the contributions of executive functions to a wide range of academic achievement outcomes (Bull et al., 2008; Hitch et al., 2001; Miller & Hinshaw, 2010).”

“…vs. ratings appears inconsistent with several studies finding that executive function ratings predicted functional outcomes better than executive function performance tests (Barkley & Murphy, 2010; Gioia et al., 2000; Gross et al., 2015; Kamradt et al., 2014; Mahone & Hoffman, 2007; Toplak, 2008).”

“…children’s lives, association with social and occupational functioning both cross-sectionally and longitudinally (Kuriyan et al., 2013; Walker, & Nabuzoka, 2007; Wentzel, 1991), and strong theoretical and empirical links with executive functioning (e.g., Ahmed et al., 2018; Samuels et al., 2016 ).”

“…validity; rating scales; performance tests; academic achievement Executive functions are higher-order neurocognitive processes associated with regulating thoughts and behaviors by maintaining problem sets to attain future goals (Miyake et al. 2000; van der Ven et al., 2013; Wiebe et al., 2011).”

“…deficits negatively impact areas of social/peer and family relations (Clark et al., 2002; Gewirtz et al., 2009; Kofler et al., 2016), organizational skills (Kofler et al., 2018), occupational functioning (Barkley et al., 2008; Miller et al., 2012), and academic achievement (Best et al., 2009).”

“…their convergent validity has been supported across multiple studies showing strong convergence with other questionnaire-based executive function assessments completed by the same informant at the same time point (Buchanan, 2016; Gross et al., 2015; McAuley et al., 2010; Muris et al., 2008).”

“Clair-Thompson & Gathercole, 2006), and following instructions (Jaroslawska et al., 2016).”

“, 2018); technology-driven methods such as virtual reality-based executive function assessment may help to balance the internal validity gained through careful, laboratory-based control with the external/ecological validity gained through field investigations (Jaroslawska et al. 2016; Jovanovski et al., 2012; Parsons…”

“…that executive function rating scales predict theoretically-linked ratings of personality traits, internalizing and externalizing disorders, social skills, mood difficulties and academic functioning (Buchanan, 2016; Gerst et al., 2017; Gross et al., 2015; Jarratt et al., 2005; Muris et al., 2008).”

“Working memory is a limited capacity system that involves the updating, manipulation/serial reordering, and dual-processing of internally held information for use in guiding behavior (Baddeley, 2007; Shelton et al., 2010; Unsworth et al., 2010).”

“Concurrent and predictive validity evidence includes demonstrations that executive function rating scales predict theoretically-linked ratings of personality traits, internalizing and externalizing disorders, social skills, mood difficulties and academic functioning (Buchanan, 2016; Gerst et al., 2017; Gross et al., 2…”

“Standardized and expected residual variances for observed variables informed by published studies utilizing executive function rating scales and performance tasks (Alderson et al., 2010; Jarratt et al., 2005; Sullivon & Riccio, 2006) were imputed to delineate the proposed path model.”

“…deficits negatively impact areas of social/peer and family relations (Clark et al., 2002; Gewirtz et al., 2009; Kofler et al., 2016), organizational skills (Kofler et al., 2018), occupational functioning (Barkley et al., 2008; Miller et al., 2012), and academic achievement (Best et al., 2009).”

“…measure-level relations between these two putative methods for assessing the same construct in child, adolescent, and adult populations (Biederman et al., 2008; Conklin et al., 2008; Mahone et al., 2002), and extended previous findings by demonstrating that this relation remained small…”

“…to measure executive functioning abilities (Snyder et al., 2015) includes replicated evidence that these tests contribute significantly and uniquely to latent estimates of global and specific executive functions (Miyake et al., 2000; St. Clair-Thompson & Gathercole, 2006; Willoughby & Blair, 2016).”

“Standardized and expected residual variances for observed variables informed by published studies utilizing executive function rating scales and performance tasks (Alderson et al., 2010; Jarratt et al., 2005; Sullivon & Riccio, 2006) were imputed to delineate the proposed path model.”

“In addition, empirical evidence indicates that executive functions play a critical role in many important functional outcomes beginning as early as young childhood (Alloway & Alloway, 2010; Bierman et al., 2008; Gathercole et al., 2004; Moffitt et al., 2011; O’Shaughnessy et al., 2003).”

“…functions, factor analytic and theoretical work provides the most empirical support for models that include two primary executive function domains in early and middle childhood: working memory and inhibitory control (Karr et al., 2018; Lerner & Lonigan, 2014; St. Clair-Thompson & Gathercole, 2006).”

“This concurrent and predictive validity evidence includes experimental and longitudinal linkages between executive function performance tests and objective and subjective measures of inattention, hyperactivity, impulsivity, emotion recognition, reading skills, math performance, social skills, organizational skills, pa…”

“Inhibitory control refers to the ability to withhold (action restraint) or suppress (action cancellation) a pre-potent behavioral response (Lewis & Carpendale, 2009).”

“As detailed below, concurrent and predictive validity studies consistently indicate that executive function rating scales and performance-based tests correlate weakly/non-significantly with one another (Biederman et al., 2008; Conklin et al., 2008; Nordvall et al., 2017).”

“, 2018); technology-driven methods such as virtual reality-based executive function assessment may help to balance the internal validity gained through careful, laboratory-based control with the external/ecological validity gained through field investigations (Jaroslawska et al. 2016; Jovanovski et al., 2012; Parsons…”

“Global Intellectual Functioning (IQ) and Socioeconomic Status (SES) All children were administered the Verbal Comprehension Index of a Wechsler (2011, 2014) scale (WISC-IV, WISC-V).”

“Working memory is a limited capacity system that involves the updating, manipulation/serial reordering, and dual-processing of internally held information for use in guiding behavior (Baddeley, 2007; Shelton et al., 2010; Unsworth et al., 2010).”

“Support for the use of construct-valid executive function tests such as the stop-signal task and Rapport working memory tests also includes well-documented internal consistency and test-retest reliability (Alderson et al., 2008; Rapport et al., 2009; Sarver et al., 2015; Soreni et al., 2009).”

“Reliability and validity evidence includes high internal consistency (α = .81–.97) and 1–3-week test-retest reliability (.76–.90; Kofler et al., 2019; Sarver et al., 2015), and expected relations with criterion working memory complex span (r = .69) and updating tasks (r = .61) (Wells et al., 2018).”

“Overall, we replicated prior findings of modest to weak associations between executive function tests and ratings (Gross et al., 2015; Miranda et al., 2015; Nordvall et al., 2017) and extended those findings by showing that, even at the construct level, these associations fall well below expectations for crossmethod a…”

“, 2018); technology-driven methods such as virtual reality-based executive function assessment may help to balance the internal validity gained through careful, laboratory-based control with the external/ecological validity gained through field investigations (Jaroslawska et al. 2016; Jovanovski et al., 2012; Parsons…”

“ratings appears inconsistent with several studies finding that executive function ratings predicted functional outcomes better than executive function performance tests (Barkley & Murphy, 2010; Gioia et al., 2000; Gross et al., 2015; Kamradt et al., 2014; Mahone & Hoffman, 2007; Toplak, 2008).”

“Neurocorrelates of working memory include the mid-lateral prefrontal cortex as well as interconnected neural networks (Nee & Jonides, 2013; Wager & Smith, 2003).”

“…their convergent validity has been supported across multiple studies showing strong convergence with other questionnaire-based executive function assessments completed by the same informant at the same time point (Buchanan, 2016; Gross et al., 2015; McAuley et al., 2010; Muris et al., 2008).”

“…functions, factor analytic and theoretical work provides the most empirical support for models that include two primary executive function domains in early and middle childhood: working memory and inhibitory control (Karr et al., 2018; Lerner & Lonigan, 2014; St. Clair-Thompson & Gathercole, 2006).”

“This conclusion warrants scrutiny, however, because to our knowledge the current evidence base relies almost exclusively on questionnaire-based outcome measures completed by the same informant at the same time point as the executive function rating scales (i.e., mono-informant, monomethod bias; Spiegel et al., 2017).”

“Performance was monitored at all times by the examiner, who was stationed just out of the child’s view to provide a structured environment while minimizing improvements related to examiner demand characteristics (Gomez & Sanson, 1994).”

“Psychometric support for these subscales include high internal consistency (α = .89–.98), test-retest reliability (r = .76–.91), and expected correspondence with other informant-based ratings of executive functions (Gioia et al., 1996; Sullivan & Riccio, 2006).”

“Further research is needed to assess the predictive validity of executive function tests and ratings relative to other important outcomes known to depend at least in part on executive function abilities, such as phonological awareness (Allan et al., 2015), social relations (Clark et al.”

“For example, executive function deficits negatively impact areas of social/peer and family relations (Clark et al., 2002; Gewirtz et al., 2009; Kofler et al., 2016), organizational skills (Kofler et al.”

“As detailed below, concurrent and predictive validity studies consistently indicate that executive function rating scales and performance-based tests correlate weakly/non-significantly with one another (Biederman et al., 2008; Conklin et al., 2008; Nordvall et al., 2017).”

“However, EF remain a controversial topic in research due to the wide range of definitions, theories, and Frontiers in Psychology 01 frontiersin.org measures associated with it (Baggetta and Alexander, 2016; Soto et al., 2020; Laureys et al., 2022).”

“Finally, the debate continues on how best to assess EF [94,95].”

“132).”

“Those presumptions are often unsupported by empirical data obtained from adults (Burmester et al., 2016; Rabin et al., 2006; Spencer et al., 2010), adolescents (Herbrich et al., 2018; Stedal & Dahlgren, 2015) and children (Soto et al., 2020).”

“Recent literature indicates a strong discord between EF rating scales and performance-based tasks [56].”

“Limitations of previous studies include the use of complex inhibition tasks that may be confounded by the working memory demands required for successful task performance (Verbruggen & Logan, 2008; Gordon & Caramazza, 1982; Kofler et al., 2018), use of simple span tasks (Egeland, 2015; Engle, 2010) that assess short-te…”

“Even though the employment of rating measures of CEFs, such as the BRIEF 2, was debated (see Soto et al., 2020; Toplak et al., 2013), evidence suggests a moderate correlation between the self-report measures and performance-based tasks of CEFs (Gerst et al., 2017).”

“The main problem is that the term EF is a complex construct with a wide variety of theoretical models and processes that influence the standards to assess them (Demetriou et al., 2019; Soto et al., 2020).”

“In contrast, rating scales, or “clinical” scales, of EF have consistently demonstrated higher ecological validity than neuropsychological scales (Soto et al., 2020; Toplak et al., 2013).”

“…everyday cognitive behaviours as separate constructs, consistent with an extensive literature suggesting they provide non ‐ overlapping information, and that functioning in the two domains makes independent contributions to clinical and academic problems (Soto et al., 2020; Toplak et al., 2013).”

“Task-based measures of EF test EF under optimal conditions (Soto et al., 2020).”

“Although performance-based tests have advantages to yield valuable information about how one operates (Barkley, 2012b; Soto et al., 2020), their ecological validity has been questioned due to an insignificant match between what the person has to do under controlled conditions 410 A. MASHHADI ET…”

“, Go/No-Go [53], Wisconsin Card Sort Task [50]).”

“Rating scales assess executive functioning through self-reports or informant reports, capturing behavior in everyday contexts (Soto et al., 2020).”

“Performance-based measures and rating scales are critical to the evaluation of EF as each captures different aspects of the construct (Burgess et al., 1998; Soto et al., 2020; Toplak et al., 2013).”

“However, a key question, that cannot be answered with these data, is the extent to which subjective reports of cognitive difficulties were accompanied by objective declines in cognitive performance (Soto et al., 2020).”

“Performance-based measures provide process-specific information, tend to have stronger psychometric characteristics, and are more robust predictors of academic skills with less potential for bias compared to rating-based measures (Soto et al., 2020).”

“…on tests that were developed to assess gross neuropsychological functioning and questionnaire-based assessments of executive function/cognitive control processes that show poor correspondence with specific tests of these cognitive processes (for reviews, see Snyder et al., 2015; Soto et al., 2020).”

“While performance-based EF tests help predict academic success (27), behavioral rating scales, which offer stronger ecological validity (28), provide further insight into real-world self-regulation challenges in KS patients; however, only a limited number of studies have incorporated these in their…”

“Importantly, there are weak associations between executive function rating scales and performance-based measures (Bernes et al., 2021; Soto et al., 2020), indicating that the different measures may be measuring different constructs of EF.”

“However, studies in paediatric populations suggest that parent ‐ rated and objectively assessed executive function measures are not highly correlated (Gross et al., 2015; Lafavor et al., 2022; Soto et al., 2020).”

“Moreover, as suggested by several authors, it seems that performance tasks do not present an adequate ecological and predictive validity of these functions (Soto et al., 2020).”

“There is considerable evidence that scores on the performance-based measures are weakly correlated with the rating scales, and the use of both has been recommended to provide incremental information (non-overlapping information) and a multidimensional perspective on EF and related behaviors in people with ADHD (Barkle…”

“These tests appear to capture the primarily cognitive aspects of EF and to identify those with scholastic difficulties (14-17).”

“Soto et al. (15), on the other hand, argued that the research design of earlier studies, such as Toplak et al. (13, 14), has an impact on the interpretation of these studies.”

“Furthermore, it can be confirmed that the version of BRIEF-P [27] and the BRIEF [73] compiled by the parents with a reduced version, receive the best scores.”

“Converging evidence supports the conclusion that performance-based and rating measures of EF assess different constructs or expressions of EF [31–34].”

“79 By contrast, behavioral rating scales of executive functioning 80 81 consistently show stronger test-retest reliability and better ecological validity because they capture the cumulative functional outcomes of executive function across time and real world contexts.”

“…may identify different individuals as having executive dysfunction and may also capture variability in the speci fi c components of executive functioning affected within the same individual, highlighting the heterogeneity of this ADHD associated feature (Snyder et al., 2015; Soto et al., 2020).”

“While lab-based measures are regarded as offering a more objective assessment of executive abilities, they typically reflect optimal performance within a highly structured setting and may not correlate with real-world functioning [29].”

“…EF skills in the fall will uniquely contribute to increases in teachers’ ratings of children’s EF behaviors in the spring, as direct assessments of EF skills have been significantly correlated with teacher report of students’ behavioral regulation (Gerst et al., 2017; Soto et al., 2020).”

“Performance-based tests have been criticized as having low generalizability to real-world settings, mainly due to the low of variance in daily functioning that is accounted for, and to their inability to reflect true everyday performance due to highly controlled conditions (Soto et al., 2020).”

“While an early review of empirical studies using tests with verisimilitude, veridicality, or a combination of both approaches found some support for the superiority of the verisimilitude approach (Chaytor and Schmitter-Edgecombe 2003), more recent studies indicate that increasing a test’s ecological validity may not i…”

“Factors such as emotional distress, fatigue, sleep problems, metacognitive capacity, and contextual demands may therefore contribute to discrepancies between subjective and objective WM measures (Gerst et al., 2017; Nin et al., 2022; Snyder et al., 2021; Soto et al., 2020; Toplak et al., 2013).”

“…the Fish-Flanker or Fish-Shark tasks) and caregiver-reported questionnaires (such as the BRIEF-P) are known to capture distinct aspects of IC, with the latter often providing a more ecologically valid assessment of complex, everyday behaviors (Barkley, 2012; Soto et al., 2020; Toplak et al., 2013).”

“Closely related work by Soto et al. (2020) evaluated the concurrent association of EF ratings and EF performance with academic ratings and academic performance.”

“…rating scales cannot be used interchangeably in the assessment of executive functions (Isquith et al., 2013; Krieger & Amador-Campos, 2018; Soto et al., 2020; Toplak et al., 2013), all these studies only used performance-based tests as executive function measures (e.g., the Working Memory…”

“…finding within other populations that caregiver ratings of executive functioning in daily life are frequently discrepant from objective testing 27,28 due to multiple factors (ie, standardized testing is completed in an optimal setting, while caregiver ratings are based on observations in daily…”

“Studies have shown that these predictors provide valuable information about di ff erent underlying mental constructs [10 – 12].”

“…Holmes et al., 2014), explored how well executive function rating scales and performance tests predict outcomes separately (e.g., Gerst et al., 2017; Soto et al., 2020), or tested relationships between different types of executive function measurement (e.g., Nin et al., 2022; Tamm & Peugh, 2019).”

“As such, interpreting scores obtained from executive function rating scales is particularly challenging, as the real-world behaviors measured by these scales undoubtedly involve non-executive cognitive processes and non-cognitive factors (Snyder et al., 2015; Soto et al., 2020; Toplak et al., 2013).”

“Moreover, the study should be replicated with a continuous performance task for attention or other direct attention measure in addition to rating scales by either parents or teacher to yield a fuller measurement of attention (Soto et al., 2020).”

“The modest correlation between [8][9] engagement based tests and rating scales further supports - performance [10] . game-Serious environments are promising as they improve engagement and automatically capture quantitative behavioural logs [13] [14].”

“Traditionally, EF assessments have been classified as either objective performance-based tasks or subjective self-and informant-report questionnaires (Dias & Malloy-Diniz, 2024; Soto et al., 2020).”