The DOI matches its Crossref record; 157 of the 160 references it cites were found in a registry or an index, and 2 are not checked yet; 14 of the 52 sources its claims cite were checked for support.
Conceptualization of the term 'ecological validity' in neuropsychological research on executive function assessment: a systematic review and call to action
Open accessAbstract
OBJECTIVE: "Ecological validity" (EV) is classically defined as test's ability to predict real-world functioning, either alone or together with test's similarity to real-world tasks. In neuropsychological literature on assessment of executive functions (EF), EV is conceptualized inconsistently, leading to misconceptions about the utility of tests. The goal of this systematic review was to examine how EV is conceptualized in studies of EF tests described as ecologically valid. METHOD: MEDLINE and PsychINFO Databases were searched. PRISMA guidelines were observed. After applying inclusion and exclusion criteria, this search yielded 90 articles. Deductive content analysis was employed to determine how the term EV was used. RESULTS: About 1/3 of the studies conceptualized EV as the test's ability to predict functional outcomes, 1/3 as both the ability to predict functional outcome and similarity to real-world tasks, and 1/3 were either unclear about the meaning of the term or relied on notions unrelated to classical definitions (e.g., similarity to real-world tasks alone, association with other tests, or the ability to discriminate between populations). CONCLUSIONS: Conceptualizations of the term EV in literature on EF assessment vary grossly, subsuming the notions of criterion, construct, and face validity, as well as sensitivity/specificity. Such inconsistency makes it difficult to interpret clinical utility of tests that are described as ecologically valid. We call on the field to require that, at minimum, the term EV be clearly defined in all publications, or replaced with more concrete terminology (e.g., criterion validity).
Verified 2026-10-05 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
This systematic review of 90 articles found that “ecological validity” (EV) is inconsistently defined and used in neuropsychological research on executive function tests, often conflating veridicality, verisimilitude, and other validity notions. The authors call for retiring the term EV or replacing it with more concrete terminology such as criterion validity.
Key findings
- Among 90 included articles, only 28 (31%) provided a formal EV definition; of those, 14 (50%) required both veridicality and verisimilitude, 10 (36%) required veridicality only, and 4 (14%) required verisimilitude only.
- Across all studies, about two-thirds subscribed to one of the two classic conceptualizations (veridicality alone or veridicality plus verisimilitude), while about one-third used verisimilitude alone, other notions, or were unclear.
- Among 30 articles that did not define EV but linked results to EV conclusions, 18 (60%) cited prediction of functional outcome alone, 5 (17%) cited prediction plus test appearance, and 7 (23%) did not rely on prediction of functional outcomes.
- Conceptualization varied by study purpose [Likelihood Ratio (12) = 34.92, p < .001, Cramer’s V = .37], publication year [Likelihood Ratio (16) = 28.25, p = .030, Cramer’s V = .29], and test type [Likelihood Ratio (8) = 18.65, p = .017, Cramer’s V = .30]; virtual reality and computer tests were most likely to rely on nontraditional definitions.
Methods
- PRISMA-guided systematic review; MEDLINE and PsychINFO were searched on February 18, 2023, with no date limits, using terms targeting “ecologic*,” neuropsychology, and executive function. After screening, 90 articles were included.
- Articles were screened by the first author and independently reviewed by at least one of four coauthors; discrepancies were adjudicated by discussion. Data extraction was done by two authors, with disagreements resolved by discussion or coded as “unspecified.”
- Deductive content analysis coded EV conceptualizations as veridicality, verisimilitude, other notions, or unspecified; correlates included publication year, journal type, test type, and study aims, analyzed with likelihood ratios and Cramer’s V.
Limitations
- The authors state that interpretation of EV conceptualization was sometimes based on subjective impressions because many articles were not explicit; they addressed this with independent double reading and adjudication, and coded 12 articles as “unspecified.”
- The authors state that the deductive approach imposed verisimilitude and veridicality categories, potentially overlooking nuances; the molar approach was conservative, so divergence may be even greater. They also note the search did not include all databases, search terms may have missed articles, and a larger sample would increase power.
- The authors state they examined only articles using EV in the context of novel, face-valid EF tests, did not offer a recommended EV definition, and did not examine methodology or scientific rigor of reviewed articles. Evident from the text: some coding agreement was moderate (e.g., Kappa = .617 for inclusion/exclusion; Kappa = .63 for presence of definition), though disagreements were resolved by discussion.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 2 Not valid · 24 Not checked · 102 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“That said, as seen in Figure 6, the past decade evidences an apparent trend toward returning to the original twopronged conceptualization, perhaps as a function of emerging criticisms of confusing usage (Araújo et al., 2007; Dunlosky et al., 2009; Holleman et al., 2020; Schmuckler, 2001).”
What the sources say
- Supportedcites Araújo et al., 2007 source
The citing sentence presents the decade-long trend as its own interpretation of Figure 6; this citation supports the proposed explanation involving criticism of confusing terminology. In the Abstract, Araújo et al. say that “ecological validity” has often been used incorrectly, and in “Representative experimental design” they criticize its interchangeable use with “representative design.” (LLM, conf=0.87)
- Not checked — no full text of the sourcecites Dunlosky et al., 2009 source
- Disputedcites Holleman et al., 2020 sourceExperimental — accuracy not yet measured
In “The Historical Roots of Ecological Validity” (page unavailable), Holleman et al. describe Brunswik’s original meaning as a specific correlation between a sensory cue and an environmental variable, not a two-pronged conceptualization. The Introduction (page unavailable) says earlier criticisms have largely been overlooked; it does not report a past-decade trend of returning to an original conception. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, this conceptualization appears to have morphed over the years to confound EV with face validity.”
What the sources say
- Could not verifycites Burgess et al., 1998 sourceExperimental — accuracy not yet measured
The Introduction discusses ecological validity in terms of how test performance relates to everyday functioning and describes some tests as formalized real-world activities. It does not address whether the concept changed over time or became confused with face validity; searches for those terms found no such discussion. (LLM, conf=0.70)
- Partly supportedcites Alderman et al., 2003 sourceExperimental — accuracy not yet measured
In “Ecological Validity of the MET-SV” (p. 7), the authors call the test inherently ecologically valid because it resembles an everyday activity. That supports the claim’s point about treating resemblance to real life as ecological validity, but the paper does not discuss face validity or establish that the conceptualization changed over the years. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites Zartman et al., 2013 source
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Criticisms of inconsistent and confusing usage of the term EV in psychological research have been repeatedly raised (Araújo et al., 2007; Dunlosky et al., 2009; Holleman et al., 2020; Schmuckler, 2001).”
What the sources say
- Supportedcites Araújo et al., 2007 source
The Abstract says that “ecological validity” has often been used incorrectly in psychology and that Rogers et al. confused it with representative design, as others had before them. The “Representative experimental design” section also criticizes the frequent interchangeable use of the two terms. These passages support the claim that criticisms of confusing usage have been raised repeatedly. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites Dunlosky et al., 2009 source
- Supportedcites Holleman et al., 2020 source
The Introduction says the term is widely used but subject to conceptual and methodological confusion. In “The ‘Real World’ or the ‘Laboratory’: Psychology’s False Dilemma?” (page not provided), the authors describe inconsistent judgments and definitions over several decades and note an earlier criticism by Hammond (1998). This supports the claim that such criticisms have been raised repeatedly. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Interestingly, as mentioned earlier, some studies that did find evidence for the test ' s ability to predict functional outcome did not link their study results with the term EV, referring instead to convergent or concurrent validity (Chevignard et al., 2010; Finnanger et al., 2022; Gilboa et al., 2019; Kenworthy et al., 2020; Oliveira et al., 2016; Orkin Simon et al., 2022; Pishdadian et al., 202”
What the sources say
- Not checked — no full text of the sourcecites Chevignard et al., 2010 source
- Partly supportedcites Finnanger et al., 2022 sourceExperimental — accuracy not yet measured
The Results section (Convergent Validity—Association Between CCT and Other Measures of EF; page unavailable) reports that better Children’s Cooking Task performance was associated with less disability and better functional outcome, and presents this finding under convergent validity. That supports the claim about terminology and an outcome relationship, but the reported association does not establish that the test predicts future functional outcome. (LLM, conf=0.86)
- Not matchedcites Gilboa et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Indeed, as reviewed above, the notion that highly naturalistic tests can be assumed to be ecologically valid has been repeatedly propagated in the literature (Burgess et al., 1998; Zartman et al., 2013).”
What the sources say
- Supportedcites Burgess et al., 1998 source
The Introduction (page unavailable) explicitly calls tests that formalize real-world activities inherently ‘ecologically valid.’ Burgess et al. thus provide an example of the notion the citing sentence says has appeared in the literature; the paper need not establish how often it has appeared. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites Zartman et al., 2013 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
70 more claims
Create a free account to see all 136 claims and every source verdict.
- Probably not a substantial claim. 42
- It may state two separate claims as one. 36
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“a recent review noted that a substantial proportion of studies claiming ‘ecological validity’ of clinic-based measures were based on qualitative evaluation rather than quantitative comparison of the measures’ relation with relevant real-world outcomes (Suchy et al., 2024).”
From Ecological Validity of Clinic-Based Actigraphy for Assessing Hyperactivity in Clinically Evaluated Children with and without ADHD (2025) · cites this paper as Suchy et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Communication breakdown and a call to action
“some studies in the present review that failed to find any associations between the test and functional outcome nevertheless continued to describe their tests as ecologically valid (Chevignard et al., 2009; Clark et al., 2017; Gilboa et al., 2019)”
§ Psychometric misconceptions
“however, a study exploring the [test] ability to predict real-world functioning was not conducted. (p. 1377)”
Citations
Cites
The Ecology of Human Development: Experiments by Nature and Design
Bronfenbrenner, U. (1979) · Harvard University Press eBooks · doi:10.4159/9780674028845 · Cited by 26,084
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Toward an experimental ecology of human development.
Bronfenbrenner, U. (1977) · American Psychologist · doi:10.1037/0003-066x.32.7.513 · Cited by 10,853
The ecology of human development: Experiments by nature and design
Garbarino, J. (1980) · Children and Youth Services Review · doi:10.1016/0190-7409(80)90036-5 · Cited by 5,527
(1996) · Institutional Research Information System University of Turin (University of Turin) · doi:10.1111/j · Cited by 5,486
Perception and the Representative Design of Psychological Experiments
Brunswik, E. (1956) · doi:10.1525/9780520350519 · Cited by 2,974
Perception and the Representative Design of Psychological Experiments
Brunswik, E. (1957) · doi:10.2307/1419597 · Cited by 2,386
BRST Cohomology of the Chapline-Manton Model
Barnich, G., Leclercq, S., & Spindel, P. (1997) · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) · doi:10.1023/b · Cited by 2,179
DEFICITS IN STRATEGY APPLICATION FOLLOWING FRONTAL LOBE DAMAGE IN MAN
SHALLICE, T., & BURGESS, P. W. (1991) · Brain · doi:10.1093/brain/114.2.727 · Cited by 2,017
Handbook of Metacognition in Education
Hacker, D. J., Dunlosky, J., & Graesser, A. C. (2009) · doi:10.4324/9780203876428 · Cited by 1,509
Research works printed but not linked: 9
- Portney, L. G. , & Gross, K. D. (2020). Concept measurement validity. In Portney, L. G. (Ed.), Foundations of clinical research: Applications to evidence-based practice (4th ed., pp. 127–140). Philadelphia, PA: F. A. Davis.
- Wiedl, 1978, Ecological validity and scholastic success prognosis in learning and intelligence tests: A specimen study, Diagnostica, 24, 175
- Torralva, 2012, “Ecological” and highly demanding executive tasks detect real life deficits in high functioning adult ADHD patients, Journal of Attentional Disorders, 17, 1
- Cripe, 1996, Ecological validity of neuropsychological testing, 171
- Franzen, 1996, Ecological validity of neuropsychological testing, 91
- Klinger, 2004, A virtual supermarket to assess cognitive planning, Annual Review of CyberTherapy and Telemedicine, 2, 49
- Sbordone, 1996, Ecological validity of neuropsychological testing, 15
- Suchy, 2015, Executive functions: A comprehensive guide for clinical practice
- Long, 1996, Ecological validity of neuropsychological testing, 1
Cited by
Horan, W. P., Moore, R. C., Belanger, H. G., & Harvey, P. D. (2024) · Schizophrenia Bulletin Open · doi:10.1093/schizbullopen/sgae025 · Cited by 17
Schlosser, K. K., Cinelli, I., Waelde, T., Álvarez, L. L., Pokorádi, G., Pósch, K., & Whiteley, I. (2025) · Frontiers in Space Technologies · doi:10.3389/frspt.2025.1391331 · Cited by 8
Selective cognitive effects of multilingualism emerge in visuospatial working memory in later life
Maurice, O. R., Jones, C., & Harris, C. B. (2025) · Scientific Reports · doi:10.1038/s41598-025-32091-x · Cited by 6
Manuhuwa, D. M., Boer, M. S.-D., de Graaf, J. W., & Fleer, J. (2024) · European Journal of Educational Research · doi:10.12973/eu-jer.13.4.1647 · Cited by 8
Hug, A., Spingler, T., Pleines, V., Heutehaus, L., Schoenfeld, M. A., Hauptmann, B., Moosburger, J., Thietje, R., Pade, O., Rössy, W., et al. (2025) · Neurological Research and Practice · doi:10.1186/s42466-025-00386-z · Cited by 2
Metadata
- Authors
- Suchy, Y., DesRuisseaux, L. A., Gereau Mora, M., Brothers, S. L., & Niermeyer, M. A.
- Affiliations
- University of Utah (US)
- Corresponding author
- Suchy, Yana
- Venue
- Journal of the International Neuropsychological Society
- Publisher
- Cambridge University Press (CUP)
- Published
- 2024-01-22
- Volume
- 30
- Issue
- 5
- Pages
- 499-522
- ISSN
- 1355-6177 (ISSN-L)
- 1469-7661
- OpenAlex
- W4391109212
- Cited by
- 32
- References
- 157
- Citations by year
- 2026: 20 · 2025: 10 · 2024: 2
- FWCI
- 2.84
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Journal of the International Neuropsychological Society (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Historical and Modern Theater Studies
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
- Keywords
- ecological validity, executive function assessment, neuropsychological assessment, criterion validity, construct validity, face validity, sensitivity and specificity, terminology consistency
- MeSH terms
- Humans, Neuropsychological Tests, Reproducibility of Results, Executive Function
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Conceptualization of the term 'ecological validity' in neuropsychological research on executive function assessment: a systematic review and call to action. https://app.cochat.ai/papers/10.1017/s1355617723000735
- The paper
- Suchy, Y., DesRuisseaux, L. A., Gereau Mora, M., Brothers, S. L., & Niermeyer, M. A. (2024). Conceptualization of the term 'ecological validity' in neuropsychological research on executive function assessment: a systematic review and call to action. Journal of the International Neuropsychological Society. https://doi.org/10.1017/s1355617723000735
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.