Threats to the validity of the Collegiate Learning Assessment (CLA+) as a measure of critical thinking skills and implications for Learning Gain
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
The University of Reading Learning Gain project is a three-year longitudinal project to test and evaluate a range of available methodologies and to draw conclusions on what might be the right combination of instruments for the measurement of Learning Gain in higher education. This paper analyses the validity of a measure of critical thinking skills, the Collegiate Learning Assessment (CLA+) and the implications of using this standardised test as a proxy for Learning Gain. The paper reviews five inferences regarding the interpretations and use of test scores: construct representation, scoring, generalisation, extrapolation and decision-making. Each section reviews some of the available evidence in support of the claims the CLA+ makes and the threats to their validity. The possible impact of these issues on Learning Gain in the UK is considered.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dual-Process Theories of Higher Cognition: Advancing the Debate
Evans, J. S. B. T., & Stanovich, K. E. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691612460685 · Cited by 4,271 · cited 1×
“Table 2 summarises some of the definitions that might have informed Facione (1990), but many others are available (e.g. Elder, 2007; Halpern, 2013; see Lai, 2011, for a review).”
Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, W. B. G. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.197 · Cited by 2,517
Validating the interpretations and uses of test scores
Kane, M. T. (2013) · Journal of Educational Measurement · doi:10.1111/jedm.12000 · Cited by 2,110
(2011) · doi:10.7208/chicago/9780226028576.001.0001 · Cited by 562
Indeterminacy in the use of preset criteria for assessment and grading
Sadler, D. R. (2009) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930801956059 · Cited by 489 · cited 2×
“By allowing for extreme opposite performances on different criteria to average out (Sadler, 2009), marks will naturally converge towards the mean.”
The Construct of Content Validity
Sireci, S. G. (1998) · Social Indicators Research · doi:10.1023/a:1006985528729 · Cited by 430
Liu, O. L., Frankel, L., & Roohr, K. C. (2014) · ETS Research Report Series · doi:10.1002/ets2.12009 · Cited by 340
The development and psychometric validation of a Critical Thinking Disposition Scale
Sosu, E. M. (2013) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2012.09.002 · Cited by 324
Do instructional interventions influence college students’ critical thinking skills? A meta-analysis
Niu, L., Behar-Horenstein, L. S., & Garvan, C. W. (2013) · Educational Research Review · doi:10.1016/j.edurev.2012.12.002 · Cited by 244
Bassok, M., & Novick, L. R. (2012) · The Oxford Handbook of Thinking and Reasoning · doi:10.1093/oxfordhb/9780199734689.013.0021 · Cited by 60
Research works printed but not linked: 87
- 10.1007/s10648-014-9283-1 · DOI
- 10.1007/s12115-013-9648-y · doi:10.1007/s12115-013-9648-y
- Arum R., 2012, Documenting uncertain times: Post-graduate transitions of the Academically Adrift cohort
- Arum R., 2011, Improving undergraduate learning: Findings and policy recommendations from the SSRC-CLA longitudinal project
- 10.1002/9780470996294.ch11 · DOI · Paper page
- 10.1097/NNR.0000000000000079 · DOI
- Benjamin R., 2013, The case for critical-thinking skills and performance assessment
- 10.1097/NNA.0b013e3182118528 · DOI
- 10.1016/j.tsc.2012.04.003 · DOI
- Bok D.C., 2006, Our underachieving colleges: A candid look at how much students learn and why they should be learning more
- 10.1097/00001416-201307000-00004 · doi:10.1097/00001416-201307000-00004
- Brunt B.A., 2005, The Journal of Continuing Education in Nursing, 36, 255, 10.3928/0022-0124-20051101-05 · DOI · Paper page
- 10.1007/s10648-014-9284-0 · DOI
- Cambridge Assessment Admissions Testing . (2017). Thinking skills assessment . Retrieved from http://www.admissionstesting.org/for-test-takers/thinking-skills-assessment/
- 10.1016/j.nedt.2015.02.023 · DOI · Paper page
- 10.1016/j.nedt.2016.03.010 · DOI
- Council for Aid to Education, 2014, CLA+: Technical FAQ
- Council for Aid to Education, 2015, Spring 2015 CLA+ mastery results: Sample university (Institutional report)
- 10.2801/3701 · doi:10.2801/3701
- 10.1037/1040-3590.7.3.309 · DOI · Paper page
- 10.1037/0021-9010.78.1.98 · DOI · Paper page
- Coutinho S.A., 2006, Educational Research and Reviews, 1, 162
- 10.1037/h0040957 · DOI · Paper page
- Dalkey N., 1962, An experimental application of the Delphi method to the use of experts (Memorandum RM-727/1-abridged)
- Del Bueno D.J., 1990, Nursing and Health Care, 11, 290
- Ennis R.H., 1962, Harvard Educational Review, 32, 81
- Ennis R.H., 1985, Cornell critical thinking tests level X and level Z manual
- Ennis R.H., 1987, Teaching thinking skills theory and practice, 9
- Ennis R.H., 1985, The Ennis-Weir critical thinking essay test
- 10.1353/jge.2003.0019 · doi:10.1353/jge.2003.0019
- Facione P.A., 1990, Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction – The Delphi Report (Executive Summary)
- Facione P.A., 1992, The California Critical Thinking Skills Test manual
- Facione P.A., 1992, The California Critical Thinking Dispositions Inventory test manual
- 10.1080/02680939.2016.1268271 · DOI
- 10.1027/1614-0001/a000119 · DOI · Paper page
- Glenn D., 2011, The Chronicle of Higher Education
- 10.1002/9780470973394 · DOI · Paper page
- Halpern D.F., 2013, Critical thinking across the curriculum: A brief edition of thought and knowledge
- 10.17507/tpls.0504.11 · DOI
- Insight Assessment, 2013, California critical thinking disposition inventory (CCTDI)
- 10.1007/s10503-015-9356-4 · DOI
- Klein S.P., 1996, The Bar Examiner, 65, 13
- 10.1214/193940307000000392 · DOI
- 10.1177/0193841X07303318 · DOI · Paper page
- 10.1007/s11162-004-1640-3 · DOI
- Klein S.P., 2009, Test Validity Study (TVS) report
- 10.1016/j.ijnurstu.2013.06.009 · DOI · Paper page
- 10.1111/j.1540-4609.2009.00243.x · DOI
- 10.1177/0098628315587624 · DOI
- 10.1186/s12909-016-0761-7 · DOI · Paper page
- 10.1007/s12115-013-9649-x · doi:10.1007/s12115-013-9649-x
- Lipman M., 1987, Analytic Teaching, 8, 5
- 10.1080/02602938.2016.1168358 · DOI · Paper page
- McGrath C.H., 2015, Learning gain in higher education, 10.7249/RR996 · DOI · Paper page
- 10.1080/00221309.1992.9921158 · DOI
- McPeck J.E., 1981, Critical thinking and education
- 10.1353/pla.2014.0020 · doi:10.1353/pla.2014.0020
- 10.1111/jedm.1984.21.issue-3 · doi:10.1111/jedm.1984.21.issue-3
- Messick S., 1989, Educational measurement, 13
- 10.1007/s10648-014-9281-3 · DOI
- 10.1080/09695940701478321 · DOI · Paper page
- 10.1080/0969594X.2015.1037241 · DOI
- 10.1016/j.ijer.2015.06.004 · DOI
- Opposs D., 2011, The reliability programme: Final report of the technical advisory group
- Pascarella E.T., 2005, How college affects students: a third decade of research
- 10.1002/cc.36819927703 · DOI · Paper page
- Popham W.J., 1997, Educational Leadership, 55, 72
- 10.1016/j.nedt.2014.01.011 · DOI
- 10.3928/01484834-20100331-05 · doi:10.3928/01484834-20100331-05
- 10.1080/10437797.2016.1151270 · DOI
- 10.1080/00091380309604739 · doi:10.1080/00091380309604739
- Shaw S.D., 2012, Research Matters: A Cambridge Assessment Publication, 3
- Siegel H., 1988, Educating reason: Rationality, critical thinking and education
- 10.1080/00461520.2015.1125787 · DOI
- Stedman N.L.P., 2009, NACTA Journal, 53, 62
- 10.1080/02602938.2011.560720 · DOI
- 10.1080/08957347.2013.853072 · DOI
- 10.1016/j.tsc.2016.11.007 · DOI · Paper page
- 10.5116/ijme.4dfb.8dfd · DOI · Paper page
- 10.1023/A:1005166416808 · DOI
- 10.1080/1356251990040409 · doi:10.1080/1356251990040409
- Watson G., 1980, Watson-Glaser critical thinking appraisal manual
- 10.1080/03075079.2015.1004239 · DOI
- Zahner D., 2014, Reliability and validity of CLA+
- Zahner D., 2014, Standard-setting study final report
- Zahner D., 2015, Predictive validity of a critical thinking assessment for post-college outcomes
- 10.1111/ijn.12347 · DOI · Paper page
Cited by
Leveraging ChatGPT for Enhancing Critical Thinking Skills
Guo, Y., & Lee, D. (2023) · Journal of Chemical Education · doi:10.1021/acs.jchemed.3c00505 · Cited by 250
Making sense of learning gain in higher education
Evans, C., Kandiko Howson, C., & Forsythe, A. (2018) · Higher Education Pedagogies · doi:10.1080/23752696.2018.1508360 · Cited by 81
Kleemola, K., Hyytinen, H., & Toom, A. (2022) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2021.1946482 · Cited by 38
Hyytinen, H., Jämsä, M., Tuononen, T., & Kleemola, K. (2025) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2025.2553341 · Cited by 9
Metadata
- Authors
- Aloisi, C., & Callaghan, A.
- Venue
- Higher Education Pedagogies
- Publisher
- Informa UK Limited
- Published
- 2018-01
- Volume
- 3
- Issue
- 1
- Pages
- 57-82
- ISSN
- 2375-2696 (ISSN-L)
- OpenAlex
- W2792650584
- Cited by
- 26
- References
- 10
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- Higher Education Funding Council for England (10007802)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Threats to the validity of the Collegiate Learning Assessment (CLA+) as a measure of critical thinking skills and implications for Learning Gain. https://app.cochat.ai/papers/10.1080/23752696.2018.1449128
- The paper
- Aloisi, C., & Callaghan, A. (2018). Threats to the validity of the Collegiate Learning Assessment (CLA+) as a measure of critical thinking skills and implications for Learning Gain. Higher Education Pedagogies. https://doi.org/10.1080/23752696.2018.1449128
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.