Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 11 of the 26 sources its claims cite were checked for support.
Abstract
A growing number of courses seek to increase the basic artificial-intelligence skills (“AI literacy”) of their participants. At this time, there is no valid and reliable measurement tool that can be used to assess AI-learning gains. However, the existence of such a tool would be important to enable quality assurance and comparability. In this study, a validated AI-literacy-assessment instrument, the “scale for the assessment of non-experts’ AI literacy” (SNAIL) was adapted and used to evaluate an undergraduate AI course. We investigated whether the scale can be used to reliably evaluate AI courses and whether mediator variables, such as attitudes toward AI or participation in other AI courses, had an influence on learning gains. In addition to the traditional mean comparisons (i.e., t-tests), the comparative self-assessment (CSA) gain was calculated, which allowed for a more meaningful assessment of the increase in AI literacy. We found preliminary evidence that the adapted SNAIL questionnaire enables a valid evaluation of AI-learning gains. In particular, distinctions among different subconstructs and the differentiation constructs, such as attitudes toward AI, seem to be possible with the help of the SNAIL questionnaire.
Overview
CoChat Summary
TL;DR
The study adapted the SNAIL AI-literacy scale for evaluating an undergraduate AI course using retrospective/post comparative self-assessment. It found preliminary evidence that the adapted questionnaire validly and reliably assessed AI-learning gains, while attitudes toward AI and prior AI education showed limited or no relationship with learning gains.
Key findings
- Across 30 AI-literacy items, most showed significant improvement; only three items failed statistical significance: explaining general vs. narrow AI, explaining data-privacy considerations, and identifying ethical issues. Cohen’s d was below 0.5 for five items, while 14 items showed strong effects (d > 0.8).
- At factor level, significant retrospective-to-post differences were found for technical understanding, t(48) = 4.38, p < 0.001, d = 1.25; critical appraisal, t(48) = 3.47, p < 0.001, d = 1.21; and practical application, t(48) = 3.30, p < 0.001, d = 0.93. Mean CSA gain was 36.9% for technical understanding, 30.1% for critical appraisal, and 37.2% for practical application; the artificial-neural-network item had 56.8% CSA gain, while the ethics item had 15.6%.
- Reliability was good to excellent: Cronbach’s α for retrospective assessments was 0.90 (technical understanding), 0.92 (critical appraisal), and 0.85 (practical application); post-assessment α was 0.83, 0.90, and 0.88, respectively.
- Correlations between SNAIL factors and the two “Attitudes Toward Artificial Intelligence” factors (fear and acceptance) did not reach significance. Prior “other courses” and “other AI education” correlated with retrospective/post technical-understanding scores but not with CSA gain; e.g., other courses with TU retrospective ρ = 0.556, p < 0.01 and TU post ρ = 0.402, p = 0.046.
Methods
- The evaluated course was an interdisciplinary undergraduate AI course with a technical focus on artificial neural networks, approximately 150 h total. Participants were 25 students (62.5% of all attendees), mean age 22.9 years (SD = 2.3), sixth semester on average (M = 6.0, SD = 2.9), with 16 men (64%) and 9 women (36%).
- The SNAIL questionnaire was translated from English into German using independent forward and back translations plus an expert panel. Each of 31 SNAIL items was presented as a retrospective assessment and a post-assessment; one technical-understanding item was excluded due to a technical error, leaving 30 items (13 instead of 14 TU items). Participants also completed the five-item “Attitudes Toward Artificial Intelligence” scale and sociodemographic/prior-AI-education questions.
- Analyses used independent t-tests at item and factor level, with Levene tests and a Welch test where homoscedasticity was violated; CSA gain was calculated; Spearman rank correlations were used for metric-ordinal relationships and Pearson correlations for metric variables; internal consistency was assessed with Cronbach’s alpha using Microsoft Excel or IBM SPSS Statistics version 27.
Limitations
- Stated in the paper: only one course and a single sample were examined; some participants came from computer science backgrounds and were technically inclined, whereas the original SNAIL was aimed at non-experts; the selection of SNAIL was relatively arbitrary; and for evaluation efficiency, other adapted questionnaires could not be tested.
- Evident from the text: the sample was small (n = 25) and formal enrollment was unavailable, with attendance estimated at about 40; the design relied on self-report and retrospective self-assessment; no retention or test-retest reliability was reported; and the course’s technical focus may limit generalizability to courses emphasizing ethical, legal, or social AI content.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
7 Valid · 3 Not valid · 5 Not checked · 55 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“The retrospective/post method is often more suitable for assessing learning gains than the traditional pre/post test (one assessment before and one after a course, [28]) because it is subject to fewer biases.”
What the sources say
- Supportedcites [28] Raupach, 2011, Towards outcome-based programme evaluation: Using student comparative self-assessments to determine teaching effectiveness, Med. Teach. source
The citation is attached to the parenthetical definition of a traditional pre/post test, not to the separate assertion that the retrospective/post method has fewer biases. The Objective performance measurements section describes testing students on the first day and again during the last week of the module, supporting that definition. No page number is available in our copy. (LLM, conf=0.88)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“To ensure a valid and systematic translation of the SNAIL items, we followed the international recommendations for translating psychological measurement instruments wherever possible [31-33].”
What the sources say
- Supportedcites [31] Tsang, 2017, Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine, Saudi J. Anaesth. source
The paper’s “Translating a Questionnaire” section provides guidelines for translating measurement questionnaires into another language, including recommendations for independent forward translators. It supports the citation as a source of translation guidance; the citing authors’ statement that they followed the guidance is their own account, not a finding attributed to this paper. (LLM, conf=0.87)
- Not matchedcites [32] Harkness, J., Pennell, B.E., and Schoua-Glusberg, A. (2004). Methods for Testing and Evaluating Survey Questionnaires, John Wiley & Sons, Inc.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [33] Chang, 2001, Translation of questionnaires and issues of equivalence, J. Adv. Nurs. source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The so-called student comparative self-assessment [28,34] is a more valid tool to assess the actual increase in competence or knowledge, as it accounts for the initial level of participants' AI literacy.”
What the sources say
- Partly supportedcites [28] Raupach, 2011, Towards outcome-based programme evaluation: Using student comparative self-assessments to determine teaching effectiveness, Med. Teach. sourceExperimental — accuracy not yet measured
In ‘Development of the new programme evaluation tool’ (page unavailable), the authors calculate gain from pre- and post-course self-assessments and adjust for differing initial performance levels. Their study concerns medical education, not participants’ AI literacy. In ‘Limitations and suggestions for future research’ (page unavailable), they say further research is needed to establish the approach’s validity for factual knowledge, so the claim that it is a more valid measure of actual knowledge gain overstates the evidence. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [34] Schiekirka, 2013, Estimating learning outcomes from pre-and posttest student self-assessments: A longitudinal study, Acad. Med. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This could be an indication of discriminant validity because, in theory, AI literacy and attitudes toward AI are assumed to be two different constructs, representing cognitive/skill and affective aspects, respectively [9,16,20-22].”
What the sources say
- Not checked — no full text of the sourcecites [9] Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., and Huber, P. (2016, January 12–15). Artificial intelligence and computer science in education: From kindergarten to university. Proceedings of the 2016 IEEE Frontiers in Education Conference (FIE), Erie, PA, USA. source
- Not matchedcites [16] Kirkpatrick, D., and Kirkpatrick, J. (2006). Evaluating Training Programs: The Four Levels, Berrett-Koehler Publishers.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [20] Schepman, 2022, The General Attitudes towards Artificial Intelligence Scale (GAAIS): Confirmatory validation and associations with personality, corporate distrust, and general trust, Int. J. Hum. Comput. Interact. source
- Partly supportedcites [21] Sindermann, 2021, Assessing the attitude towards artificial intelligence: Introduction of a short measure in German, Chinese, and English language, KI Künstl. Intell. sourceExperimental — accuracy not yet measured
The Results (p. 5) identify acceptance and fear as the two factors measured by the AI-attitude scale, supporting the attitude side of the distinction. The paper does not examine AI literacy or establish that literacy and attitudes are distinct cognitive/skill and affective constructs. The weak correlation and proposed discriminant validity are the citing authors’ interpretation, not a finding attributed to this source. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [22] Wang, 2022, Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale, Behav. Inf. Technol. source
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some items showed rather small improvements, in the range of 15 to 30%”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
27 more claims
Create a free account to see all 70 claims and every source verdict.
- Probably not a substantial claim. 21
- It may state two separate claims as one. 26
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We distinguish between generic AIL (Beginner level in all dimensions) as the baseline required for all HE students (Laupichler et al., 2023; Long and Magerko, 2020; Ng et al., 2021)”
From The AI literacy heptagon: A structured approach to AI literacy in higher education (2026) · cites this paper as Laupichler et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“SNAIL45 was constructed on an elaborate Delphi study63, it has been revalidated in another three studies49–51 including one with comparative self-assessment gains49, it is one of the few scales with evidence of reliability and responsiveness, and it demonstrated good structural validity and internal consistency, which makes it a promising instrument.”
From A systematic review of AI literacy scales (2024) · cites this paper as [49]
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentt(48) = 4.38reported p < .001 · recomputed p < .001
“t (48) = 4.38, p < .001”
- Consistentt(48) = 3.47reported p < .001 · recomputed p = .001
“t (48) = 3.47, p < .001”
The paper states a one-tailed test
- Consistentt(48) = 3.3reported p < .001 · recomputed p = .002
“t (48) = 3.30, p < .001”
The paper states a one-tailed test
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1.3. Using an AI-Literacy-Assessment Instrument to Evaluate Learning Gains · p. 3
“the original scale by Laupichler et al. [28] was originally validated in English”
§ References · p. 12
“28. Raupach, T.; Münscher, C.; Beißbarth, T.; Burckhardt, G.; Pukrop, T. Towards outcome-based programme evaluation: Using student comparative self-assessments to determine teaching effectiveness.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.3. Evaluation Procedure · p. 4
“Participants were then presented with the five items of the 'Attitudes Toward Artificial Intelligence' scale by Sindermann et al. [22].”
§ References · p. 12
“22. Wang, B.; Rau PL, P.; Yuan, T. Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4. Relationship between AI Education Prior to the Course and Learning Gains · p. 8
“Although the correlations between 'other courses' and CSA gain were negative for all three factors, this correlation did not reach statistical significance (significance level of α = 0.5).”
§ 3.4. Relationship between AI Education Prior to the Course and Learning Gains · p. 8
“| 10. Other courses | - 0.111 0.597 | - 0.348 0.088 | - 0.244 0.239 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.4. Data Analysis · p. 5
“The calculation of the comparative self-assessment (CSA) gain is described in Raupach et al.'s 2011 article, 'Towards outcome-based programme evaluation: using student comparative self-assessments to determine teaching effectiveness' [34]”
§ References · p. 12
“34. Schiekirka, S.; Reinhardt, D.; Beibarth, T.; Anders, S.; Pukrop, T.; Raupach, T. Estimating learning outcomes from pre-and posttest student self-assessments: A longitudinal study. Acad. Med. 2013”
Citations
Cites
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 2×
“They described AI literacy as “a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace” [8], (p. 2).”
BackgroundArtificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Conceptualizing AI literacy: An exploratory review
Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100041 · Cited by 2,205 · cited 2×
“For reasons of evaluation efficiency, it was not possible to examine how respondents would have responded to other adapted questionnaires (e.g., [23–25]).”
BackgroundMethodTsang, S., Royse, C., & Terkawi, A. (2017) · Saudi Journal of Anaesthesia · doi:10.4103/sja.sja_203_17 · Cited by 1,692 · cited 1×
“To ensure a valid and systematic translation of the SNAIL items, we followed the international recommendations for translating psychological measurement instruments wherever possible [31–33].”
MethodWang, B., Rau, P.-L. P., & Yuan, T. (2023) · Behaviour & Information Technology · doi:10.1080/0144929x.2022.2072768 · Cited by 1,036 · cited 4×
“The first AI literacy scale was published by [22] and included four sub-factors of AI literacy: “awareness”, “usage”, “evaluation”, and “ethics”.”
BackgroundMethodInfluentialApplications of artificial intelligence in intelligent manufacturing: a review
Li, B.-H., Hou, B.-C., Yu, W.-T., Lu, X.-B., & Yang, C.-W. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601885 · Cited by 826
Initial validation of the general attitudes towards Artificial Intelligence Scale
Schepman, A., & Rodway, P. (2020) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2020.100014 · Cited by 785 · cited 1×
“Several relatively well-validated scales already exist that try to capture affective attitudes toward AI [19–21].”
BackgroundAI literacy in K-12: a systematic literature review
Casal-Otero, L., Catala, A., Fernández-Morante, C., Taboada, M., Cebreiro, B., & Barro, S. (2023) · International Journal of STEM Education · doi:10.1186/s40594-023-00418-7 · Cited by 724
Artificial intelligence literacy in higher and adult education: A scoping literature review
Laupichler, M. C., Aster, A., Schirch, J., & Raupach, T. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100101 · Cited by 658
Self-citationSouthworth, J., Migliaccio, K., Glover, J., Glover, J., Reed, D., McCarty, C., Brendemuhl, J., & Thomas, A. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100127 · Cited by 647 · cited 1×
“Introductory courses on AI are offered at a wide variety of educational levels, starting as early as kindergarten and elementary school [9,10], continuing through K-12 education [11–13], and ending with higher education and adult education in universities and similar institutions [14,15].”
Background
Research works printed but not linked: 2
- Cambridge Dictionary (2023, September 14). Artificial Intelligence. Available online: https://dictionary.cambridge.org/dictionary/english/artificial-intelligence.
- Harkness, J., Pennell, B.E., and Schoua-Glusberg, A. (2004). Methods for Testing and Evaluating Survey Questionnaires, John Wiley & Sons, Inc.
Cited by
A systematic review of AI literacy scales
Lintner, T. (2024) · npj Science of Learning · doi:10.1038/s41539-024-00264-4 · Cited by 292
Laupichler, M. C., Aster, A., Meyerheim, M., Raupach, T., & Mergen, M. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05400-7 · Cited by 170
“In another study, it was demonstrated that the final SNAIL questionnaire is also suitable for assessing AI literacy among university students who have just completed an AI course [22].”
BackgroundMethodSelf-citationBiagini, G. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00466-w · Cited by 336
Artificial intelligence literacy among university students—a comparative transnational survey
Mansoor, H. M. H., Bawazir, A., Alsabri, M. A., Alharbi, A., & Okela, A. H. (2024) · Frontiers in Communication · doi:10.3389/fcomm.2024.1478476 · Cited by 84
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
Metadata
- Authors
- Laupichler, M. C., Aster, A., Perschewski, J.-O., & Schleiss, J.
- Affiliations
- University Hospital Bonn (DE)
- Otto-von-Guericke-Universität Magdeburg (DE)
- Corresponding author
- Laupichler, Matthias Carl
- Venue
- Education Sciences
- Publisher
- MDPI AG
- Published
- 2023-09-26
- Volume
- 13
- Issue
- 10
- Pages
- 978
- ISSN
- 2227-7102 (ISSN-L)
- OpenAlex
- W4387048697
- Cited by
- 48
- References
- 30
- Citations by year
- 2026: 19 · 2025: 18 · 2024: 11
- FWCI
- 11.33
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Education Sciences (publishedVersion, cc-by)
- bonndoc (University of Bonn) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Social Sciences › Education › Digital literacy in education
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- AI literacy, AI literacy assessment, learning gains, attitudes toward AI, measurement instrument validation, reliability statistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment. https://app.cochat.ai/papers/10.3390/educsci13100978
- The paper
- Laupichler, M. C., Aster, A., Perschewski, J.-O., & Schleiss, J. (2023). Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment. Education Sciences. https://doi.org/10.3390/educsci13100978
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.