A systematic review of AI literacy scales
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index; 44 of the 85 sources its claims cite were checked for support.
Abstract
With the opportunities and challenges stemming from the artificial intelligence developments and its integration into society, AI literacy becomes a key concern. Utilizing quality AI literacy instruments is crucial for understanding and promoting AI literacy development. This systematic review assessed the quality of AI literacy scales using the COSMIN tool aiming to aid researchers in choosing instruments for AI literacy assessment. This review identified 22 studies validating 16 scales targeting various populations including general population, higher education students, secondary education students, and teachers. Overall, the scales demonstrated good structural validity and internal consistency. On the other hand, only a few have been tested for content validity, reliability, construct validity, and responsiveness. None of the scales have been tested for cross-cultural validity and measurement error. Most studies did not report any interpretability indicators and almost none had raw data available. There are 3 performance-based scale available, compared to 13 self-report scales.
Overview
CoChat Summary
TL;DR
This systematic review assessed the quality of AI literacy scales using COSMIN and GRADE criteria, identifying 22 studies that validated or revalidated 16 scales. It found generally good structural validity and internal consistency, but major gaps in content validity, reliability, construct validity, responsiveness, cross-cultural validity, measurement error, interpretability, and open data.
Key findings
- The review identified 22 studies validating 16 AI literacy scales, including 13 self-report scales and 3 performance-based scales, targeting general population, higher education students, secondary education students, and teachers.
- Most scales showed good structural validity and internal consistency, but only a few had been tested for content validity, reliability, construct validity, and responsiveness; none had been tested for cross-cultural validity or measurement error.
- Most studies did not report interpretability indicators, and almost none had raw data available; only MAILS, SAIL4ALL, and SNAIL reported completion times, with MAILS at 16:05 min (SD 5:46), SAIL4ALL true/false at 8:51 min (SD 5:30), SAIL4ALL Likert at 9:38 min (SD 5:52), and SNAIL at 5:39–8:01 min across studies.
- The search began with 5,574 records, reduced to 5,560 after duplicates, then 59 full texts assessed, with 44 excluded, yielding 22 included studies.
Methods
- Systematic review conducted according to PRISMA 2020 and preregistered at OSF; literature searched in Scopus and arXiv on June 18, 2024, with no publication date, type, or stage limits, plus reference-list and reverse-citation searches.
- Inclusion required studies that developed or revalidated an AI literacy scale, provided the full item list, described item formulation, described participants, and described validation techniques.
- Quality assessment used COSMIN criteria for content validity, structural validity, internal consistency, cross-cultural validity, reliability, measurement error, construct validity, responsiveness, interpretability, and feasibility; GRADE was used for overall evidence ratings, and revalidated scales were pooled with random-effect meta-analysis in R using metafor.
Limitations
- The paper states that the review was performed by a single author, which might have caused bias in quality assessment despite quantitatively stated COSMIN criteria.
- The paper states that the search was limited to Scopus and arXiv, so some AI literacy scales may have been missed if published in grey literature, though reverse searches in Scopus and Google Scholar reduced this risk.
- The text makes evident that criterion validity was not evaluated because no gold-standard AI literacy tool existed as of January 2024, and that most included studies lacked open data, missing-data reporting, and target-population content validation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
29 Valid · 7 Not valid · 30 Not checked · 74 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“While the COSMIN tool was originally devised for the medical field, it has since been used in both psychological66,67 and educational research68.”
§ A systematic review of AI literacy scales > Methods > Quality assessment
What the sources say
- Supportedcites [66] Shoman, Y. et al. Psychometric properties of burnout measures: a systematic review. Epidemiol. Psychiatr. Sci. 30, e8 (2021). source
The Data collection and analysis section states that the authors used the COSMIN checklist to assess validation studies of burnout measures. The Methods and analysis section identifies the measures as those used for mental-health professionals, supporting this source’s citation for psychological research. The sentence cites a separate source for educational research. (LLM, conf=0.88)
- Supportedcites [67] Wittkowski, A., Vatter, S., Muhinyi, A., Garrett, C. & Henderson, M. Measuring bonding or attachment in the parent-infant relationship: a systematic review of parent-report assessment measures, their psychometric properties and clinical utility. Clin. Psychol. Rev. 82, 101906 (2020). source
The Methods section (‘Assessing the psychometric properties of included measures’) says the authors used COSMIN criteria and checklists in their review of parent–infant bonding and attachment measures. This supports the psychological-research part of the citing sentence; educational research is assigned a separate citation. (LLM, conf=0.92)
- Supportedcites [68] Rahmatpour, P., Nia, H. S. & Peyrovi, H. Evaluation of psychometric properties of scales measuring student academic satisfaction: a systematic review. J. Educ. Health Promot. 8, 256 (2019). source
Citation 68 supports the educational-research part of the sentence; the psychological-research citations are separate. The Methods (Quality assessment) say the authors used the COSMIN checklist to assess studies of student academic satisfaction scales. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Pinski & Belian’s instrument44 also showed at least some evidence for content validity, but it has been validated on a limited sample, requiring revalidation on a bigger sample in the future.”
§ A systematic review of AI literacy scales > Discussion
What the sources say
- Supportedcites [44] Pinski, M. & Benlian, A. AI Literacy-towards measuring human competency in artificial intelligence. In Hawaii International Conference on System Sciences. https://hdl.handle.net/10125/102649 2023. source
Section 3.3 reports expert interviews and card sorting to assess content validity. Section 3.4 describes a 50-person pre-test, and Section 5 says the sample was at the lower end of the recommended size and calls for testing the model with more observations. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Pinski & Belian’s instrument44 also showed at least some evidence for content validity, but it has been validated on a limited sample, requiring revalidation on a bigger sample in the future.”
§ A systematic review of AI literacy scales > Discussion
What the sources say
- Supportedcites [44] Pinski, M. & Benlian, A. AI Literacy-towards measuring human competency in artificial intelligence. In Hawaii International Conference on System Sciences. https://hdl.handle.net/10125/102649 2023. source
The scale-refinement section reports expert interviews and card sorting to assess content validity, and concludes that the refined item set had appropriate content validity. The pre-test section reports 50 participants, while the Limitations section calls for testing the model with a larger sample. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Lastly, most SAIL4ALL34 subscales showed good evidence for structural validity and internal consistency, however, the psychometric properties of “What can AI do?” subscale are questionable.”
§ A systematic review of AI literacy scales > Discussion
What the sources say
- Supportedcites [34] Soto-Sanfiel, M. T., Angulo-Brunet, A. & Lutz, C. The scale of artificial intelligence literacy for all (SAIL4ALL): a tool for assessing knowledge on artificial intelligence in all adult populations and settings. Preprint at arXiv https://osf.io/bvyku/ (2024). source
Section 4.3.2 reports that the two-factor “What Can AI Do?” model had unsatisfactory fit in Sample 2 and could not be tested in Sample 3 because of convergence issues. Fit was excellent for its strengths dimension but unacceptable for its weaknesses dimension, supporting the qualified assessment that the subscale’s psychometric properties are questionable. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“It is the only scale with evidence for minimal floor and ceiling effects.”
§ A systematic review of AI literacy scales > Discussion
What the sources say
- Could not verifycites [42] Carolus, A. et al. MAILS—meta AI literacy scale: development and testing of an AI literacy questionnaire based on well-founded competency models and psychological change- and meta-competencies. Comput. Hum. Behav. 1, 100014 (2023). sourceExperimental — accuracy not yet measured
Our copy of MAILS nowhere reports or discusses floor or ceiling effects: searches for 'floor and ceiling effects minimal', 'item distribution response categories skewness kurtosis', and psychometric/descriptive-statistics passages returned nothing on that point (Section 3.2 Measures, p. 6 gives only descriptive statistics for AI attitudes and willingness to use technology; Section 3.3 Analysis, pp. 7-8 reports only CFA model fit). The paper's claimed advantages over existing instruments (Sections 2.1, pp. 3-4; 2.3, p. 5; 4.1, p. 10) are grounding in competency taxonomies, modularity, inclusion of psychological meta-competencies, a tested factorial structure and external-criteria validation — not floor/ceiling behaviour — so the uniqueness assertion in the citing sentence cannot be checked against this source. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
63 more claims · 14 with their cited sources checked
Create a free account to see all 140 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 47
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In medicine, this creates a strong requirement for AI literacy: learners need sufficient technical, ethical, legal, and disciplinary knowledge to evaluate output rather than merely consume it (Çalışkan et al., 2022; Gazquez-Garcia et al., 2025; Lintner, 2024).”
From Between Co-Intelligence and Cognitive Offloading: Educators’ Perspectives from Dutch Academic Medical Settings on Generative AI, Critical Thinking, and Adult Professional Learning (2026) · cites this paper as Lintner, 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Reviews of measurement instruments find good internal consistency but persistent gaps in content validity and performance-based assessment”
From From AI Literacy to AI Fluency: A Developmental Framework for Higher Education (2026) · cites this paper as Lintner, 2024
The cited source says what the claim attributes to it.
In the citing paper
“Lintner (2024) found that existing AI literacy scales show good structural validity yet gaps in content validity, cross-cultural validity, and performance-based assessment.”
From From AI Literacy to AI Fluency: A Developmental Framework for Higher Education (2026) · cites this paper as Lintner, 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A systematic review of AI literacy scales > Results > Overview of AI literacy scales
“| MAIRS-MC43,48 | self-report | 22 5-point Likert items | medical students | 865, 502 | 21.3 ± 2.0, 22.7 ± 2.8 | 4 | (F1) cognition (F2) ability (F3) vision (4) ethics |”
§ A systematic review of AI literacy scales > Results > Quality assessment
“#### MAIRS-MS—medical artificial intelligence readiness scale for medical students43”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A systematic review of AI literacy scales > Results > Quality assessment
“| AI literacy test30 | 18.8 | 5.6 | ? | ? | ? | ? |”
§ A systematic review of AI literacy scales > Results > Overview of AI literacy scales
“| AI literacy test32 | performance-based | 31 including 30 multiple-choice items with 4 options each and 1 sorting item | university students | 1286 | 23.6 ± 4.5 | 1 | single factor |”
Cited works that were retracted
Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.
A note, not an issue: for example, a cited work that was retracted only after this paper was published, which its authors could not have known.
In the paper
“Kumar, Y. et al. Artificial intelligence in disease diagnosis: a systematic literature review, synthesizing framework and future research agenda. J. Ambient Intell. Humaniz. Comput. 14, 8459–8486 (2023).”
Cited work 10.1007/s12652-021-03612-z · retracted 2026-03-09 · notice 10.1007/s12652-026-05063-w · this paper published 2024-08-06
Citations
Cites
1 retracted
Kumar, Y., Koul, A., Singla, R., & Ijaz, M. F. (2023) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03612-z · Cited by 1,040
RetractedConducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge
Mishra, P., & Koehler, M. J. (2006) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1111/j.1467-9620.2006.00684.x · Cited by 12,604
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Computer Self-Efficacy: Development of a Measure and Initial Test
Compeau, D. R., & Higgins, C. A. (1995) · MIS Quarterly · doi:10.2307/249688 · Cited by 6,553
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. W. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.006 · Cited by 4,465
COSMIN guideline for systematic reviews of patient-reported outcome measures
Prinsen, C. A. C., Mokkink, L. B., Bouter, L. M., Alonso, J., Patrick, D. L., de Vet, H. C. W., & Terwee, C. B. (2018) · Quality of Life Research · doi:10.1007/s11136-018-1798-3 · Cited by 3,345
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241
COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures
Mokkink, L. B., de Vet, H. C. W., Prinsen, C. A. C., Patrick, D. L., Alonso, J., Bouter, L. M., & Terwee, C. B. (2018) · Quality of Life Research · doi:10.1007/s11136-017-1765-4 · Cited by 2,479 · cited 3×
“Quality assessment I assessed the quality of the scales based on the COSMIN 52 – 56 measurement properties and additionally on interpretability and feasibility.”
Terwee, C. B., Prinsen, C. A. C., Chiarotto, A., Westerman, M. J., Patrick, D. L., Alonso, J., Bouter, L. M., de Vet, H. C. W., & Mokkink, L. B. (2018) · Quality of Life Research · doi:10.1007/s11136-018-1829-0 · Cited by 2,348 · cited 2×
“…quality of the individual studies by using the COnsensus ‑ based Standards for the selection of health Measurement INstruments (COSMIN) 52 – 54 for the self-report scales, and additionally the COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of…”
Research works printed but not linked: 4
- Kim, S. W. & Lee, Y. The artificial intelligence literacy scale for middle school. Stud. J. Korea Soc. Comput. Inf. 27, 225–238 (2022).
- Çelebi, C. et al. Artificial intelligence literacy: an adaptation study. Instr. Technol. Lifelong Learn 4, 291–306 (2023).
- Schünemann, H., Brożek, J., Guyatt, G. & Oxman, A. (Eds.) GRADE Handbook. https://gdt.gradepro.org/app/handbook/handbook.html (2013).
- Ali, S., Payne, B. H., Williams, R., Park, H. W. & Breazeal, C. Constructionism, ethics, and creativity: Developing primary and middle school artificial intelligence education. International Workshop on Education in Artificial Intelligence K-12 (EDUAI’19) 1–4 (2019).
Cited by
Educational Strategies for Clinical Supervision of Artificial Intelligence Use
Abdulnour, R.-E. E., Gin, B., & Boscardin, C. K. (2025) · New England Journal of Medicine · doi:10.1056/nejmra2503232 · Cited by 209
Mah, D.-K., & Groß, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00490-1 · Cited by 218
Wijaya, T. T., Yu, Q., Cao, Y., He, Y., & Leung, F. K. S. (2024) · Behavioral Sciences · doi:10.3390/bs14111008 · Cited by 103
Artificial Intelligence and Radiologist Burnout
Liu, H., Ding, N., Li, X., Chen, Y., Sun, H., Huang, Y., Liu, C., Ye, P., Jin, Z., Bao, H., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.48714 · Cited by 119
GLAT: The generative AI literacy assessment test
Jin, Y., Martinez-Maldonado, R., Gašević, D., & Yan, L. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100436 · Cited by 76
Metadata
- Authors
- Lintner, T.
- Affiliations
- Masaryk University (CZ)
- Corresponding author
- Lintner, Tomáš
- ORCID
- Lintner, Tomáš
- Venue
- npj Science of Learning
- Published
- 2024-08-06
- Volume
- 9
- Issue
- 1
- Pages
- 50-50
- ISSN
- 2056-7936 (ISSN-L)
- PMC
- PMC11303566
- OpenAlex
- W4401371637
- Cited by
- 292
- References
- 61
- Citations by year
- 2026: 209 · 2025: 72 · 2024: 11
- FWCI
- 16.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Science of Learning (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Veřejné služby Informačního systému (Masarykiana Brunensis Universitas) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- European Commission (LX22NPO5101)
- NextGenerationEU
- Topics
- Social Sciences › Education › Digital literacy in education
- Social Sciences › Education › Artificial Intelligence in Education
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- AI literacy, scale validation, psychometric properties, COSMIN, measurement invariance, content validity, construct validity, reliability statistics, self-report scale, performance-based assessment, university students, teachers, high school students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A systematic review of AI literacy scales. https://app.cochat.ai/papers/10.1038/s41539-024-00264-4
- The paper
- Lintner, T. (2024). A systematic review of AI literacy scales. npj Science of Learning. https://doi.org/10.1038/s41539-024-00264-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
