CoChat

A systematic review of AI literacy scales

Open access
Lintner, Tomáš · 2024 · npj Science of Learning

The DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index; 44 of the 85 sources its claims cite were checked for support.

Abstract

With the opportunities and challenges stemming from the artificial intelligence developments and its integration into society, AI literacy becomes a key concern. Utilizing quality AI literacy instruments is crucial for understanding and promoting AI literacy development. This systematic review assessed the quality of AI literacy scales using the COSMIN tool aiming to aid researchers in choosing instruments for AI literacy assessment. This review identified 22 studies validating 16 scales targeting various populations including general population, higher education students, secondary education students, and teachers. Overall, the scales demonstrated good structural validity and internal consistency. On the other hand, only a few have been tested for content validity, reliability, construct validity, and responsiveness. None of the scales have been tested for cross-cultural validity and measurement error. Most studies did not report any interpretability indicators and almost none had raw data available. There are 3 performance-based scale available, compared to 13 self-report scales.

Overview

CoChat Summary

TL;DR

This systematic review assessed the quality of AI literacy scales using COSMIN and GRADE criteria, identifying 22 studies that validated or revalidated 16 scales. It found generally good structural validity and internal consistency, but major gaps in content validity, reliability, construct validity, responsiveness, cross-cultural validity, measurement error, interpretability, and open data.

Key findings

  • The review identified 22 studies validating 16 AI literacy scales, including 13 self-report scales and 3 performance-based scales, targeting general population, higher education students, secondary education students, and teachers.
  • Most scales showed good structural validity and internal consistency, but only a few had been tested for content validity, reliability, construct validity, and responsiveness; none had been tested for cross-cultural validity or measurement error.
  • Most studies did not report interpretability indicators, and almost none had raw data available; only MAILS, SAIL4ALL, and SNAIL reported completion times, with MAILS at 16:05 min (SD 5:46), SAIL4ALL true/false at 8:51 min (SD 5:30), SAIL4ALL Likert at 9:38 min (SD 5:52), and SNAIL at 5:39–8:01 min across studies.
  • The search began with 5,574 records, reduced to 5,560 after duplicates, then 59 full texts assessed, with 44 excluded, yielding 22 included studies.

Methods

  • Systematic review conducted according to PRISMA 2020 and preregistered at OSF; literature searched in Scopus and arXiv on June 18, 2024, with no publication date, type, or stage limits, plus reference-list and reverse-citation searches.
  • Inclusion required studies that developed or revalidated an AI literacy scale, provided the full item list, described item formulation, described participants, and described validation techniques.
  • Quality assessment used COSMIN criteria for content validity, structural validity, internal consistency, cross-cultural validity, reliability, measurement error, construct validity, responsiveness, interpretability, and feasibility; GRADE was used for overall evidence ratings, and revalidated scales were pooled with random-effect meta-analysis in R using metafor.

Limitations

  • The paper states that the review was performed by a single author, which might have caused bias in quality assessment despite quantitatively stated COSMIN criteria.
  • The paper states that the search was limited to Scopus and arXiv, so some AI literacy scales may have been missed if published in grey literature, though reverse searches in Scopus and Google Scholar reduced this risk.
  • The text makes evident that criterion validity was not evaluated because no gold-standard AI literacy tool existed as of January 2024, and that most included studies lacked open data, missing-data reporting, and target-population content validation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41539-024-00264-4

Cites

71 of 72 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 PRISMA flowchart. The PRISMA flowchart shows the study identification procedure.
Fig. 1 PRISMA flowchart. The PRISMA flowchart shows the study identification procedure.CC BY · PubMed Central

Verification

Claims the paper makes

29 Valid · 7 Not valid · 30 Not checked · 74 Own results and interpretations

63 more claims · 14 with their cited sources checked

Create a free account to see all 140 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 30
  • It may state two separate claims as one. 47
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

1 of 1

Citations

Cites

1 retracted

Show all 72 references

Research works printed but not linked: 4

  1. Kim, S. W. & Lee, Y. The artificial intelligence literacy scale for middle school. Stud. J. Korea Soc. Comput. Inf. 27, 225–238 (2022).
  2. Çelebi, C. et al. Artificial intelligence literacy: an adaptation study. Instr. Technol. Lifelong Learn 4, 291–306 (2023).
  3. Schünemann, H., Brożek, J., Guyatt, G. & Oxman, A. (Eds.) GRADE Handbook. https://gdt.gradepro.org/app/handbook/handbook.html (2013).
  4. Ali, S., Payne, B. H., Williams, R., Park, H. W. & Breazeal, C. Constructionism, ethics, and creativity: Developing primary and middle school artificial intelligence education. International Workshop on Education in Artificial Intelligence K-12 (EDUAI’19) 1–4 (2019).

Cited by

See all 292

Metadata

Authors
Lintner, T.
Affiliations
Masaryk University (CZ)
Corresponding author
Lintner, Tomáš
Venue
npj Science of Learning
Published
2024-08-06
Volume
9
Issue
1
Pages
50-50
ISSN
2056-7936 (ISSN-L)
OpenAlex
W4401371637
Cited by
292
References
61
Citations by year
2026: 209 · 2025: 72 · 2024: 11
FWCI
16.77
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Science of Learning (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Veřejné služby Informačního systému (Masarykiana Brunensis Universitas) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Funders
  • European Commission (LX22NPO5101)
  • NextGenerationEU
Topics
  • Social Sciences › Education › Digital literacy in education
  • Social Sciences › Education › Artificial Intelligence in Education
  • Health Sciences › Nursing › Research and Theory › Health Education and Validation
Keywords
AI literacy, scale validation, psychometric properties, COSMIN, measurement invariance, content validity, construct validity, reliability statistics, self-report scale, performance-based assessment, university students, teachers, high school students

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A systematic review of AI literacy scales. https://app.cochat.ai/papers/10.1038/s41539-024-00264-4
The paper
Lintner, T. (2024). A systematic review of AI literacy scales. npj Science of Learning. https://doi.org/10.1038/s41539-024-00264-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.