CoChat
Sign Up
Cited by 1,036Digital literacy in educationEthics and Social Impacts of AIGender and Technology in Education

Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale

Wang, Bingcheng, Rau, Pei-Luen Patrick, Yuan, Tianyi·2023·Behaviour & Information Technology
DOI 10.1080/0144929x.2022.2072768
Open this paper in CoChat — free account

Abstract

As artificial intelligence (AI) became a part of daily life, it has become important to determine user competence in using AI technology. Here, we propose the concept of AI literacy and develop a quantitative scale for obtaining accurate data regarding the AI literacy of ordinary users. We first identified the primary core constructs of AI literacy, including awareness, use, evaluation, and ethics. Next, we generated 65 items to capture these four constructs; only 31 items were retained after a three-step content validation process. Then, we conducted a survey, and collected two samples of data. By reducing the number of items using the first sample and performing reliability and validity tests on the second sample, we obtained a 12-item instrument for the quantitative measurement of AI literacy. The results confirmed that the proposed four-construct model is an adequate representation of AI literacy. Further, AI literacy is significantly related to digital literacy, attitude towards robots, and users’ daily usage of AI. This study will not only aid researchers in understanding how user competence in using AI technology affects human–AI interactions but will also help designers develop AI applications that are aligned with the AI literacy levels of the target users.

Integrity signals

Registry

Crossref · 10.1080/0144929x.2022.2072768

Cites

69 of 69 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The attitude scale by Sindermann et al. [22] is one of the shortest and most resource-efficient instruments designed to capture attitudes and had already been validated in the native language of the course participants.

    “This attitude scale was used because it is one of the shortest and, thus, one of the most resource-efficient instruments designed to capture attitudes; it had already been validated in the native language of the course participants.”

    From Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment (2023) · cites this paper as [22]

  • The weak correlation between learning gain scores and attitudes toward AI could be an indication of discriminant validity because AI literacy and attitudes toward AI are assumed to be two different constructs.

    “This could be an indication of discriminant validity because, in theory, AI literacy and attitudes toward AI are assumed to be two different constructs, representing cognitive/skill and affective aspects, respectively [9,16,20-22].”

    From Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment (2023) · cites this paper as [22]

  • The AI Literacy Scale has reliability coefficients of sub-dimensions between 0.72 and 0.76 and overall Cronbach's alpha of 0.84, indicating good internal consistency.

    “The reliability coefficients of the sub-dimensions range between 0.72 and 0.76, and the overall reliability (Cronbach’s alpha) of the scale is 0.84, indicating good internal consistency.”

    From Integrating artificial intelligence into preservice teacher education: enhancing AI literacy and digital material design competency (2026) · cites this paper as Wang et al. (2022)

  • The Artificial Intelligence Literacy Scale (Wang et al., 2023) measures AI understanding and competence across four subscales (awareness, usage, evaluation, ethics) with 12 items and demonstrates good reliability (Cronbach's alpha = 0.83).

    “The Artificial Intelligence Literacy Scale (AILS; Wang et al., 2023) measures AI understanding and competence across four subscales: awareness, usage, evaluation and ethics, using 12 items on a 7-­ point Likert scale.”

    From The role of critical thinking on undergraduates' reliance behaviours on generative <scp>AI</scp> in problem‐solving (2025) · cites this paper as Wang et al., 2023

  • The validity of self-report measurement instruments for AI literacy [11,12,14,15] has only been tested regarding their factorial structure.

    “The validity of the self-report measurement instruments for AI literacy [11,12,14,15] has only been tested regarding their factorial structure.”

    From Meta AI literacy scale: Further validation and development of a short version (2024) · cites this paper as [15]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Wang, B., Rau, P.-L. P., & Yuan, T.
Affiliations
Tsinghua University (CN)
Corresponding author
Rau, Pei-Luen Patrick
ORCID
Venue
Behaviour & Information Technology
Publisher
Informa UK Limited
Published
2022-05-10
Volume
42
Issue
9
Pages
1324-1337
ISSN
  • 0144-929X (ISSN-L)
  • 1362-3001
DOI
10.1080/0144929x.2022.2072768
OpenAlex
W4280549083
Cited by
1,036
References
69
Citations by year
2026: 549 · 2025: 323 · 2024: 137 · 2023: 27
FWCI
120.30
Type
journal article
Language
en
Open access
closed
Locations
  • Behaviour and Information Technology (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
Ministry of Science and Technology of China (2018AAA0101700)
Topics
  • Physical Sciences › Computer Science › Information Systems › Digital literacy in education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Gender Studies › Gender and Technology in Education
Keywords
AI literacy, AI literacy scale, user competence, AI awareness, AI ethics, AI evaluation, digital literacy, artificial intelligence use, human-AI interaction, scale development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review