The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Measuring user competence in using artificial intelligence: validity and reliability of artificial intelligence literacy scale
Abstract
As artificial intelligence (AI) became a part of daily life, it has become important to determine user competence in using AI technology. Here, we propose the concept of AI literacy and develop a quantitative scale for obtaining accurate data regarding the AI literacy of ordinary users. We first identified the primary core constructs of AI literacy, including awareness, use, evaluation, and ethics. Next, we generated 65 items to capture these four constructs; only 31 items were retained after a three-step content validation process. Then, we conducted a survey, and collected two samples of data. By reducing the number of items using the first sample and performing reliability and validity tests on the second sample, we obtained a 12-item instrument for the quantitative measurement of AI literacy. The results confirmed that the proposed four-construct model is an adequate representation of AI literacy. Further, AI literacy is significantly related to digital literacy, attitude towards robots, and users’ daily usage of AI. This study will not only aid researchers in understanding how user competence in using AI technology affects human–AI interactions but will also help designers develop AI applications that are aligned with the AI literacy levels of the target users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This attitude scale was used because it is one of the shortest and, thus, one of the most resource-efficient instruments designed to capture attitudes; it had already been validated in the native language of the course participants.”
From Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment (2023) · cites this paper as [22]
In the citing paper
“This could be an indication of discriminant validity because, in theory, AI literacy and attitudes toward AI are assumed to be two different constructs, representing cognitive/skill and affective aspects, respectively [9,16,20-22].”
From Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment (2023) · cites this paper as [22]
In the citing paper
“The reliability coefficients of the sub-dimensions range between 0.72 and 0.76, and the overall reliability (Cronbach’s alpha) of the scale is 0.84, indicating good internal consistency.”
From Integrating artificial intelligence into preservice teacher education: enhancing AI literacy and digital material design competency (2026) · cites this paper as Wang et al. (2022)
In the citing paper
“The Artificial Intelligence Literacy Scale (AILS; Wang et al., 2023) measures AI understanding and competence across four subscales: awareness, usage, evaluation and ethics, using 12 items on a 7- point Likert scale.”
From The role of critical thinking on undergraduates' reliance behaviours on generative AI in problem‐solving (2025) · cites this paper as Wang et al., 2023
In the citing paper
“The validity of the self-report measurement instruments for AI literacy [11,12,14,15] has only been tested regarding their factorial structure.”
From Meta AI literacy scale: Further validation and development of a short version (2024) · cites this paper as [15]
In the citing paper
“It appears that individuals with higher positive attitudes, lower negative attitudes, and more willingness to use technology are also more likely to consider themselves competent [1, 46, 48].”
In the citing paper
“According to Wang et al. [47], the degree of AI literacy can determine individuals' readiness to embrace AI technologies.”
From Modeling teachers’ acceptance of generative artificial intelligence use in higher education: the role of AI literacy, intelligent TPACK, and perceived trust (2024) · cites this paper as [47]
In the citing paper
“AI literacy improves understanding of GenAI's advantages and functions, which encourages a favorable view of its acceptance and use in educational contexts [3,47,48,64].”
From Modeling teachers’ acceptance of generative artificial intelligence use in higher education: the role of AI literacy, intelligent TPACK, and perceived trust (2024) · cites this paper as [47]
In the citing paper
“Wang et al. [58] affirmed a notable correlation between AI literacy and attitudes toward robots’ and users’ regular use of AI.”
From ChatGPT in Learning: Assessing Students’ Use Intentions through the Lens of Perceived Value and the Influence of AI Literacy (2024) · cites this paper as [58]
In the citing paper
“Higher AI literacy enables users to better understand the capabilities and limitations of AI technologies, leading to more effective adoption and usage [22,58].”
From ChatGPT in Learning: Assessing Students’ Use Intentions through the Lens of Perceived Value and the Influence of AI Literacy (2024) · cites this paper as [58]
In the citing paper
“Wang et al. [58] indicated that the level of AI literacy determines whether students embrace AI technologies.”
From ChatGPT in Learning: Assessing Students’ Use Intentions through the Lens of Perceived Value and the Influence of AI Literacy (2024) · cites this paper as [58]
In the citing paper
“Their ability to leverage the tool efficiently contributes to their perception of the AI tool as highly useful [22,58,62].”
From ChatGPT in Learning: Assessing Students’ Use Intentions through the Lens of Perceived Value and the Influence of AI Literacy (2024) · cites this paper as [58]
In the citing paper
“The support for H10, which states that AI literacy positively influences the perceived value of ChatGPT, underscores the significance of students’ understanding of and familiarity with AI in appreciating the tool’s potential and contributions to their academic and personal endeavours.”
From ChatGPT in Learning: Assessing Students’ Use Intentions through the Lens of Perceived Value and the Influence of AI Literacy (2024) · cites this paper as [58]
Citations
Cites
Evaluating Structural Equation Models with Unobservable Variables and Measurement Error
Fornell, C., & Larcker, D. F. (1981) · Journal of Marketing Research · doi:10.2307/3151312 · Cited by 109,314
Evaluating structural equation models with unobservable variables and measurement error
Fornell, C., & Larcker, D. F. (1981) · Journal of Marketing Research · doi:10.1177/002224378101800104 · Cited by 109,190
(1994) · Journal of the American Dietetic Association · doi:10.1016/0002-8223(94)92216-0 · Cited by 46,726
Principles and Practice of Structural Equation Modeling
Tabri, N., & Elliott, C. M. (2012) · Canadian Graduate Journal of Sociology and Criminology · doi:10.15353/cgjsc.v1i1.3787 · Cited by 39,094
A new criterion for assessing discriminant validity in variance-based structural equation modeling
Henseler, J., Ringle, C. M., & Sarstedt, M. (2015) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-014-0403-8 · Cited by 38,459
null (2016) · Philosophy Study · doi:10.17265/2159-5313/2016.09.003 · Cited by 28,317
Einax, J. W., Zwanziger, H. W., & Geiß, S. (1997) · Wiley eBooks · doi:10.1002/352760216x.ch5 · Cited by 18,779
Scale Development: Theory and Applications.
Xie, Y.-P., & Devellis, R. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075704 · Cited by 14,143
Costello, A. B., & Osborne, J. (2005) · University of Massachusetts Amherst · doi:10.7275/jyj1-4868 · Cited by 10,297
Constructing validity: Basic issues in objective scale development.
Clark, L. A., & Watson, D. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.309 · Cited by 6,567
A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires
Hinkin, T. R. (1998) · Organizational Research Methods · doi:10.1177/109442819800100106 · Cited by 4,095
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“Although Long and Magerko (2020) has summarised a set of core competencies in using AI technology, there neither exists a mature framework nor a practical tool for measuring it.”
BackgroundArtificial intelligence and the future of work: Human-AI symbiosis in organizational decision making
Jarrahi, M. H. (2018) · Business Horizons · doi:10.1016/j.bushor.2018.03.007 · Cited by 2,772
Can we teach digital natives digital literacy?
Ng, W. (2012) · Computers & Education · doi:10.1016/j.compedu.2012.04.016 · Cited by 2,442
Making Sense of Factor Analysis
Pett, M., Lackey, N., & Sullivan, J. (2003) · doi:10.4135/9781412984898 · Cited by 2,168
(2013) · Cited by 1,951
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Porter, C. E., & Donthu, N. (2006) · Journal of Business Research · doi:10.1016/j.jbusres.2006.06.003 · Cited by 1,066
Anderson, J. C., & Gerbing, D. W. (1991) · Journal of Applied Psychology · doi:10.1037/0021-9010.76.5.732 · Cited by 1,038
Journal of Computing Sciences in Colleges
(2019) · Journal of computing sciences in colleges · doi:10.36614/jcsc · Cited by 1,035
The Optimal Number of Response Alternatives for a Scale: A Review
Cox, E. P. (1980) · Journal of Marketing Research · doi:10.2307/3150495 · Cited by 867
(2012) · Cited by 852
DigEuLit: Concepts and Tools for Digital Literacy Development
Martin, A. R., & Grudziecki, J. (2006) · Innovation in Teaching and Learning in Information and Computer Sciences · doi:10.11120/ital.2006.05040249 · Cited by 684
Attitude, digital literacy and self efficacy: Flow-on effects for online learning behavior
Prior, D. D., Mazanov, J., Meacheam, D., Heaslip, G., & Hanson, J. M. (2016) · The Internet and Higher Education · doi:10.1016/j.iheduc.2016.01.001 · Cited by 605
Schriesheim, C. A., Powers, K. J., Scandura, T. A., Gardiner, C. C., & Lankau, M. J. (1993) · Journal of Management · doi:10.1177/014920639301900208 · Cited by 558
Tao, J., & Tan, T. (2005) · Lecture notes in computer science · doi:10.1007/11573548_125 · Cited by 531
Nomura, T., Kanda, T., & Suzuki, T. (2005) · AI & Society · doi:10.1007/s00146-005-0012-7 · Cited by 530
Mapping Digital Competence: Towards a Conceptual Understanding.
Ala-Mutka, K. (2011) · European Commission, JRC, Institute for Prospective Technological Studies · doi:10.13140/rg.2.2.18046.00322 · Cited by 508
Survey Measures of Web-Oriented Digital Literacy
Hargittai, E. (2005) · Social Science Computer Review · doi:10.1177/0894439305275911 · Cited by 489
Journal of Media Literacy Education
(2017) · Journal of Media Literacy Education · doi:10.23860/jmle · Cited by 481
Gunkel, D. J. (2007) · Philosophy/communication · doi:10.5840/philcomm200745 · Cited by 471
Digital Literacy and Privacy Behavior Online
Park, Y. J. (2011) · Communication Research · doi:10.1177/0093650211418338 · Cited by 437
Lee, S., & Choi, J. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.02.005 · Cited by 388
International Journal of Business, Humanities and Technology
(2021) · International Journal of Business Humanities and Technology · doi:10.30845/ijbht · Cited by 347
The role of expressiveness and attention in human-robot interaction
Bruce, A., Nourbakhsh, I., & Simmons, R. (2002) · Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292) · doi:10.1109/robot.2002.1014396 · Cited by 336
Inclusive AI literacy for kids around the world
Druga, S., Vu, S. T., Likhith, E., & Qiu, T. (2019) · Proceedings of FabLearn 2019 · doi:10.1145/3311890.3311904 · Cited by 318
Control your Facebook: An analysis of online privacy literacy
Bartsch, M., & Dienlin, T. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.11.022 · Cited by 236
Coming of (Old) Age in the Digital Age: ICT Usage and Non-Usage among Older Adults
Neves, B. B., Amaro, F., & Fonseca, J. R. S. (2013) · Sociological Research Online · doi:10.5153/sro.2998 · Cited by 216
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
Testing Information Literacy in Digital Environments: ETS's iSkills Assessment
Katz, I. R. (2007) · Information Technology and Libraries · doi:10.6017/ital.v26i3.3271 · Cited by 204
Student Attitudes and Behaviors Towards Digital Textbooks
Weisberg, M. (2011) · Publishing Research Quarterly · doi:10.1007/s12109-011-9217-4 · Cited by 182
Understanding and measuring user competence
Munro, M. C., Huff, S. L., Marcolin, B. L., & Compeau, D. (1997) · Information & Management · doi:10.1016/s0378-7206(97)00035-9 · Cited by 170
A comparison of World-Wide Web and paper-and-pencil personality questionnaires
Pettit, F. A. (2002) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195423 · Cited by 158
Rethinking ICT literacy: From computer skills to social network settings
Wilson, M. R., Scalise, K., & Gochyyev, P. (2015) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2015.05.001 · Cited by 125
Information literacy: Essential skills for the information age
Agosto, D. E. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20155 · Cited by 123
Identifying the Random Responder
Beach, D. A. (1989) · The Journal of Psychology · doi:10.1080/00223980.1989.10542966 · Cited by 104
A study on the effect of digital literacy on information use behavior
Noh, Y. (2016) · Journal of Librarianship and Information Science · doi:10.1177/0961000615624527 · Cited by 102
Understanding Is Key: An Analysis of Factors Pertaining to Trust in a Real-World Automation System
Balfe, N., Sharples, S., & Wilson, J. R. (2018) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720818761256 · Cited by 74
Digital Literacy: A Vital Competence for 2010?
Leahy, D., & Dolan, D. (2010) · Key Competencies in the Knowledge Society · doi:10.1007/978-3-642-15378-5_21 · Cited by 60
Attitudes and the Digital Divide: Attitude Measurement as Instrument to Predict Internet Usage
Donat, E., Brandtweiner, R., & Kerschbaum, J. (2009) · Informing Science The International Journal of an Emerging Transdiscipline · doi:10.28945/427 · Cited by 60
Robert S. Engelmore Award Article: Building AI Applications: Yesterday, Today, and Tomorrow
Smith, R. G., & Eckroth, J. (2017) · AI Magazine · doi:10.1609/aimag.v38i1.2709 · Cited by 54
Literacy and computer literacy: Definitions and comparisons
McMillan, S. (1996) · Computers & Education · doi:10.1016/s0360-1315(96)00026-7 · Cited by 50
Technology literacy: the extension of cognition
Moore, D. R. (2010) · International Journal of Technology and Design Education · doi:10.1007/s10798-010-9113-9 · Cited by 41
Machines versus Humans: The Impact of AI Chatbot Disclosure on Customer Purchases
Luo, X., Tong, S., Fang, Z., & Qu, Z. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3435635 · Cited by 41
Livingstone, S., & van der Graaf, S. (2010) · The International Encyclopedia of Communication · doi:10.1002/9781405186407.wbiecm039 · Cited by 35
Su, G. (2018) · AI Matters · doi:10.1145/3175502.3175511 · Cited by 34
New Perspectives on the System Usage Construct
Burton‐Jones, A. (2022) · Digital Archive @ GSU · doi:10.57709/1063232 · Cited by 29
Information Technology Literacy: Task Knowledge and Mental Models
Brandt, D. S. (2001) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 28
A business model template for AI solutions
Metelskaia, I., Ignatyeva, O., Denef, S., & Samsonowa, T. (2018) · International Conference on Intelligent Science and Technology (ICIST) · doi:10.1145/3233740.3233750 · Cited by 26
Vossen, S., Ham, J. R. C., & Midden, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13226-1_7 · Cited by 24
How Does Explanation-Based Knowledge Influence Driver Take-Over in Conditional Driving Automation?
Zhou, H., Itoh, M., & Kitazaki, S. (2021) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3051342 · Cited by 24
Assessing Digital Competence in Secondary Education - Issues, Models and Instruments
Calvani, A., RANIERI, M., & FINI, A. (2009) · Florence Research (University of Florence) · Cited by 20
Internet Privacy, Social Awareness, And Internet Technical Literacy - An Exploratory Investigation
Dinev, T., & Hart, P. (2004) · Journal of the Association for Information Systems · Cited by 19
Comunello, F., Mulargia, S., Belotti, F., & Fernández-Ardèvol, M. (2015) · Interacción · doi:10.1007/978-3-319-20892-3_43 · Cited by 10
Development and application of new social power measures in superior-subordinate relationships
Hinkin, T. R. (1985) · University Microfilms International eBooks · Cited by 9
Tobin, C. D. (1983) · The Arithmetic Teacher · doi:10.5951/at.30.6.0022 · Cited by 8
Improving Student Mental Models in a New University Information Setting
Rosling, A., & Littlemore, K. (2011) · SensePublishers eBooks · doi:10.1007/978-94-6091-299-3_5 · Cited by 7
Accounting for the human when designing with AI : challenges identified
Stembert, N., & Harbers, M. (2019) · Data Archiving and Networked Services (DANS) · doi:10.1145/3290607 · Cited by 6
The Reform and Practice of Educational Technology Major in the Age of Artificial Intelligence 2.0
Xiao, W., & Bie, M. (2019) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/677/5/052094 · Cited by 4
Research works printed but not linked: 15
- Buckingham D., 1989, Radical Philosophy, 51, 12
- Buckingham D., 2007, Journal of Educational Multimedia and Hypermedia, 16, 323
- Calvani A., 2008, Journal of E-Learning and Knowledge Society, 4, 183
- 10.1177/002224378001700401 · DOI
- Eshet Y., 2004, Journal of Educational Multimedia and Hypermedia, 13, 93
- Finstad K., 2010, Journal of Usability Studies, 5, 104
- Gapski, H. 2007. “Some Reflections on Digital Literacy.” Paper Presented at the Proceedings of the 3rd International Workshop on Digital Literacy.
- Gilster P., 1997, Digital Literacy
- 10.7551/mitpress/8975.001.0001 · DOI · Paper page
- 10.17265/2159-5313/2016.09.003 · DOI · Paper page
- Kim Y., 2013, Journal of Digital Convergence, 11, 9
- Likert R., 1932, Archives of Psychology
- Mayerl J., 2016, Green European, 210
- Tarafdar M., 2019, MIT Sloan Management Review, 60, 37
- Wilson, H. J., and P. R. Daugherty. 2018. “Collaborative Intelligence: Humans and AI Are Joining Forces.” Harvard Business Review .
Cited by
Sorted by citations
Carolus, A., Koch, M. J., Straka, S., Latoschik, M. E., & Wienrich, C. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100014 · Cited by 488
Ng, D. T. K., Wu, W., Leung, J. K. L., Chiu, T. K. F., & Chu, S. K. W. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13411 · Cited by 390
Wang, C., Wang, H., Li, Y., Dai, J., Gu, X., & Yu, T. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2383033 · Cited by 468
Yilmaz, R., & Karaoglan Yilmaz, F. G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100005 · Cited by 344
Almatrafi, O., Johri, A., & Lee, H. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100173 · Cited by 341
Metadata
- Authors
- Wang, B., Rau, P.-L. P., & Yuan, T.
- Affiliations
- Tsinghua University (CN)
- Corresponding author
- Rau, Pei-Luen Patrick
- Venue
- Behaviour & Information Technology
- Publisher
- Informa UK Limited
- Published
- 2022-05-10
- Volume
- 42
- Issue
- 9
- Pages
- 1324-1337
- ISSN
- 0144-929X (ISSN-L)
- 1362-3001
- OpenAlex
- W4280549083
- Cited by
- 1,036
- References
- 69
- Citations by year
- 2026: 549 · 2025: 323 · 2024: 137 · 2023: 27
- FWCI
- 120.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Behaviour and Information Technology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Ministry of Science and Technology of China (2018AAA0101700)
- Topics
- Social Sciences › Education › Digital literacy in education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Gender Studies › Gender and Technology in Education
- Keywords
- AI literacy, AI literacy scale, user competence, AI awareness, AI ethics, AI evaluation, digital literacy, artificial intelligence use, human-AI interaction, scale development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
