Development and Validation of a Digital Literacy Scale in the Artificial Intelligence Era for College Students
Open accessAbstract
This study developed digital literacy instruments and tested their effectiveness on college students' perceptions of AI technologies.In creating a new digital literacy test tool, we reviewed the concept and scale of digital literacy based on previous studies that identified the characteristics and measurement of AI literacy.We developed 23 preliminary questions for our research instrument and used a quantitative approach to survey 318 undergraduates.After conducting exploratory and confirmatory factor analysis, we found that digital literacy in the age of AI had four ability sub-factors: critical understanding, artificial intelligence social impact recognition, artificial intelligence technology utilization, and ethical behavior.Then we tested the sub-factors' predictive powers on the perception of AI's usefulness and ease of use.The regression result shows that the most common powerful predictor of the usefulness and ease of use of AI technology was the ability to use AI technology.This finding implies that for college students, the ability to use various tools based on AI technology is an essential competency in the AI era.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The critical understanding dimension covered by a relevant study refers to users' skills for critically evaluating content generated by AI.
“Likewise, the critical understanding dimension covered by a relevant study (Hwang et al., 2023) refers to the users' skills for critically evaluating the content generated by AI.”
From Development and validation of the Generative AI Literacy for Learning Scale (GenAI-LLs) (2025) · cites this paper as Hwang et al., 2023
The Measures and Conclusion and Discussion sections define the paper’s critical comprehension dimension as evaluating the truthfulness, objectivity, and informativeness of content presented by an AI recommendation service. They do not specify content generated by AI; that is a narrower, different description of the content being evaluated. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPrior studies partially cover prompt and language skills within usage dimensions.
“Although the prompt and language skills are partially covered by the usage dimensions in the relevant studies (Hwang et al., 2023;B. Wang et al., 2023)”
From Development and validation of the Generative AI Literacy for Learning Scale (GenAI-LLs) (2025) · cites this paper as Hwang et al., 2023
Section 3.3 (Measures; page unavailable) includes an AI-technology-use dimension covering the ability to search for and obtain desired information, which could overlap with part of the claimed skills. It does not identify prompt-writing or language skills as competencies covered by that dimension. (LLM, conf=0.76)
Experimental — accuracy not yet measuredGSE-6AI, Hwang et al.'s instrument, and Chan & Zhou's EVT-based instrument (knowledge of general AI subscale) are promising instruments.
“GSE-6AI30, Hwang et al.’s instrument39, and Chan & Zhou’s EVT based instrument (knwl. of gen. AI subscale)37 are also promising instruments with good evidence for structural validity and internal consistency, however, their content validity needs to be checked on the higher-education students.”
From A systematic review of AI literacy scales (2024) · cites this paper as [39]
GSE-6AI, Hwang et al.'s instrument, and Chan & Zhou's EVT-based instrument (knowledge of general AI subscale) showed good evidence for internal consistency.
“GSE-6AI30, Hwang et al.’s instrument39, and Chan & Zhou’s EVT based instrument (knwl. of gen. AI subscale)37 are also promising instruments with good evidence for structural validity and internal consistency, however, their content validity needs to be checked on the higher-education students.”
From A systematic review of AI literacy scales (2024) · cites this paper as [39]
GSE-6AI, Hwang et al.'s instrument, and Chan & Zhou's EVT-based instrument (knowledge of general AI subscale) showed good evidence for structural validity.
“GSE-6AI30, Hwang et al.’s instrument39, and Chan & Zhou’s EVT based instrument (knwl. of gen. AI subscale)37 are also promising instruments with good evidence for structural validity and internal consistency, however, their content validity needs to be checked on the higher-education students.”
From A systematic review of AI literacy scales (2024) · cites this paper as [39]
Sections 4.1 and 4.2 report factor analyses and construct validity for this paper’s own AI digital-literacy scale. The passages do not assess the structural validity of GSE-6AI, Hwang et al.’s instrument, or Chan and Zhou’s EVT-based knowledge-of-general-AI subscale; searches for those instruments found no such assessment in our copy. (LLM, conf=0.73)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483 · see citing works
Principles and Practice of Structural Equation Modeling
Tabri, N., & Elliott, C. M. (2012) · Canadian Graduate Journal of Sociology and Criminology · doi:10.15353/cgjsc.v1i1.3787 · Cited by 39,065 · see citing works
(1985) · Cited by 6,185 · see citing works
Willmott, C. J. (1981) · Physical Geography · doi:10.1080/02723646.1981.10642213 · Cited by 4,916 · see citing works
Digital Natives, Digital Immigrants Part 2: Do They Really Think Differently?
Prensky, M. (2001) · On the Horizon · doi:10.1108/10748120110424843 · Cited by 4,035 · see citing works
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 3× · see citing works
“…competencies of AI literacy, recently, Long and Magerko (2020) defined AI literacy as “a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace” (p. 598) [25].”
BackgroundFrom digital literacy to digital competence: the teacher digital competency (TDC) framework
Falloon, G. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09767-4 · Cited by 1,372 · cited 1× · see citing works
“However, the definition evolved by adding new technologies, and the knowledge of academic and skills demands of the workforce (or education) changed [3].”
BackgroundDigital Literacy: A Conceptual Framework for Survival Skills in the Digital era
Eshet‐Alkalai, Y. (2004) · Journal of educational multimedia and hypermedia · Cited by 1,333 · see citing works
Information and digital literacies: A review of concepts
Bawden, D. (2001) · Journal of Documentation · doi:10.1108/eum0000000007083 · Cited by 1,207 · see citing works
Saadé, R. G., & Bahli, B. (2004) · Information & Management · doi:10.1016/j.im.2003.12.013 · Cited by 1,055 · see citing works
(2012) · Cited by 852 · see citing works
A systematic review on digital literacy
Tınmaz, H., Lee, Y.-T., Fanea‐Ivanovici, M., & Baber, H. (2022) · Smart Learning Environments · doi:10.1186/s40561-022-00204-y · Cited by 592 · see citing works
Understanding the effect of e-learning on individual performance: The role of digital literacy
Mohammadyari, S., & Singh, H. (2015) · Computers & Education · doi:10.1016/j.compedu.2014.10.025 · Cited by 496 · see citing works
Blau, I., Shamir‐Inbal, T., & Avdiel, O. (2019) · The Internet and Higher Education · doi:10.1016/j.iheduc.2019.100722 · Cited by 390 · see citing works
Rencher, A. C., & Christensen, W. F. (2012) · Wiley series in probability and statistics · doi:10.1002/9781118391686.ch13 · Cited by 297 · see citing works
Workplace literacy skills—how information and digital literacy affect adoption of digital technology
Nikou, S., De Reuver, M., & Mahboob Kanafi, M. (2022) · Journal of Documentation · doi:10.1108/jd-12-2021-0241 · Cited by 257 · see citing works
Digital Readers: The Next Chapter in E‐Book Reading and Response
Larson, L. C. (2010) · The Reading Teacher · doi:10.1598/rt.64.1.2 · Cited by 213 · see citing works
Martínez-Alcalá, C. I., Rosales-Lagarde, A., Alonso-Lavernia, M. D. L. Á., Ramírez-Salvador, J. Á., Jiménez-Rodríguez, B., Cepeda-Rebollar, R. M., López‐Noguerola, J. S., Díaz, M. L. B., & Agis-Juárez, R. A. (2018) · Frontiers in ICT · doi:10.3389/fict.2018.00021 · Cited by 134 · see citing works
Lean Transformation Integrated with Industry 4.0 Implementation Methodology
Satoğlu, Ş. I., Üstündağ, A., Çevikcan, E., & Durmuşoğlu, M. B. (2017) · Lecture notes in management and industrial engineering · doi:10.1007/978-3-319-71225-3_9 · Cited by 57 · see citing works
Multiliteracy Centers Spanning the Digital Divide: Providing a Full Spectrum of Support
Bancroft, J. (2016) · Computers & composition/Computers and composition · doi:10.1016/j.compcom.2016.04.002 · Cited by 35 · see citing works
León, L. D., Corbeil, R. R., & Corbeil, M. E. (2021) · Journal of Research on Technology in Education · doi:10.1080/15391523.2021.1974988 · Cited by 33 · see citing works
Ahn, J. I., & Seo, Y. K. (2014) · Journal of Digital Convergence · doi:10.14400/jdc.2014.12.2.69 · Cited by 21 · see citing works
Establishing a Definition of AI Literacy and Designing a Liberal Arts Education Program
Yi, Y., & Park, Y. (2021) · 어문론집 · doi:10.15565/jll.2021.03.85.451 · Cited by 16 · see citing works
Passing the Baton: Digital Literacy and Sustained Implementation of eLearning Technologies
Herckis, L. (2018) · Cited by 9 · see citing works
Direction of Software Education in Practical Arts for Cultivating Competencies in the AI Era
경인교육대학교, & Lee, C.-H. (2020) · Journal of Korean Practical Arts Education · doi:10.29113/skpaer.2020.26.2.003 · Cited by 9 · see citing works
A Study on Development and Validation of Digital Literacy Measurement Tool
Chung, M.-H., Kim, J., & Hwang, H.-S. (2021) · Journal of Internet Computing and services · doi:10.7472/jksii.2021.22.4.51 · Cited by 5 · see citing works
Self-citationLiu, C. Z., Lum, W. Z., & Hwang, H. (2022) · Journal of Social Science · doi:10.22418/jss.2022.12.61.3.121 · Cited by 4 · see citing works
Self-citationChoi, M., 서필교, Il, C. M., et al. (2014) · Korean Journal of Journalism & Communication Studies · Cited by 3 · see citing works
Oh, J., & Hahn, J. (2007) · Journal of Educational Technology · doi:10.17232/kset.23.1.25 · Cited by 1 · see citing works
Lim, J., & Kim, H. (2021) · Advertising Research · doi:10.16914/ar.2021.131.110 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 12
- R. Reynolds, "Reconstructing "digital literacy" in a Constructionist computer club: The role of motivation, interest, and inquiry in children's purposive technology use," Ph.D. dissertation, Dept. Mass Com., Syracuse Univ., New York, USA, 2008.
- M. Prensky, "Digital wisdom and homo sapiens digital," in Deconstructing digital natives, 2011, pp. 15-29. [Online]. Available: https://www.taylorfrancis.com/chapters/edit/10.4324/9780203818848-3/digital-wisdom-homo- sapiens-digital-marc-prensky · DOI
- S. Y. Choi, "A Study on Digital Competence in the Era of the 4th Industrial Revolution," Journal of Computer Education Society, vol. 21, no. 5, pp. 25-35, 2018. Article (CrossRef Link).
- H. Z. Hou, Y. G. Wang, "The Discussion on the Framework Construction and Training Mechanism of Primary and Middle School Students' Intelligent Literacy In the Era of Artificial Intelligence," Digital Education, vol. 6, no. 6, pp. 50-55, 2020. Article (CrossRef Link).
- M. Wang, "Construction and Cultivation of Students' Intelligent Literacy Based on Core Literacy," Contemporary Educational Science, no.2, pp. 83-85, 2018. Article (CrossRef Link).
- C. G. Jang and W. J. Sung, "A Study on the Intent of Adopting Artificial Intelligence-Based Public Service Policy: Focusing on the Effects of Individual Perception and Digital Literacy Level," Informatization Policy, vol. 29, no. 1, pp. 60-83, 2022. Article (CrossRef Link). · DOI
- D. Bawden, "Information and digital literacies: a review of concepts," Journal of Documentation, Vol. 57 No. 2, pp. 218-259, 2001. Article (CrossRef Link). · DOI · Paper page
- W. J. Lee, S. H. Kim, E. H. Lee, "Developing a Digital Literacy Curriculum Framework," Journal of educational research, vol. 40, no. 3, pp. 201-221, 2019. Article (CrossRef Link).
- H. S. Jung, K. Heo, "Convergence of Media Literacy and Computational Thinking for Critical Thinking," Journal of korean Culture (JKC), vol. 48, pp. 105-128, 2020. Article (CrossRef Link).
- L. C. Zhu, C. Wang, H. S. Hwang, "A Study on Factors Affecting University Students' Satisfaction with YouTube AI Recommendation System," Journal of Internet Computing and Services, vol. 23, no. 3, pp. 77-85, 2022. Article (CrossRef Link). · DOI
- J. Y. Kim, et al, "Basic Research for the Development of Cognitive Domain Assessment Tools for Digital Literacy," Cheongram Language and Literature Education, vol. 62, pp. 7-39, 2017. Article (CrossRef Link).
- Y. N. Choi, "A Study on the Effect of Digital Technology Acceptance on the Life Satisfaction of Urban Residents in the Era of Covid 19: Focusing on the Moderating Effect of Digital Literacy," Journal of local government studies, vol. 33, no. 3, pp. 187-219, 2021. Article (CrossRef Link).
Cited by
Sorted by citations
A systematic review of AI literacy scales
Lintner, T. (2024) · npj Science of Learning · doi:10.1038/s41539-024-00264-4 · Cited by 292 · see citing works
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
Students' perceptions of artificial intelligence integration in higher education
Slimi, Z., Benayoune, A., & Alemu, A. E. (2025) · European Journal of Educational Research · doi:10.12973/eu-jer.14.2.471 · Cited by 38 · see citing works
Soto-Sanfiel, M. T., Angulo-Brunet, A., & Lutz, C. (2024) · doi:10.31235/osf.io/bvyku · Cited by 20 · see citing works
Development and validation of the Generative AI Literacy for Learning Scale (GenAI-LLs)
Gümüş, M. M., & KARA, M. (2025) · Australasian Journal of Educational Technology · doi:10.14742/ajet.10236 · Cited by 10 · see citing works
- Authors
- Hwang, H. S., Zhu, L. C., & Cui, Q.
- Affiliations
- Dongguk University (KR)
- Corresponding author
- Ha Sung Hwang
- Venue
- KSII Transactions on Internet and Information Systems
- Published
- 2023-08-31
- Volume
- 17
- Issue
- 8
- ISSN
- 1976-7277 (ISSN-L)
- OpenAlex
- W4386376613
- Cited by
- 60
- References
- 31
- Citations by year
- 2026: 22 · 2025: 31 · 2024: 7
- FWCI
- 14.08
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- KSII Transactions on Internet and Information Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Digital literacy in education
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- digital literacy, AI literacy, scale development, university students, perceived usefulness, perceived ease of use
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
