AI Literacy in Higher Education: A Systematic Approach to Questionnaire Development and Validation
Open accessAbstract
This paper presents the development, refinement, and validation of the Critical Artificial Intelligence Literacy Scale, an instrument designed to measure artificial intelligence literacy across four dimensions: knowledge-related, operational, critical, and ethical. The initial version of the questionnaire, based on a robust theoretical framework and expert consultation, included 40 items and was tested with 57 doctoral students. It demonstrated strong psychometric properties (comparative fit index = 0.946, Tucker-Lewis index = 0.92) but showed limitations such as item redundancy (α = 0.947) and low performance of general items. To address these issues, the questionnaire was refined to a concise 24-item version. The revised instrument was evaluated using a sample of 314 first-year student teachers. Exploratory and confirmatory factor analyses confirmed a four-factor structure, with each dimension demonstrating strong reliability (Cronbach's alpha ranging from 0.838 to 0.912) and excellent model fit indices (comparative fit index = 0.960, root mean square error of approximation = 0.0441). The results validate the Critical Artificial Intelligence Literacy Scale as a reliable and efficient tool for assessing artificial intelligence literacy in educational settings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ranieri et al. developed the Critical Artificial Intelligence Literacy Scale across knowledge-related, operational, critical, and ethical dimensions.
“Ranieri et al. (2025) developed the Critical Artificial Intelligence Literacy Scale across knowledgerelated, operational, critical, and ethical dimensions”
From From AI Literacy to AI Fluency: A Developmental Framework for Higher Education (2026) · cites this paper as Ranieri et al., 2025
Cites
Evaluating structural equation models with unobservable variables and measurement error
Fornell, C., & Larcker, D. F. (1981) · Journal of Marketing Research · doi:10.1177/002224378101800104 · Cited by 109,190 · see citing works
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · see citing works
Moore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150 · see citing works
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774 · see citing works
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., et al. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102642 · Cited by 4,584 · see citing works
A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires
Hinkin, T. R. (1998) · Organizational Research Methods · doi:10.1177/109442819800100106 · Cited by 4,095 · see citing works
Scale Development Research: A Content Analysis and Recommendations for Best Practices
Worthington, R. L., & Whittaker, T. A. (2006) · The Counseling Psychologist · doi:10.1177/0011000006288127 · Cited by 4,087 · see citing works
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · see citing works
Jourdan, A., & Laffly, D. (2020) · doi:10.1002/9781119720492.ch8 · Cited by 3,170 · see citing works
Conceptualizing AI literacy: An exploratory review
Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100041 · Cited by 2,205 · see citing works
What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00237-x · Cited by 2,003 · see citing works
Anderson, J. C., & Gerbing, D. W. (1991) · Journal of Applied Psychology · doi:10.1037/0021-9010.76.5.732 · Cited by 1,038 · see citing works
Wang, B., Rau, P.-L. P., & Yuan, T. (2023) · Behaviour & Information Technology · doi:10.1080/0144929x.2022.2072768 · Cited by 1,036 · see citing works
AI and education: guidance for policy-makers
UNESCO (2021) · doi:10.54675/pcsp7350 · Cited by 929 · see citing works
Challenges and Opportunities of Generative AI for Higher Education as Explained by ChatGPT
Michel-Villarreal, R., Vilalta-Perdomo, E., Salinas-Navarro, D. E., Thierry-Aguilera, R., & Gerardou, F. S. (2023) · Education Sciences · doi:10.3390/educsci13090856 · Cited by 877 · see citing works
ChatGPT in higher education: considerations for academic integrity and student learning
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.17 · Cited by 853 · see citing works
Wang, Y., & Wang, Y. (2019) · Interactive Learning Environments · doi:10.1080/10494820.2019.1674887 · Cited by 830 · see citing works
Initial validation of the general attitudes towards Artificial Intelligence Scale
Schepman, A., & Rodway, P. (2020) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2020.100014 · Cited by 785 · see citing works
McDonald, R. P. (2013) · Psychology Press eBooks · doi:10.4324/9781410601087 · Cited by 751 · see citing works
What are artificial intelligence literacy and competency? A comprehensive framework to support them
Chiu, T. K. F., Ahmad, Z., Ismailov, M., & Sanusi, I. T. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100171 · Cited by 570 · see citing works
Schriesheim, C. A., Powers, K. J., Scandura, T. A., Gardiner, C. C., & Lankau, M. J. (1993) · Journal of Management · doi:10.1177/014920639301900208 · Cited by 558 · see citing works
In search of artificial intelligence (AI) literacy in teacher education: A scoping review
Sperling, K., Stenberg, C.-J., McGrath, C., Åkerfeldt, A., Heintz, F., & Stenliden, L. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100169 · Cited by 498 · see citing works
Survey Measures of Web-Oriented Digital Literacy
Hargittai, E. (2005) · Social Science Computer Review · doi:10.1177/0894439305275911 · Cited by 489 · see citing works
Carolus, A., Koch, M. J., Straka, S., Latoschik, M. E., & Wienrich, C. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100014 · Cited by 488 · see citing works
The future of AI and education: Some cautionary notes
Selwyn, N. (2022) · European Journal of Education · doi:10.1111/ejed.12532 · Cited by 428 · see citing works
A review of AI teaching and learning from 2000 to 2020
Ng, D., Lee, M., Tan, R. J. Y., Hu, X., Downie, J. S., & Chu, S. K. (2022) · Education and Information Technologies : Official Journal of the IFIP technical committee on Education · doi:10.1007/s10639-022-11491-w · Cited by 409 · see citing works
Artificial intelligence self-efficacy: Scale development and validation
Wang, Y., & Chuang, Y.-W. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12015-w · Cited by 407 · see citing works
Zhang, H., Lee, I., Ali, S., DiPaola, D., Cheng, Y., & Breazeal, C. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00293-3 · Cited by 396 · see citing works
The Impact of Artificial Intelligence on Learning, Teaching, and EducationPreprint
Ilkka, T. (2018) · RePEc: Research Papers in Economics · doi:10.2760/12297 · Cited by 355 · see citing works
Sindermann, C., Sha, P., Zhou, M., Wernicke, J., Schmitt, H. S., Li, M., Sariyska, R., Stavrou, M., Becker, B., & Montag, C. (2021) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00689-0 · Cited by 343 · see citing works
Inclusive AI literacy for kids around the world
Druga, S., Vu, S. T., Likhith, E., & Qiu, T. (2019) · Proceedings of FabLearn 2019 · doi:10.1145/3311890.3311904 · Cited by 318 · see citing works
Developing Middle School Students' AI Literacy
Lee, I. A., Ali, S., Zhang, H., DiPaola, D., & Breazeal, C. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432513 · Cited by 317 · see citing works
Biagini, G. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00466-w · Cited by 236 · see citing works
Self-citationEmpowering learners for the age of artificial intelligence
Gašević, D., Siemens, G., & Sadiq, S. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100130 · Cited by 234 · see citing works
Ding, A.-C. E., Shi, L., Yang, H., & Choi, I. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100178 · Cited by 228 · see citing works
Laupichler, M. C., Aster, A., & Raupach, T. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100126 · Cited by 149 · see citing works
AI Literacy - Towards Measuring Human Competency in Artificial Intelligence
Pinski, M., & Benlian, A. (2023) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2023.021 · Cited by 94 · see citing works
Assessing the assessments: toward a multidimensional approach to AI literacy
Biagini, G. (2024) · Media Education · doi:10.36253/me-15831 · Cited by 63 · see citing works
Self-citationWenhua, Z., Hasan, M. K., Jailani, N. B., Islam, S., Safie, N., Albarakati, H. M., Aljohani, A. A., & Khan, M. A. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108134 · Cited by 34 · see citing works
Cuomo, S., Biagini, G., & Ranieri, M. (2022) · Media Education · doi:10.36253/me-13374 · Cited by 22 · see citing works
Self-citationA Psychometric Validation of the PAILQ-6: Perceived Artificial Intelligence Literacy Questionnaire
Grassini, S. (2024) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3679318.3685359 · Cited by 17 · see citing works
Introduction – The Importance of an Ethics-First Approach to the Development of AI
Floridi, L. (2021) · Philosophical studies series · doi:10.1007/978-3-030-81907-1_1 · Cited by 8 · see citing works
The global landscape of academic guidelines for generative AI and Large Language ModelsPreprint
Jiao, J., Afroogh, S., Chen, K., Atkinson, D., & Dhurandhar, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.18842 · Cited by 7 · see citing works
Biagini, G., Cuomo, S., & Ranieri, M. (2024) · Italian Journal of Educational Technology · doi:10.17471/2499-4324/1337 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Mirza, K. B., Arif, M., & Rafiq, M. (2026) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2026.103201 · Cited by 3 · see citing works
Bulut, M. A. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2647982 · Cited by 3 · see citing works
Che, C. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1755238 · Cited by 10 · see citing works
Kaya, İ., & Meylani, R. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1948694 · Cited by 2 · see citing works
Higher Education Language Teachers' Attitudes and Approaches to AI Integration into their Work
Korkealehto, K., & Ohinen-Salvén, M. (2025) · The EuroCALL Review · doi:10.4995/eurocall.2025.23992 · Cited by 1 · see citing works
From AI Literacy to AI Fluency: A Developmental Framework for Higher Education
Zentner, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7014818
Espinosa, F., Maya, E. O. A., González, M. C., Cevallos, K. P. A., & Reyes, Y. M. (2025) · Salud Ciencia y Tecnología - Serie de Conferencias · doi:10.56294/sctconf20251748
Designing AI Literacy for Primary Education
Lekota, T., & Ajani, O. A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-4893-3.ch008
Kong, S. C., & Hou, C. (2026) · Technology Knowledge and Learning · doi:10.1007/s10758-026-10003-w
Lee, D., & Tang, L. (2026) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2026.29.6.010
Integrating Open Science Into Teacher Education Programs
Popescu, C. R. G. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-4052-4.ch005
Predicting AI-Integrated Performance from Perceived AI Literacy Among Senior High STEM Learners
GUMANGAN, J. C. T., & Torres, J. (2026) · International Conference on e-Education, e-Business, e-Management and e-Learning · doi:10.1109/ic4e69618.2026.11631578
Culturally Responsive AI-Mediated Feedback
Kodithuwakku, D. S. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9898-3.ch008
Chahid, A., Marzak, Y. E., AOUANE, O., & Mansouri, K. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02001-9
Sharma, S., & Choudhary, A. (2026) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2026.2736629
Chen, P., Ravindran, L., Amini, M., & Arumugan, N. (2026) · Discover Education · doi:10.1007/s44217-026-02323-x
- Authors
- Ranieri, M., Biagini, G., & Cuomo, S.
- Affiliations
- University of Florence (IT)
- Venue
- International Journal of Digital Literacy and Digital Competence
- Publisher
- IGI Global Scientific Publishing
- Published
- 2025-09-11
- Volume
- 16
- Issue
- 1
- Pages
- 1-25
- ISSN
- 1947-3494 (ISSN-L)
- 1947-3508
- OpenAlex
- W4414116539
- Cited by
- 15
- References
- 44
- Citations by year
- 2026: 13 · 2025: 2
- FWCI
- 15.60
- Type
- journal article
- Language
- ng
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Journal of Digital Literacy and Digital Competence (publishedVersion, cc-by)
- Florence Research (University of Florence) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Digital literacy in education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Keywords
- AI literacy, scale development, psychometric validation, university students, doctoral students, AI knowledge, AI ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
